Rakuten, Yohei Sasaki, Waldemar Quevedo, at Fluentd meetup, August 22nd, 2012

1. Logging infrastructure in PaaS
with Fluentd
Yohei Sasaki / Waldemar Quevedo
Paas Dev&Ops team
1
Rakuten. Inc, 22/08/2012

2. About us
§ ささきと申します。日本人です。 § ワリと申します。メキシコ人です。
(Sasaki, from Japan) (Wally, from Mexico)
§ Living in Japan since 1982. § Living in Tokyo since 2010.
§ Tech Lead of the PaaS Dev&Ops § Member of the PaaS Dev&Ops team
team at Rakuten, Architect of at Rakuten, programming in Ruby.
Rakuten PaaS.
§ Emacs user (org-mode rocks, try .org
§ Node.js user / node-fluentd-logger markup in Github)
§ CouchDB, …etc § OpenStreetMap (Mapnik rocks)
2

3. Agenda
§ Cloud Foundry
§ Cloud Foundry + Fluentd
§ Fluentd Pros & Cons
§ Misc
3

4. Cloud Foundry
• Open Application Platform as a Service project
• By VMware
• Written in Ruby
• OpenSource
http://github.com/cloudfoundry/vcap
• Hosted version
http://www.cloudfoundry.com/
4

5. Why Cloud Foundry?
• Last year it was decided to introduce Cloud Foundry to build a PaaS
in Rakuten
http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html
http://www.slideshare.net/rakutentech/cloud-foundry-10423477
• Main points:
– Open Source
– Separation of concerns from IaaS
– Scalable architecture
– Multi language
5

6. What Cloud Foundry does?
• Easy to deploy applications!!
$ mkdir app.js
$ vi app.js
$ vmc push yssk22-myapp
$ vmc instances yssk22-myapp 3
• Not easy to deploy business ready applications yet.
– We are now solving our own issues by modifying Cloud Foundry source
– One critical thing is Logging!
– Now we’ve solved this matter basically, let’s share our case study.
6

7. Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!
!
$ curl http://yssk22-myapp.cloudfoundry.com/!
!
$ vmc logs yssk22-myapp!
!
[Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !
!
$ vmc update yssk22-myapp!
!
$ vmc logs yssk22-myapp!
7

8. Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!
!
$ curl http://yssk22-myapp.cloudfoundry.com/!
!
$ vmc logs yssk22-myapp!
!
[Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !
!
$ vmc update yssk22-myapp!
!
$ vmc logs yssk22-myapp!
Oops, logs are gone.
8

9. Persisting logs is a must for business applications!
• Monitoring
• Analysis
• Visualization
Ask a data scientist =>
Sorry we lost
everything :P
http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist
9

10. Cloud Foundry does not persists the logs from apps…
• Enter Fluentd!
10

11. Brief overview of Cloud Foundry architecture
Droplet
Execution
Agent
Source: http://www.slideshare.net/derekcollison/design-of-cloud-foundry
11

12. Approach
Attach a Fluentd collector to one application instance upon dispatch.
DEA
app fluentd
app fluentd
Log Storage
DEA
app fluentd
app fluentd
12

13. Collecting the logs from the apps
DEA
• Log collector monitors changes in the files
app fluentd – Stdout, stderr logs in general
– Catalina, manager, host-manager logs for Java
– error.log for PHP apps
app fluentd
app fluentd • Also makes available:
– VCAP_LOG_PORT env variable to flush logs
directly
app fluentd
– VCAP_LOG_STREAM_PORT to inspect the
logs in realtime
app fluentd
13

14. Collecting the logs from the apps
DEA
app fluentd Log Storage
Fluent::Logger => VCAP_LOG_PORT
Sends logs directly to Fluentd making
unnecessary having to write to file. VCAP_LOG_STREAM_PORT
Streams the logs to a client
{‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’}
14

15. Small setup for collecting the logs
DEA CloudController
app fluentd
Log Storage Server
app fluentd /logs/:path
fluentd Rack app
Collected log files
DEA
Log Storage Server CloudController
app fluentd
Rack app
app fluentd
Collected log files /logs/:path
15

16. How is it setup?
app/ !
fluentd/! Attached to an application instance on
|-- stderr.stat (1)!
startup by DEA
|-- stdout.stat (2)!
• (1) and (2) is a “stat file” used by the input plugin,
|-- fluentd.conf (3)!
which allows it to pick up from the last position it
|-- startup (4)! had monitored.
|-- stop (5)!
fluentd_child.pid (6)!
• (3) is the configuration of the plugins dynamically
generated by the DEA
fluentd.pid (7)!
logs/! • (4) and (5) are the scripts executed to manage the
|-- fluentd.log (8)! fluentd process
|-- stdout.log!
• (6) and (7) are the supervisor and main process
|-- stderr.log! ids from Fluentd
ruby!
run.pid! • (8) is where any logs produced by the logging
daemon are stored
startup!
stop!
16

17. fluentd.conf :: Input configuration
!
Set up VCAP_LOG_PORT to flush the logs
<source>!
directly
type forward!
port 12345!
</source>!
!
<source>!
type cf_app_logs!
instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a!
runtime ruby19!
framework sinatra!
tag logs.storage!
</source>!
!
!
Specify runtime and dir to monitor files
17

18. fluentd.conf :: Output configuration
<match logs.**>!
type copy! Settings for when forwarding
<store>! to the log storage, we include user
type cf_app_forward! information here so that it cannot be
flush_interval 5s! overrided by a logging library.
buffer_type file!
buffer_path fluentd/fluentd_buffer!
!
cf_app_user foo@example.com!
cf_app_name appname!
cf_app_instance_id 3e9676485562c64d81a61de2df05903a!
cf_app_instance_index 0!
18

19. fluentd.conf :: Output configuration
<server>!
name logserver! Location of the log storage server:
host 192.168.2.1! - We use a active-standby setup at the
port 4224! moment.
</server>!
- When primary Logserver is down, logs
<server>!
are forwarded to the one on standby.
name backup_logserver!
host 192.168.2.2!
port 4224!
When both are down, fluentd buffers to
standby!
files the logs locally until one of them is
</server>! available.
</store>!
<store>!
type cf_app_logs_streaming!
port 43208!
</store>!
</match>!
Set up VCAP_LOG_STREAM port to
enable tail –f like support.
19

20. …And now that we managed to collect the logs,
we also can support the following 2 commands:
20

21. vmc tail
21

22. vmc log-storage
22

23. What we could do
• Persisting the logs in Cloud Foundry is a must!
• Support to persist the logs is a key issue in adoption of Cloud
Foundry for developing application services.
• Thanks to Fluentd we could enable users to have log archives in a
straightforward way.
• Cloud Foundry and Fluentd themselves are both written in Ruby so
integration was relatively easy.
23

24. What we could not do
• It was difficult to let the user have access to the logs from Nginx
(used as a Router in CF)
# Sample release process!
vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp!
vmc map app-v1 olympics.rakuten.co.jp => Released!!
vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp!
vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps!
vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.!
• Then the app-v1.cf.rakuten.co.jp subdomain becomes available
to other user.
• Need to keep track that user X had Y domain at Z time for
application named A in real time while storing the logs.
24

25. Fluentd: Pros and Cons
• Pros
– Easy to include in Cloud Foundry compared to other solutions
• Possible to include it like gem 'fluentd' in a Ruby component
– Easy to change storage technology depending on scale
– Easy to extend
– Growing community o/
• Cons
– Fluentd needs better testing tools
– Better materials about Fluentd from the community?
• Learning curve is already low
• Just we need to share more about how we are using Fluentd
• e.g. 'The Little Fluentd Book', 'Advanced Fluentd Recipes’
– Cool.io is not maintained
25

26. Misc: Playing with the message bus
• Cloud Foundry is built with Eventmachine.
• Fluentd is built with Cool.io.
• Cool.io and Eventmachine do not play along very well.
• The message bus from Cloud Foundry named NATS is also built
using Eventmachine.
26

27. Misc: NATS input plugin
# This works, though a warning is displayed from Cool.io overriding Eventmachine!
require 'nats/client'!
!
module Fluent!
class NatsInput < Input!
Plugin.register_input('nats', self)!
def run!
NATS.start(:uri => @nats_connection_uri) do!
!
# Triggered when an url changing event occurs (remap, change)!
NATS.subscribe('dea.update') do |msg|!
Engine.emit('logs.dea_update', Time.now.to_i, msg)!
end!
!
# Triggered when an application is started or dea is started!
NATS.subscribe('dea.*.start') do |msg|!
Engine.emit('logs.dea_start', Time.now.to_i, msg)!
end!
end!
end!
27

28. Misc: Adding new forwarding nodes
dynamically through NATS?
# DEA
@droplets.each_value do |instance|
begin
instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port])
instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 })
rescue => e
@logger.error "Error: #{e} -- #{e.backtrace}"
end
end
# Extended Forward plugin
module Fluent
class CloudfoundryAppForwardOutput < ForwardOutput
Plugin.register_output('cf_app_forward', self)
def emit(tag, es, chain)
es.each do |emit_time, record|
# Add a new forwarding server
if record.has_key?('type') and record['type'] == 'serverConf’
add_node(record['host'], record['port’])
next
end
28

29. FIN
You can enable log collection to your Cloud Foundry by merging this
patch:
https://github.com/rakutentech/dea/commit/
64d271904de1e195cc90ca1958eadf704f530d94
Checkout our github repo for more Cloud Foundry related things:
https://github.com/rakutentech/
That’s all, thanks!
ご清聴ありがとうございました！
29