Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Fluentd v1.0 in a nutshell

7 380 vues

Publié le

Fluentd mini summit in OSSummit Japan

Publié dans : Technologie
  • Login to see the comments

  • Soyez le premier à aimer ceci

Fluentd v1.0 in a nutshell

  1. 1. Fluentd v1.0 in a nutshell June 1, 2017 Masahiro Nakagawa
  2. 2. Fluentd v0.12 • Current stable and widely used on production • Input, Parser, Filter, Formatter, Buffer, Output plugins • Known issues • Event time is second unit • No Windows support • No multi core support • Need to improve plugin API to support more various use cases
  3. 3. • Development version of v1 • Latest version is v0.14.17: March 30, 2017 • Implemented New features • New Plugin APIs • Event Time with Nanosecond resolution • ServerEngine based Supervisor • Windows support • Multicore support • New Plugin Helpers & Plugin Storage Fluentd v0.14
  4. 4. Fluentd v1 • Stable announcement for APIs / features • No breaking API changes in v1.x • Compatible with v0.12 and v0.14 • exclude v0 config syntax and detach_process • Release plan • Q3, 2017 • Need v0.14 feedback from developers and users
  5. 5. New Plugin APIs • Input/Output plugin APIs w/ well-controlled lifecycle • stop, shutdown, close, terminate • Integrate all output plugin into Fluent::Plugin::Output • New Buffer API for delayed commit and flexible chunking with metadata • parallel/async "commit" operation for chunks • For high latency case: forward’s at-least-once, issuing job, etc… • Users can choose chunk keys by configuration for dynamic parameters • Compatible w/ v0.12 plugins • compatibility layer for traditional APIs • it will be supported between v1.x versions
  6. 6. Router buffer_chunk_limit enqueue: exceed flush_interval or buffer_chunk_limit Key pattern: - BufferedOutput empty string or specified key -ObjectBufferedOutput tag -TimeSlicedOutput time slice emit emit Buffer Queue buffer_queue_limit Output OutputInput / Filter Tag Time Record Chunk Chunk Chunk Chunk Chunk key:foo key:bar key:baz v0.12 buffer design
  7. 7. v0.14 buffer design
  8. 8. Buffer keys and placeholders • Dynamic parameters for table name, object path and more • We can embed time, tag and any field with placeholder
 <match s3.**> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY']}" aws_sec_key "#{ENV['AWS_SECRETA_KEY']}" s3_bucket fluent-plugin-s3 path test/%Y/%m/${tag}/${key}/ <buffer time,tag,key> timekey 3600 </buffer> </match> http://docs.fluentd.org/v0.14/articles/buffer-section for more details time: 2017-06-01 12:00:00 +0700 tag: “test” record: {“key”:”hello”} - Event sample test/2017/6/test/hello/ - Generated “path”
  9. 9. Time with nanosecond • For sub-second systems: Elasticsearch, InfluxData, etc… • Fluent::EventTime • behaves as Integer for v0.12’s second unit compatibility • has methods to get sub-second resolution • be serialized into msgpack using Ext type • Fluent::Engine.now now returns EventTime, not Integer • Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-system (e.g., fluent-logger, Docker logging driver)
  10. 10. ServerEngine based Supervisor • ServerEngine is a framework for building robust server • https://github.com/treasure-data/serverengine • Replacing supervisor process with ServerEngine • it has SocketManager to share listening sockets between 2 or more worker processes • Replacing Fluentd's processing model from fork to spawn • to support Windows environment
  11. 11. Windows support • Fluentd and core plugins work on Windows • Windows service registration is also supported • http://docs.fluentd.org/v0.14/articles/install-by-msi • Use HTTP RPC instead of signals • https://github.com/fluent/fluent-plugin-windows-eventlog • We can collect windows eventlog :)
  12. 12. Symmetric multi core processing • 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine) • Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one (or some) output plugin • buffer paths are managed automatically by Fluentd. Need root_dir and @id parameters
  13. 13. Worker0 Supervisor v0.14’s multi process feature grep forward tdlog Worker1 Worker2 grep forward tdlog grep forward tdlog socket
  14. 14. Configuration example <system> workers 2 root_dir /var/log/fluentd </system> <source> @type forward </source> <filter pattern> @type grep </filter> <match pattern> @type tdlog @id out_td </match> /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta - buf_file’s path is automatically generated worker id root_dir plugin’s @id
  15. 15. TLS/Authn/Authz support for forward plugin • Support v1 forward protocol spec • secure-forward is merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/o SSL • Different points • secure-forward uses keep-alive, but forward doesn’t • secure-forward uses thread per connection, but forward uses cool.io, libev based IO. http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
  16. 16. Plugin Storage & Helpers • Plugin Storage: new plugin type for plugins • provides key-value storage to persistent intermediate status • built-in plugins: in-memory, local file • pluggable: 3rd party plugin to store data into storage • storage-redis • Plugin Helpers: • collections of utility methods for plugins • fully integrated with test drivers to run test code after setup phase of helpers (e.g. test started after created threads)
  17. 17. server helper: before def start @loop = Coolio::Loop.new @handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, @delimiter, method(:on_message)) @loop.attach(@handler) @thread = Thread.new(&method(:run)) end def shutdown @loop.watchers.each { |w| w.detach } @loop.stop @handler.close @thread.join end def run @loop.run rescue => e log.error "unexpected error", error: e log.error_backtrace end def on_message(msg, addr) # body end
  18. 18. server helper: after def start server_create(:foo_server, @port, bind: @bind) do |data, conn|
 # body
 end end
  19. 19. v0.12 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish”
  20. 20. v0.14 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish” Storage Helper
  21. 21. Other helpers • Timer: one-shot / periodic timer • Event Loop: Low-layer event loop • Socket: TCP/UDP/TLS support • Formatter/Parser: Manage parser/formatter plugins • Chile Process: Manage process for exec like plugin • etc…
  22. 22. Is v0.14 stable? • Not yet but it will be stable in this month (I hope!) • Remaining tickets: • Nested record support • Fix core bugs • v0.14 stable means ready for production, not feature freeze. Feature freeze is done by v1 release
  23. 23. TODO list for v1 • Counter API to store metrics between processes • https://github.com/fluent/fluentd/tree/counter-api • Optimize performance and remove deprecated code • Migrate more plugin into v0.14 API • Core plugins are migrated • Popular plugins release beta version for v0.14 • Write document: design, plugin, operation, etc…
  24. 24. Treasure Agent 3.0 serise (td-agent 3) • fluentd v0.14/v1, Ruby 2.4, and latest core components • Latest beta version is 3.0.1: May 19, 2017 • Environments • Add msi Windows package • Remove CentOS 5, Ubuntu 10.04 support • Beta packages have beed released • http://docs.fluentd.org/v0.14/categories/installation • beta will be removed after v0.14 becomes stable :)