Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Fluentd Hacking Guide at RubyKaigi 2014

68 954 vues

Publié le

Fluentd Hacking Guide at RubyKaigi 2014

Publié dans : Ingénierie
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Fluentd Hacking Guide at RubyKaigi 2014

  1. 1. Fluentd Hacking Guide (Fluentd ソースコード完全解説) September 20th, 2014 ! Naotoshi Seo @sonots DeNA Co., Ltd. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  2. 2. 2 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Who am I? ・Naotoshi Seo @sonots ・DeNA Co, Ltd. ・Infrastructure Engineer ・Fluentd Commiter
  3. 3. ⁃ An application (or a framework) to process log streaming ⁃ Powerful plugin architecture (+250 plugins) ⁃ written in Ruby Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  4. 4. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Disclaimer 4 1. I do not explain what is Fluentd, how to use Fluentd 2. I assume audiences have ever used Fluentd, have ever created Fluentd plugins, have interests inside Fluentd ! 3. Source Codes are from v0.10 branch as of September 2014
  5. 5. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 5 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works 4. How Fluentd parses the config file 5. The event-driven programming using cool.io, and effects of GVL
  6. 6. 6 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Let's Dive into Source Codes
  7. 7. 7 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. vim
  8. 8. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 8 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  9. 9. The Bootstrap Sequence 9 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Fluent::Supervisor#run_configure 1. require 2. new 3. configure(conf ) Fluent::Supervisor#run_engine 4. start 5. shutdown (if signal received) Input plugin creates threads on #start Output plugin does nothing on #start (typically)
  10. 10. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 10 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  11. 11. Dataflow from Input to Output 11 Input Engine Output Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag, es) emit(tag, es) If an input thread receives data, call Engine.emit
  12. 12. CAUTION: Output Plugin 12 Input Engine Output Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag, es) emit(tag, es) BLOCK!!! Can not receive new input during blocking ex) HTTP POST
  13. 13. 13 Use BufferedOutput Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. to avoid blocking
  14. 14. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 14 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  15. 15. BufferedOutput Plugin Input Engine BufferedOutput BasicBuffer 15 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag,es) emit(tag,es) emit(tag,data) enqueue immediately return!! Can receive new input. No blocking.
  16. 16. BufferedOutput Plugin 16 BufferedOutput BasicBuffer Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. try_flush (push) OutputThread pop write(chunk) do some EXPENSIVE things Run heavy processing in OTHER threads
  17. 17. CAUTION: BufferedOutput 17 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Stuck if enqueued over its capability ! HOW TO IMPROVE 1. Increase num_threads 2. Enlarge buffer_chunk_limit 3. Set smaller queued_chunk_flush_interval, try_flush_interval (Secret parameters)
  18. 18. 18 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. But,
  19. 19. 19 Improving actual processing throughputs is most important fluent-plugin-elasticsearch gets stuck? Then, tune Elasticsearch! Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  20. 20. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Conclusion 20 1. Output plugin blocks 2. BufferedOutput does not block, but stuck if enqueued over its capability • Tune with option parameters such as num_threads 3. In either case, improving actual processing capability itself is most important!! 4. I am happy if this talk helps your operation, your plugin development, and contributions

×