Keeping it realtime
•
6 j'aime•2,875 vues
Keeping it Real Time: Globally distributed, high volume data processing optimized for scale, reliability & cost Krux is an infrastructure provider for many of the websites and services you use online today, like NYTimes.com, WSJ.com, Roku and Ticketmaster. For every request on those properties, Krux will get one or more as well. We grew from zero traffic to several billion requests per day from all over the world in the span of 2 years, and we did so exclusively in AWS. As anyone building a big data infrastructure will be able to tell you, is that turn around time requirements for processing data are getting shorter and shorter and (near) real time is becoming the norm. However, processing data efficiently, reliably, resiliently, at scale, at minimal cost and without burdening your developers and operators is a tremendous challenge, especially in a cloud environment. This is the story of all the challenges & pitfalls we encountered, and how, through architecture, convention and sensible trade offs, we managed to build a real time infrastructure that is at the core of our data processing and incredibly economical to build, scale & operate. I will share with you the details of how we set up and operate our globally distributed real-time pipeline, based on Kafka and related tools, to capture and process over 100k messages every second from virtually every part of the system, including external requests, user apps and the operating system itself, for under $4500/month using off the shelf Open Source software and some code we've released as Open Source ourselves. This content will be applicable for anyone processing or desiring to process vast amounts of messages in real-time in a cloud or datacenter setting.
Recommandé
Recommandé
Contenu connexe
Dernier
Dernier (20)
En vedette
En vedette (20)
Keeping it realtime
- 1. KEEPING IT REALTIME Globally distributed Kafka, optimized for scale, reliability & cost ! Jos Boumans - @jiboumans http://www.jamesallenonf1.com/2015/02/why-the-look-of-cars-is-so-important-to-f1-revamp/
- 3. http://fotos.subefotos.com/7a6b3e6df9453d5adf150087e5300834o.jpg How to measure everything Architecting in AWS for resilience & cost www.slideshare.net/jiboumans/aws-architecting-for-resilience-cost-at-scale http://www.slideshare.net/jiboumans/how-to-measure-everything-a-million-metrics-per-second-with-minimal-developer-overhead
- 4. KRUX VP of Operations & Infrastructure http://www.krux.com/
- 5. SOME OF OUR CUSTOMERS
- 7. AVERAGE DATA EVENTS / SEC http://investor.fb.com/results.cfm http://www.statisticbrain.com/twitter-statistics/ http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm 0 30,000 60,000 90,000 120,000 150,000 Twitter: NewTweets Wikipedia: PageViews Facebook: Messages Sent (Oct 2014) Krux: PageView Data Points
- 8. MONTHLY UNIQUE USERS 0M 500M 1,000M 1,500M 2,000M 2,500M http://reportcard.wmflabs.org/ http://www.statisticbrain.com/twitter-statistics/ http://newsroom.fb.com/company-info/
- 10. USE CASES Discovery, Processing & Statistics http://www.antiquefarmhouse.com/past/southern-inspired-decor/vintage-style-handcrafted-wooden-suitcases-set-of-3.html
- 14. MAKING GOOD CHOICES It All Dependstm http://myyearwithoutclothesshopping.com/shopping-clothes-emotions/defining-moments-and-powerful-choices/
- 15. Kinesis SQS RabbitMQ Kafka Features Scale up via API call Scale up by AWS Open Source (erlang) Apache Open Source (scala) Very consistent latencies Dead letter queue Dead letter queue Performance Disk IO bound Works with AWS tools Works with AWS tools Message & queue TTL TTL on age or queue size Concerns Limited amount of supported clients libs Cross datacenter replication Poor partition tolerance (documented) Metrics & insights need custom code All complex logic in client Maximum payload 64kb Performance/ availability memory bound Built in management via CLI only Maximum life cycle 24 hours Maximum life cycle 14 days Drop acked messages Reliance on Zookeeper 5 QPS/Shard Cost at scale Verdict ? X X :) https://aphyr.com/tags/jepsen http://www.slideshare.net/adamw1pl/eval-repl-mq
- 16. BASIC QUEUE SETUP The ‘Hello world!’ of Kafka https://tammydotts.wordpress.com/2012/08/15/ikea-hangover/
- 17. KAFKA At least once: Default At most once: Client logic http://www.thessdreview.com/daily-news/latest-buzz/current-ssd-projects-on-the-go/ http://kafka.apache.org/documentation.html#quickstart http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen
- 18. ZOOKEEPER Stores state for Kafka Producers & Consumers Use Netflix’ Exhibitor to manage it http://www.amazon.com/If-Ran-Zoo-Classic-Seuss-ebook/dp/B00ESF29CA https://zookeeper.apache.org https://github.com/Netflix/exhibitor/wiki
- 20. ADVANCED SETUP Filling in the missing pieces http://www.eurobricks.com/forum/index.php?showtopic=100681
- 21. Web Server User Request Logs Kafka ??? Web Server User Request Logs Kafka ??? Transport & Stream Listener Message Processing Web Server Request Kafka Transport & Krux Stream INSERTING MESSAGES
- 23. OPTIMIZE LOGS FOR USAGE Log directly to JSON LogFormat ‘{ "TS": "%{%s}t", "PATH": "%U", "RESP": "%>s", "RT": "%D", "V": "1" }' response_time http://httpd.apache.org/docs/2.4/mod/mod_log_config.html
- 24. RELAY DATA IN REALTIME netcat & udp are your friends CustomLog "| /bin/nc -uj localhost 9999 " response_time https://code.google.com/p/openbsd-netcat/source/browse/trunk/netcat.c#73
- 25. TRANSPORT OPTIONS We use PipeD (Krux OSS) on localhost Also consider Heka, FluentD or Flume http://www.launchgram.com/videogames/super-mario-bros-u/ https://github.com/krux/piped
- 26. { udp_port: 9002, tcp_port: 9002, admin_port: 9003, ! statsd_port: 8125, statsd_prefix: "prod.piped.topic1.", ! servers: [ [ "tcp://listener1:9002", "tcp://listener2:9002", "tcp://listener3:9002"," ] ] } https://github.com/krux/piped/blob/master/etc/config.js.example
- 27. $ echo stats | telnet localhost 9003 { "stats": { "connections": { "admin": 6, "tcp": 0, "udp": 889481, "socket": 0, "file": 0, "failed": 0 "idle": 0, "total": 889481, "last": 1442619384, }, "start_time": 1442614299, "uptime": 5084.679370715 }, "active_servers": [ "tcp://listener1:9002" ], "all_servers": { "tcp://listener1:9002": { "available": true, "slow": false, "down": false, "messages": 889481, "state_changes": 1, "state_changed": 5083, "idle": 0, "last_sent": 1442619384, }, .... } } https://github.com/krux/piped
- 28. CONVERSION Use Krux Kafka Stream listener to convert log lines to Kafka Messages http://www.huffingtonpost.com/linda-roy/life-lessons-through-rock-music_b_5551259.html https://github.com/krux/tcp-stream-kafka-producer
- 29. java -jar krux-tcp-stream-kafka-producer-1.3.0-full.jar --client.id listener1 --port.topic 9001:topic1 --port.topic 9002:topic2 --http-port 9999 --heartbeat-topic listener.heartbeat --compression.codec snappy --stats https://github.com/krux/java-stdlib https://github.com/krux/tcp-stream-kafka-producer
- 30. $ curl -v http://localhost:9999/__status | jq . {"state": "OK", "status": "tcp_stream_listener_server is running nominally", "version": "1.2.5", "client_id": "server1.example.com", "last_msg_proc_time_nsec": { "topic1": 81768, "topic2": 168378 }, "topic_message_rates": { "topic1": { "count": 1624, "mean_rate": 0.12189879603244635, "5_min_rate": 0.1373407202192449, "15_min_rate": 0.13443727558716745, "1_min_rate": 0.15729613021961694 }, .... "port_mappings": { "9001": ["topic1"], "9002": ["topic2"] } } https://stedolan.github.io/jq/
- 31. snappy compression.codec https://kafka.apache.org/08/ops.html async producer.type 1 required.acks
- 33. Web Server User Request Logs Kafka ??? Web Server User Request Logs Kafka ??? Transport & Stream Listener Message Processing Web Server User Request Logs Kafka Transport & Krux Stream Listener Message Processing Kafka Stdlib or Stream Processors MESSAGE PROCESSING Several ways to skin a cat
- 35. ### Logstash configuration for consuming from Kafka - Part 1 ! input { kafka { auto_offset_reset => smallest group_id => "logstash-cluster" white_list => "topic1,topic2" decorate_events => true zk_connect => "zk1:2181,zk2:2181,zk3:2181/kafka-root" } } ! filter { mutate { add_field => { "_temp_type" => "kafka:%{[kafka][topic]}" } } mutate { rename => { "_temp_type" => "type" } } } https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html
- 36. ### Logstash configuration for consuming from Kafka - Part 2 ! output { elasticsearch { host => [ "es1", "es2", "es3" ] cluster => "elasticsearch-cluster" index => "%{[kafka][consumer_group]}-%{[kafka][topic]}-% {+YYYY.MM.dd.HH}" } ! statsd { namespace => "prod.logstash" increment => ["%{[kafka][consumer_group].%{[kafka][topic]}"] } } https://www.elastic.co/guide/en/logstash/current/plugins-outputs-statsd.html https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
- 37. DIY PROCESSING Krux Kafka Stdlib (java) http://www.amazon.com/SK-Hand-Tools-86255-Combination/dp/B0002NYCSQ https://github.com/krux/java-stdlib https://github.com/krux/java-kafka-client-libs
- 38. // democonsumer --topic-threads topic1,5 --topic-threads topic2 public class DemoConsumer { public static void main( String[] args ) { // Sets up stdlib & kafka CLI options OptionParser parser = new OptionParser(); KafkaConsumer.addStandardOptionsToParser( parser ); KruxStdLib.setOptionParser( parser ); OptionSet options = KruxStdLib.initialize( args ); // Attach the handler to the consumer final DemoHandler<Object> myHandler = new DemoHandler<Object>(); KafkaConsumer runner = new KafkaConsumer( options, myHandler ); // Run the consumer runner.start(); } public class DemoHandler<T> implements MessageHandler { public void onMessage( Object message ) { // Business logic for message processing goes here KruxStdLib.STATSD.incr("message_size", message.length); } } https://github.com/krux/java-kafka-client-libs
- 42. ??? Web Request in DC2 User Kafka in DC2 & Krux Mirror Maker Web Server User Request Logs Kafka in DC1 Transport & Krux Stream Listener Message Processing Krux Kafka Stdlib or Stream Processors Web Request in DC2 User Web Server User Request Logs Kafka in DC1 Transport & Krux Stream Listener Message Processing Krux Kafka Stdlib or Stream Processors REPLICATION Optimize for central processing
- 44. java -jar krux-kafka-mirror-maker-0.4.1.jar --consumer-config mirror.consumer --producer-config mirror.producer --blacklist listener.heartbeat --http-port 9090 --stats ! # mirror.producer via SSH Tunnel (optional) metadata.broker.list = localhost:19092,localhost:19093,... https://github.com/krux/kafka-mirror-maker
- 45. $ curl http://localhost:9090/__status | jq . { "state": "OK", "status": "krux_mirror_maker is running nominally", "version": "0.4.1", "mirrored_topics": [ "topic1", "topic2" ], "topic_consumption_stats": { "topic1": { "count": 105031, "mean_rate": 0.12150422084817518, "5_min_rate": 0.12091734189375856, "15_min_rate": 0.11807509586505133, "1_min_rate": 0.14123037951258793 }, ... } } https://stedolan.github.io/jq/
- 47. MANAGEMENT & INSIGHTS Trust but verify http://www.dilbertcelart.com/dale/c26.jpg
- 48. Kafka in DC2 & Krux Mirror Maker Web Request in DC2 User Web Server User Request Logs Kafka in DC1 Transport & Krux Stream Listener Message Processing Krux Kafka Stdlib or Stream Processors Stats & Insights ??? A PICTURE IS WORTH A THOUSAND CLI CALLS
- 49. KAFKA MANAGER FromYahoo - manage clusters, topics & partitions via UI https://github.com/yahoo/kafka-manager
- 50. INTEGRATING GRAPHITE Continuous metrics & reporting via Krux Kafka Metrics Reporter https://github.com/krux/kafka-metrics-reporter https://kafka.apache.org/documentation.html#monitoring
- 51. ### ### Kafka metrics config settings - server.properties ### ! # Add this jar to your Java Classpath kafka.metrics.reporters= com.krux.metrics.reporter.KafkaGraphiteMetricsReporter kafka.graphite.metrics.reporter.enabled=true kafka.graphite.metrics.host=graphite.example.com kafka.graphite.metrics.port=2013 kafka.graphite.metrics.env=prod https://github.com/krux/kafka-metrics-reporter
- 52. TOPIC LAG As provided by Krux Kafka Metrics Reporter
- 54. Kafka in DC2 & Krux Mirror Maker Web Request in DC2 User Web Server User Request Logs Kafka in DC1 Transport & Krux Stream Listener Message Processing Krux Kafka Stdlib or Stream Processors Stats & Insights Krux Kafka Graphite Kafka Manager THE FINAL PICTURE