Modern Fast Streaming Data
•
2 likes•532 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1j8YoCp. Todd Montgomery challenges some of the common myths and misconceptions about high performance streaming data, and takes a look at what is really possible today. Filmed at qconnewyork.com. Todd Montgomery is a networking hacker who has researched, designed, and built numerous protocols, messaging-oriented middleware systems, and real-time data systems, done research for NASA, contributed to the IETF and IEEE, and cofounded two startups. He currently works for Kaazing as Chief Architect. @toddlmontgomery
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Modern Fast Streaming Data
- 1. Modern Fast Streaming Data Todd L. Montgomery @toddlmontgomery
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /myths-data-streaming
- 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. Why Should We Care? Myths & Misconceptions You can’t escape the Math Technologies & Techniques
- 5. Why Should We Care?
- 6. Human Knowledge is now doubling every year* * by discipline, 12-18 months
- 8. Middle Ages - 1500 yrs Renaissance - 250 yrs Industrial Revolution - 150 yrs WWII - 25 years Buckminster Fuller - Critical Path 1981
- 9. IoT
- 11. In the near future, Human Knowledge could double every 72 hours
- 12. What this could mean for our systems…
- 14. Updates/Sec = Devices * Frequency * Market Share Either ingest or streaming. 2x for Request/Response
- 15. Updates/Sec = Devices * Frequency * Market Share 9 Billion (Today) 50 Billion by 2020 (Cisco) 26 Billion by 2020 (Smartphone/Tablet - Gartner) 75 Billion by 2020 (Morgan Stanley)
- 16. Updates/Sec = 50 Billion * 6/min * 1% = 50 Million/sec
- 17. Bandwidth = 50 Billion * 6/min * 1% * 200 bytes = 9.3 GB/s (74.5 Gb/s)
- 20. Social & Societal demands will require processing an intense stream of data in real-time
- 23. Myth (CPUs, Storage, Networks) are not capable of processing in real-time* * for some unknown, unquantified data volume
- 25. http://en.wikipedia.org/wiki/Instructions_per_second Year Processor MIPS 1974 Intel 8080 0.29 1982 Intel 286 1.28 1993 PowerPC 601 157 2003 Pentium 4 Extreme 9,726 2008 Intel Core i7 920 (Quad) 82,300 2011 Intel Core i7 2600K (4/8) Sandy Bridge 128,300 2014 Intel Core i7 5960x (8/16) Haswell 298,190
- 26. Raspberry Pi 2 (Quad) 1,186 MIPS! http://en.wikipedia.org/wiki/Instructions_per_second
- 27. DDRSSD PCIe - 3 100 GbE … OmniPath
- 28. 1 thread of awesome > 128 cores of so-so http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://blog.acolyer.org/2015/06/05/scalability-but-at-what-cost/
- 29. Misconception Data needs to come to rest to be processed
- 30. Data at rest is a liability
- 31. Data at rest is a liability MDMILM WarehouseETL
- 32. Why would you want transient data at rest?
- 33. But what about sort? …REALLY?!
- 34. You can’t escape the Math
- 35. … and you don’t want to The Math will guide you to a solution
- 36. "AmdahlsLaw" by Daniels220 at English Wikipedia - Own work based on: File:AmdahlsLaw.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons
- 37. Setup & Scheduling Work Unit Work UnitWork UnitWork Unit Post Processing
- 38. Setup & Scheduling Work Unit Work UnitWork UnitWork Unit Post Processing Contention Contention
- 39. Contention isn’t the biggest enemy
- 40. Coherence is!
- 41. Universal Scalability Law 0 2 4 6 8 10 12 14 16 18 20 1 2 4 8 16 32 64 128 256 512 1024 Speedup Processors Amdahl USL
- 42. Setup & Scheduling Work Unit Work UnitWork UnitWork Unit Post Processing Contention Contention Contention + Coherence Contention + Coherence
- 43. Up Front Partitioning Work Unit Work UnitWork UnitWork Unit
- 44. … and more Queuing Theory ⭐ Complexity Theory CAP Theorem
- 46. The Essence of Architecture Data Structures Protocols of Interaction Mechanical Sympathy
- 50. Technique Smart Batching (Natural Batching) http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
- 51. Resource
- 53. Batching Thread Resource Pull off as much waiting data as possible
- 54. Single Writer Principle Avoid Resource Contention Batching only when needed Rate Decoupling Back Pressure
- 57. Words Matter
- 58. Obstruction-Freedom Partially completed operations aborted & changes made rolled back
- 59. Lock-Freedom Individual thread may starve, but guaranteed system-wide throughput Lock-Free is Obstruction-Free
- 60. Wait-Freedom Starvation free and guaranteed system-wide throughput Wait-Free is Lock-Free
- 61. These properties are awesome! Who wouldn’t want them?
- 62. System-wide properties start at the lowest level
- 63. Essence Just because we could take an action right now, doesn’t mean we should
- 70. [2] = 0 [1] = 2 [0] = 4[N] = 0 4 2 0 0… 4200 Shared View
- 75. What If…? The Data Structure could be directly sent to the “network”?
- 76. Header Message
- 84. Natural for broadcast replication
- 85. In Closing…
- 86. A flood of data is coming, Many say it is already here, How will you deal with it?
- 87. Ever seen a PitBull drink from a Fire Hydrant?
- 89. It won’t give up… Be the Pitbull at the Fire Hydrant!
- 90. @toddlmontgomery Questions? • Aeron https://github.com/real-logic/Aeron • SlideShare http://www.slideshare.com/toddleemontgomery • Twitter @toddlmontgomery Thank You!
- 91. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/myths- data-streaming