Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Real-time Big Data at FPT (for TechCamp University)

1 036 vues

Publié le

some key ideas to build real-time big data platform from open source tools:
+ Apache Spark
+ Reactive Function X (RFX)

Publié dans : Données & analyses
  • Identifiez-vous pour voir les commentaires

Real-time Big Data at FPT (for TechCamp University)

  1. 1. Real-time Big Data at FPT and some key ideas to build real-time big data platform from open source tools ○ Apache Spark ○ Reactive Function X (RFX) Presented by @tantrieuf31 http://nguyentantrieu.info
  2. 2. about me ? ● Full Stack Engineer and Tech Lead at AdsPlay, startup project from FPT Telecom ● Founder at RFXLab.com, building RFX framework and Fast Data Intelligence Platform for Data-driven Organization ● Tech Blogger at http://engineering.adsplay.net
  3. 3. Abstract 1. Just 5 minutes about the history of “Big Data” 2. Does Big Data solve big problems ? 3. Overview about Open Source Tools a. Netty (Event Collector) b. Kafka (Event Queue) c. RFX-Stream (Event Processor) d. Apache Spark (Big Data processing engine) e. RFX-Iris (Fast Data Query Interface)
  4. 4. 5 minutes about the history of “Big Data”
  5. 5. Imagine what if you have to build a GREAT pyramid ? In fact, the Big Data was born in 3000 years ago. When you have to build a great thing, you would face with making decisions with lots of data.
  6. 6. How ? Decisions without Data ?
  7. 7. OK, let’s get back to 2015
  8. 8. What if the business is not driven by data? Refer: http://www.nytimes.com/2011/04/24/business/24unboxed.html
  9. 9. Since 2015, the Fast Data, a new trend, has been replacing Big Data http://www.tibco.com/blog/2015/03/27/how-analytics- facilitates-fast-data
  10. 10. 1970s 1990s 2000s 2010s Data Management Technology and Trends ● Netty.io ● Apache Storm ● Apache Kafka ● Apache Spark ● RFX ● ... ● Hadoop Ecosystem ● NoSQL Ecosystem ● ... ● Oracle ● MySQL ● PostgreSQL ● ...
  11. 11. “ Does Big Data solve our big problems ?
  12. 12. tracking all access logs and user’s activities Processing in real- time( seconds) ! Storing multiple types of log (video, web, mobile, like, comment, play, … )
  13. 13. http://www.rfxlab.com
  14. 14. boosting Sale Revenue / Profit Log events Reactive events
  15. 15. How is the Big Data used at FPT ?
  16. 16. Does Vietnamese love football ? The correlation said YES
  17. 17. Analyzing trending events in real-time !
  18. 18. Visualizing all user’s devices
  19. 19. Real-time Big Data Architecture
  20. 20. “ How to build an “Just-Work” real- time big data system ?
  21. 21. KEY IDEA is “divide and conquer”
  22. 22. User Story in plain English 1. Hercules is thinking about some questions. E. g: What’s hot songs of Nhacso on Facebook ? 2. He decides to ask Iris about this question. 3. Iris analyzes the question into “query messages” and deliver them to Zeus. 4. Zeus uses his power of “large-scale data processing” to answer the question. 5. Done, Zeus return the result “hot songs on Facebook” for Iris. 6. She sends the result to Hercules
  23. 23. Visualizing our user story Question about Big Data: What’s hot songs of NhacSo.net on Facebook ? messages ZeusIris Hercules
  24. 24. Let’s see how it works
  25. 25. Awesome Open Source Projects to follow RFXLab.com ◎ http://www.rfxlab.com ◎ https://github.com/rfxlab Kafka : http://kafka.apache.org Hadoop http://hadoop.apache.org Apache Spark https://spark.apache.org
  26. 26. Awesome Open Source Projects to follow Native Kafka driver: https://github. com/edenhill/librdkafka/ PHP Kafka driver: https://github. com/EVODelavega/phpkafka Data Visualization JavaScript Library https://github.com/nvd3-community/nvd3
  27. 27. Good ref books "Spend some time alone and learn to develop your personal resources." Alexander Reid Martin
  28. 28. More info at http://engineering.adsplay.net/jobs-at-adsplay-team