To capture and perform analysis the data stream generated by the VRT NU and VRT NWS platforms, we built a high-performance and real-time data streaming platform based on Apache Kafka. These slides contain the lessons learned and a brief introduction to our open-sourced architecture

1. Data streaming at
matthias@matthiasdv.comgithub.com/dataprism

2. What and why?

3. Lots of data!

4. Interesting problems!
- User preferences
- User context
- Content firehose
- Publication strategy
- Segment analysis
- …

5. Design considerations

6. Design considerations
- Continuous data
- Latency matters
- High fault tolerance
- Team size

7. kappa architecture

9. Technology stack

10. stream processing platform
( + persistence )

11. Tech stack
- Apps are easy to scale and deploy
- Optimised for throughput
- Streaming model

13. Synchronization

14. “Old programs read like quiet conversations
between a well-spoken research worker and
a well-studied mechanical colleague, not as a
debate with a compiler.”

15. Tech stack
- Simplicity
- Concurrency model
- Small footprint

16. logic
Streams API

18. Tech stack
Sync Logic

19. ingestion stream logic serve

20. Even with this setup lots of time is spent on
routine tasks such as deploying containers,
and operating the cluster

21. We have a lean data team

22. dataprism-ui
( DataOps )

24. - node js
- does the scaffolding for you
Dataprism
Logic Development Kit

26. We’re on GitHub!
github.com/dataprism

27. Reading list
- kappa-architecture.com by Martin Kleppman
- Dataprism whitepaper: https://github.com/dataprism/dataprism-whitepaper

28. questions?