Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP"" Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.

1. Accelerating Innovation
with Apache Kafka
Heikki Nousiainen, CTO, Aiven
Andreas Hechenberger, Just Eat Takeaway.com
Cristian Dugacicu, Just Eat Takeaway.com

2. Heikki Nousiainen
CTO, Aiven
htn@aiven.io
@hnousiainen
Andreas Hechenberger
Team Lead Software Developer, Just Eat Takeaway.com
andreas.hechenberger@justeattakeaway.com
Cristian Dugacicu
Lead Software Developer, Just Eat Takeaway.com
cristian.dugacicu@justeattakeaway.com
@razorcodes
@npoxy

3. Making Developers’ Lives Easier
Data
MONITORING
Data
VISUALIZATION
Data
ANALYSIS
Data
STORAGE
Data
INGESTION
Apache Kafka,
Kafka Connect (25+ connectors)
Stream Processing
Framework
THIS YEAR
Prometheus
Jolokia Aiven metrics
Syslog
THIS
YEAR
Relational Databases
Flink
PostgreSQL MySQL
Non-relational / nosql DB
Redis
Cassandra
Elasticsearch
InfluxDB
Time Series DB
Search Engine
M3
Clickhouse
Grafana
Visualization
Software
Log Integration
Metrics systems
integration
Datadog

4. Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Decoupled Architecture
Kafka centric architecture, breaking down the monoliths.
Independent processing steps bring about faster development cycles,
scalability and resilience.

5. Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Message Content as Contract
Versioned Message Schema as the primary communication contract.
Rapid evolution, versioning with ability to process messages from the
past and the future.
Version transformations and parallel flows.

6. Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Access to & Re-Use of Existing Data
Single source of truth. Always accurate, always complete.
Tap on existing data readily available for experimentation.
Parallel flows, development & verification.
Data Projections.

7. Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Business Agility
Common API. Common architecture stack.
Active community & expertise available.
Rich ecosystem and links to other systems and services.

8. Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Business Continuity
Open Source.
Guaranteed access to your data.
Run everywhere; revisit when necessary.

9. Challenges and Vision
Expansion in new cities
Increasing complexity
Resilience of the reactive services using event driven
architecture
Scalable platform
Build teams around bounded context with a clear
mission and less dependencies
Enable team growth

10. Just Eat Takeaway.com
Logistics Department Scoober

12. Challenges and Vision
Monolith
Restaurants
Couriers
External Partner
...
???

13. Switch to an event driven architecture

14. Why event driven and Why Kafka
Fast data processing ability
Great community support
Supports our believe in Domain Driven Design (DDD)
Scalable in matter of microservices and growth

15. Transformation
Monolith
Restaurants
Couriers
External Partner
...
Routing
Integrations
Disposition-API
Restaurants Courier External Partner ...
Courier-API
Situation
Tracking
LiveOperations FE
Foodtracker FE
...

16. What you need
Healthy working environment to share doubts and ideas
Strong team and mindset for event driven architecture
Convincing management and business for this need
Small steps with every feature (build a habit)
Strong partnership with Kafka provider like Aiven

17. When you have all this
you can do amazing things

18. Realtime analytics with Kafka-Streams
Job
Events
Driver
Events
Driver
Locations
Restaurant
Events
Restaurant
Aggregation
Job
Aggregation
Jobs Before
Fetch Count
Jobs After
Fetch Count
Job Store
Transformer
Driver
Aggregation
Grouped Driver
Aggregation
City Workload
Restaurant
Workload
Final Workload
DynamicDeliveryFee
DriverToRestaurantEta
DriverToCustomerEta
Ranking/Closing
No Duplicates
No Duplicates
No Duplicates
No Duplicates
KTABLE
KTABLE
KTABLE KTABLE
KTABLE
KTABLE
KTABLE
KTABLE
KSTREAM
SCHEDULER
KSTREAM
KSTREAM
KSTREAM
KSTREAM
KSTREAM
KSTREAM
KSTREAM
KSTREAM
K
A
F
K
A
KEY:RESTAURANTID
KEY: JOBID
KEY: DRIVERID KEY: CITY
KEY: CITY
KEY: CITY
KEY: CITY
KEY: CITY
KEY:
RESTAURANTID
KEY: RESTAURANTID
KEY: RESTAURANTID
KEY: RESTAURANTID
KEY: RESTAURANTID
KEY: JOBID
KAFKA TOPIC
KAFKA TOPIC
KAFKA TOPIC
KAFKA TOPIC

19. Q & A

20. Thank you!
Heikki Nousiainen
htn@aiven.io
@hnousiainen
Andreas Hechenberger
andreas.hechenberger@justeattakeaway.com
@npoxy
Cristian Dugacicu
cristian.dugacicu@justeattakeaway.com
@razorcodes
Promotional code: ks2021aiven