In this talk we’ll describe the evolution of stream processing at Tesla and the challenges that are specific to our needs, such as large skews in message-processing latencies. We’ll describe how we built a reliable and performant ingestion platform that allows us to take an idea from a whiteboard to production in just a matter of hours. We’ll also discuss the design principles, tools, and incident response processes that have enabled a small team to support Kafka and downstream services in highly-available and multi-tenant environments at scale.

1. From a Million to a Trillion
Events Per Day: Stream
Processing in Ludicrous Mode
Shrijeet Paliwal
@shrijeet
Kafka Summit NYC 2019

2. Tesla fleet

3. The quirks
•Physical system constraints limited resources, prolong
disconnectivity
•Skewed processing latencies seconds to hours
•Payload explosion 10000x

4. Payload explosion
300/sec 3M/sec

5. 100s powerpacks
10s pods/pack
1000s signals/sec/pod
Dimensions of growth

6. Ingest - the evolution
2015
Data-
Pump
- Full fleet
- Big messages
- 5M/day
2017
Kafka
Connect
- Canonical messages
- 50B/day
2012
- Subset of the fleet
- 50K/day
2018
- 500B/day
- Prolific
adaption

7. Ingest - needs & wants
•One stream One app vastly simpler operationally
•Isolated dependencies, resources, ownership
•Micro batching align flushes with commits
•Scaling through multiple degrees of freedom

8. - Supreme composability
- Minimal and consistent APIs
- Back-pressure, buffering, transformations, failure recovery
Image source: https://www.slideshare.net/kpciesielski/reactive-kafka-with-akka-streams
Kafka Channels

9. Avro - you say?

10. Expectation Reality

11. Raw to Canonical channel
FlowSource Sink
{ k : v }

12. Filter
Broadcast
FileSystem
Batch
Source
Database
Kafka Channel building blocks
KafkaToKafka
KafkaToDB
KafkaToFS

13. - Implement parser
- Define schema
- Configure the pipe
From whiteboard to production

14. •Kafka is very simple to monitor & observe
•Kafka dashboard(s) - most valuable self service tool
•Collect (all) > Plot (some) > Alert (start with one!)
[de]Cluttered observability

15. Start with one alert!
ProducerSource Sink
x msg/sec y msg/sec
Kafka Availability SLI

16. •How fresh is my data?
•What is the ETA for lag recovery?
•Can we rollback the stream to 2pm?
…towards time based tooling!
Moving off from the offsets…

17. How fresh is my data?

18. Help me, Help you!
on-boarding checklist
Size compression, schema
Throughput msg/sec, req/sec
Rebalance rate gc, process latency
Resources cpu, memory, network

20. Dry-ops profiling
Disables all
external calls!

21. Generation Storage Consumption
Accelerate the world’s transition to
sustainable energy
Thank you!