Large-scale, near-real-time stream processing with Apache Flink at OfferUp, at Seattle Apache Flink Meetup (01/17/2018)

1. OfferUp Confidential
Large-scale Near-real-time Stream Processing
with Apache Flink @ OfferUp
Bowen Li

2. User Survey
● Who has used OfferUp?
● Who has used Apache Flink?
● Who has developed code in Apache Flink?

3. OfferUp, create the simplest, most trustworthy way to buy and sell locally
At a Glance
● the largest mobile marketplace for local
buyers and sellers in the U.S.
● Top shopping app on iOS and Android
● $14+ Billion In Transactions in 2016

4. Speaker Background
Bowen Li
○ Offerup
■ Develop stream processing infra with Apache Flink
■ Apache Airflow, Apache Avro, etc
○ Tableau
■ Lots of Apache ZooKeeper and Apache Curator

5. Stream Processing @ OfferUp
We are expanding our stream processing footprint.
We developed OfferUp’s stream processing platform with a few primitives:
● Flink installation on EMR
● HDFS, YARN, metrics, checkpoint/savepoint, etc. configuration help for Flink cluster
● User apps deployment
● Connecting to streams
● Data Ser/Deser

6. Use Case
Business Requirement:
● Calculate time-decaying personalization scores based on user activity within
the last month

7. Use Case - The old pipeline
The old pipeline:
● batch processing
● ~3 hours of end-to-end latency

8. Use Case - The new near-real-time pipeline!

9. Pipeline Stats
● Data Volume: processing billions of records per day
● Average end-to-end latency: ~1 min
○ The 2min aggregation in 1st Flink cluster (NRT scores) dominates the latency
○ Depending on when the event enters the 2min window, the minimum latency of
this pipeline can be a few seconds, the expected maximum is ~2min
We dramatically lowered the end-to-end latency from ~3h to ~1min!

10. Design Considerations Explained
● Latency - why 2min aggregation
○ User-facing services will only batch-read new scores every 5min
○ Any latency smaller than 5min is good

11. Design Considerations Explained (con’t)
● Data Correctness
○ at-least-once guarantee
○ Be careful with merging NRT scores with historical scores, ensure no overlap and
no gap

12. Design Considerations Explained (con’t)
● Failure Recovery (assuming AWS Kinesis are reliable)
○ Two Flink clusters have checkpointing enabled and they can auto recover
○ Redis data has 3 day TTL, enough time to fix things up
○ Other parts are stateless

13. Design Considerations Explained (con’t)
● Replay Capability
○ Each component can handle data load of replay, and its latency is not greatly
impacted

14. Contribution to Apache Flink
I’ve been contributing to Apache Flink since Mar 2017

15. Related Contributions to Apache Flink
● FLINK-7508 Improved flink-connector-kinesis write performance by 10+X
○ Released version 1.3.2
○ Many other comprehensive improvements of flink-connector-kinesis
○ 13 out of my 43 commits
● FLINK-7475 Improved Flink’s ListState APIs() and boost its performance by 15~35X
○ Will be released in version 1.5.0
● FLINK-6013 Created flink-metrics-datadog module
● Other contributions include Flink’s DataStream APIs, side output, build system, etc

16. ● Problem: flink-connector-kinesis used to create one http connection for each request
● Improvement: Switched it to a connection pool mode
● Result: Improved flink-connector-kinesis’s write performance by 10+X
Contribution: Improved flink-connector-kinesis
# Records Sent
# Records Pending in Client
In this basic test, you can tell
1) write throughput goes up
2) # pending records has dropped significantly

17. Benchmarking:
● Running a Flink hourly-sliding windowing job
● Enough Kinesis shards
● ~70 bytes/record
Limitations: My Flink job is not developed for benchmarking. It only generates 21million records at maximum, which gives us
a 10X or more improvement estimate. In reality, the perf improvement should be more than 10X. You’re welcome to do your
own benchmarking
Contribution: Improved flink-connector-kinesis

18. Contribution: Improved Flink’s ListState performance 20~35X
Background on RocksDBStateBackend
Problems:
● ListState has only two APIs - add() and get()
● RocksDBListState translate add() as RocksDB.merge()
○ adding 100 elements takes 100 memtable write, very slow….
Improvements:
● Developed two new APIs in Flink 1.5 - update() and addAll()
● update() and addAll() will both simulate RocksDB’s byte merge operation, pre-merge all elements
upfront and write to memtable only once

19. Benchmarking added to source code: org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest
Result: 15 ~35X faster!

20. Q&A
Thank you!
We’re hiring!