Presented at All Things Open 2018 Presented by Kevin Xu with MailChimp 10/23/18 - 4:00 PM - Case Study/Demo Track

1. Building Real-Time Search
at Mailchimp
Kevin Xu
Software Engineer

3. 2018
JAN ‘17 OCT ‘17

4. 2018
MC Technical Overview
US1 - US19

5. 2018
SEARCH APP
(downstream)
search queries
indexing requests
(every 5 min)
SQLite Search

6. 2018
SEARCH APP
Problem: Querying
search queries

7. 2018
SEARCH APP
Problem: Indexing
indexing requests
(every ???)

8. 2018
SEARCH APP
search queries
indexing requests
(every ???)
SQLite Search

9. 2018
Considering a Replacement
- Use cases: full-text search,
logging/log analysis, events
and metrics
- Fast queries
- Scale horizontally

10. 2018
Elasticsearch Docs

11. 2018
Querying ES

12. 2018
Capturing Events from MC
US1
US19

13. 2018
Possible Solution: Direct Indexing?

14. 2018
Adding a Message Queue

15. 2018
Real-Time Streaming

16. Proprietary & Confidential 2018 16
Kafka: Topics

17. 2018
Kafka: Partitions

18. 2018
Connecting the Dots

19. 2018
Tracking
Events
All Changes
Searchable
Changes

20. 2018
App-Layer Filtering
config.searchable_model: “Contact”;
config.searchable_model: “Campaign”;
...
$contact = new Contact(“Ben”);
$contact->save(); // onSave()

21. 2018
Write to File, Ship to Kafka

22. 2018
Indexing to ES
PHP Consumers
?

23. 2018
Generating Documents

24. 2018
No Ordering Guarantees!

25. 2018
No Order, No Problem

26. 2018
PHP Consumers

27. 2018
PHP Consumers

28. 2018
Queries > 1s
9/26/17 10/10/17
NumberofQueries
release!

29. 2018
release!
Queries > 2sNumberofQueries
9/26/17 10/10/17

30. 2018 30
250msMedian (p50) Query Response Time

31. 2018 31
400msp95 Query Response Time

32. 2018 32
19Number of Elasticsearch clusters

33. 2018 33
373 billion
Total docs across all ES clusters

34. 2018 34
93,000
Total Changelog events per second (peak)

35. 2018 35
3.7 minutes
Average Time to Index

36. 2018 36
0Support tickets post-launch

37. Proprietary & Confidential 2018 37
What Now?
- Explore other applications of this
infrastructure
- Ongoing technical challenges
- Data Drift, Consumer Lag

38. Proprietary & Confidential 2018
Thanks!
38
kevin.xu@mailchimp.com