Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2NLFuB7. Katerina Domenikou talks about how her team built the Bloomberg real-time alerting platform using open source search technology, and explores the challenges that arise at that scale. She also talks about the Bloomberg News Search ecosystem. Filmed at qconnewyork.com. Katerina Domenikou joined Bloomberg in 2010 as a software developer and has worked for six years on developing a high-capacity trading platform. In 2017, she joined the News Infrastructure Engineering Team. Her passion is to make people excited about tech, and she has participated in several panel discussions regarding technology and diversity and inclusion.

1. © 2018 Bloomberg Finance L.P. All rights reserved.
Closer to the Wire:
Real-time News Alerting @Bloomberg
Katerina Domenikou
Senior Software Engineer
@kat_in_gloves

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
news-alerting-bloomberg

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

5. Source: instagram/@elonmusk

6. NEWS MOVES MARKETS

9. Bloomberg

10. Bloomberg
Source: www.pinkcakebox.com

11. Agenda
• System Requirements
• Architecture Overview #birdseye
• Matcher Application #deepdive

12. Agenda
• System Requirements
• Architecture Overview #birdseye
• Matcher Application #deepdive

14. News Search Ecosystem
Suggest
Understand
Search Sort
Alert Expose facetsRecommend
Deliver

15. The Document
Elon Musk
Tesla Inc
Lyft Inc
Technology
Automobile Industry
United States (US)
Transportation
Gaby Toledano

16. NEWS ON..

17. The Query

18. Key Statistics

19. Flexibility A.K.A Complexity
125K+ sources 40 languages
Queries with
1000s of companies

20. Agenda
• System Requirements
• Architecture Overview #birdseye
• Matcher Application #deepdive

21. System Overview

22. Pipeline

23. Enrichment

24. Pipeline

25. Deliveries

26. Pipeline

27. Even Faster Alerts

28. Agenda
• System Requirements
• Architecture Overview #birdseye
• Matcher Application #deepdive

29. Matcher

30. Matcher

31. Search

32. Streaming Search

33. Streaming Search

34. (Not efficient) Streaming Search

35. Luwak

36. “ ”
Matching
AND
AND
Q1:
Q2:
ORQ3:
Stored Queries
“ ”
“ ”
Incoming News Stories

37. Query Pre-Processing
AND
AND
Q1:
Q2:
ORQ3: ,
Stored Queries
Term-frequency analysis

38. Query Pre-Processing
AND
AND
Q1:
Q2:
ORQ3: ,
Q3
Q1
Q2
Q3
Stored Queries Stored Query Index

39. Document Disjunction
“ ” “ ” “ ”“ ”
Q3
Q1
Q2
Q3
OR OR
Candidate Queries
Stored Query Index
Can’t match

40. Query Matching
“ ” “ ” “ ”OR OR
AND
AND
Q1:
Q2:
ORQ3:

41. Matcher Application
Query Shard
Document
Query Updates

42. Sharding

43. QC: Mirror System

44. Agenda
• System Requirements
• Architecture Overview #birdseye
• Matcher Application #deepdive

45. References
• Real-time full-text search with Luwak and Samza http://www.flax.co.uk/
blog/2015/08/26/real-time-full-text-search-with-luwak-and-samza/
• Turning search upside down: search for queries with documents
https://berlinbuzzwords.de/14/session/turning-search-upside-down-search-
queries-documents

46. © 2018 Bloomberg Finance L.P. All rights reserved.
Katerina Domenikou
Senior Software Engineer, Bloomberg
@kat_in_gloves
Thank you

47. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/news-
alerting-bloomberg