@IndeedEnd March: Wednesday, March 27th Video available: http://www.youtube.com/watch?v=MeRHetCMiHg The goal of Indeed's aggregation engine is to find and retrieve every job in the world, as quickly and accurately as possible. As we described in our previous tech talk, we strive to build products that are simple, fast, comprehensive, and relevant. The world's most comprehensive job search site is fueled by the more than 35 million job postings we process every day, which we deliver to jobseekers within minutes of discovery. Our original aggregation architecture was implemented using standard patterns. Our growth required levels of scalability, performance, and resilience this architecture simply could not handle. In a case study of scaling for the web, we will discuss how we tackled this problem. We will cover the issues we saw with our original architecture, how we analyzed our options to guide a solution, how we used RabbitMQ as a key component in the new architecture, and benchmarks to evaluate how successful we were. Speaker Ketan Gangatirkar is the development manager responsible for Indeed's continuous deployment infrastructure as well as its aggregation system. Speaker Cameron Davison is a software engineer on the aggregation team at Indeed and a graduate of UT Austin. He re-architected Indeed's aggregation pipeline using RabbitMQ to sustain high write volumes, and continues to improve products in the aggregation system to make it run more efficiently.

1. How to Get a Job 35 Million Times a Day
Using RabbitMQ
Ketan Gangatirkar and Cameron Davison

2. One search. All jobs.

3. Aggregation gets jobs

4. Aggregation gets jobs so
Jobseekers get jobs

5. Aggregation != Spidering
Spiders see pages.
Aggregation sees jobs.

6. How spiders see job sites
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page

7. How Indeed sees job sites
Start
Job List
Job Job Job
Job List
Job Job Job
Job List
Job Job Job
Navigation Navigation
Job
Job
Job

8. Aggregation != Spidering
Job sites have structure
Job pages have semantics
Navigation is more than following links

10. Remember
this

11. Agg
every
job

13. {
Url: http://www.applytracking.com/track.aspx/3VYzR
Title: Senior Erlang Engineer
Company: Machine Zone
Location: Palo Alto,CA,US, 94301
Source Type: Employer
Job Type: Full-time
...
Description: The Senior Erlang Engineer is an integral ...
...
Createdate: 2013-02-05 23:18:05
...
}
What's in a job

14. location
description
Company
Title

15. Title
salary
location
job type
description
Company

17. How we build products
simple
fast
comprehensive
relevant

18. Simple
Tough problems, simple solutions

19. Fast
Discover the jobs quickly
Get them to jobseekers in minutes

20. 10% of jobseekers sort by date

21. Do you want only new jobs?

22. 20% of jobseekers want only new jobs

23. Daily new job emails

24. Speed matters

25. Comprehensive
Get every job

26. Relevant
Semantic extraction
The job is still available
Ignore non-jobs

27. This is a hard problem
Flaky sites
Site redesigns
Javascript
Missing or bad information

28. Big N makes it even harder
Examine 38M jobs every day

29. Do this in minutes
Search
100M
Jobseekers
Aggregation
Employers
Job Boards
Staffing firms
Recruiters

30. Strawman* architecture
Datacenter B
MySQL
Engine
Datacenter A
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Primary
Datacenter

31. Limitations

32. N connections
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary
Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A

33. N concurrent writers
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary
Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A

34. High latency
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary
Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A

35. Limitation: failure points
Datacenter B
MySQL
Engine
Datacenter A
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Primary
Datacenter
X
X

36. Scaling Patterns
What has worked for us so far?

37. Service-Oriented Architecture
Engine
Engine
Engine
Job Write
Service MySQL
Remote
Datacenter
Primary
Datacenter
see http://go.indeed.com/boxcar

38. Standard Service Interaction
Client Service Database

39. Our Interaction
Client Service Database

40. Does this do what we need?
● Lots of workers...
● Sending lots of results...
● Over a long distance...
● That need to get processed fast...
● Reliably?

41. Engine Failure
Engine
Engine
Engine
Job Write
Service MySQL
Remote
Datacenter
X
Primary
Datacenter

42. Engine failure fix:
Buffer to disk
Engine
Engine
Engine
Job Write
Service MySQL
Remote
Datacenter
disk
disk
disk
Primary
Datacenter
X

43. Network Failure
Engine
Engine
Engine
Job Write
Service MySQL
Remote
Datacenter
X
Primary
Datacenter

44. Network failure fix:
Disks solve that too
Engine
Engine
Engine
Job Write
Service MySQL
Remote
Datacenter
disk
disk
disk
X
Primary
Datacenter

45. Write Service Failure
Job Write
Service MySQL
Remote
Datacenter
X
Engine
Engine
Engine
Primary
Datacenter

46. Write Service Failure fix:
Disks solve that too
Job Write
Service MySQL
Remote
Datacenter
X
Engine
Engine
Engine
Primary
Datacenter
disk
disk
disk

47. Write Service Failure fix:
Redundancy
Job Write
Service
MySQL
Remote
Datacenter
Primary
Datacenter
X
Engine
Engine
Engine
Job Write
Service
Job Write
Service

48. Database Failure
Job Write
Service MySQL
Remote
Datacenter
X
Engine
Engine
Engine
Primary
Datacenter

49. Database Failure fix:
Buffer to disk
Job Write
Service
MySQL
Remote
Datacenter
X
Engine
Engine
Engine
disk
Primary
Datacenter

50. Our new architecture
Job Write
Service
MySQL
Remote
Datacenter
Primary
Datacenter
Engine
Engine
Engine
disk
disk
disk
Job Write
Service
Job Write
Service
disk
disk
disk

51. We could build this...
Job Write
Service
MySQL
Remote
Datacenter
Primary
Datacenter
Engine
Engine
Engine
disk
disk
disk
Job Write
Service
Job Write
Service
disk
disk
disk

52. ... maybe someone already has
Job Write
Service
MySQL
Remote
Datacenter
Primary
Datacenter
Engine
Engine
Engine
disk
disk
disk
Job Write
Service
Job Write
Service
disk
disk
disk

53. We should use a message queue

54. Cameron Davison

55. Aggregation Requirements
● Durable
● Multi-Data Center (latency)
● 38 million jobs a day
● 2KB average job size
○ 76 GB a day
● Target peaks of 1000 jobs / second
● Programming language agnostic

56. Selection

57. What we found
High Availability
Open Source/Free
Self-hosted
Performant

58. Out-of-the-box Experience

59. Advanced Message Queuing
Protocol (AMQP)
● Open Standard
● Wire protocol
● Existing Clients in Multiple Languages

60. Concepts
● Confirmation and Ack
● At least once
● Asynchronous Confirms
● Persistent
● Clustering

61. Confirmation and Ack
MQ
Producer Consumer
m
sg
confirm
ack
m
sg
1
2 3
4

62. At least once
MQ
At most once
Consumer
Message
Ack
MQ
Consumer
Message
Auto Ack

63. Asynchronous Confirms
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Producer
m
essages
confirm #6

64. Persistent
MQ
Producer Consumer

65. Persistent
MQ
Producer Consumer

66. Persistent
MQ
Producer Consumer
X

67. Persistent
MQ
Producer Consumer

68. Persistent
MQ
Producer Consumer

69. Clustering
SlaveMaster
Producer
1
2
3
4

70. Testing

71. Test RabbitMQ
● Send millions of 2KB messages
● 20 producers and 20 consumers
● 1000 messages / second
● Simulate multiple failures

72. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
Slave
Master

73. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
Master
Slave

74. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
Master
Slave

75. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
X
Master

76. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
X
Master

77. Test Consistency
Producers
Rabbit
MQ
Rabbit
MQ
Consumers
Master
Slave

78. RabbitMQ Clustering
Master Slave

79. RabbitMQ Clustering
Master Slave

80. RabbitMQ Clustering
Master
X

81. RabbitMQ Clustering
Master
X

82. RabbitMQ Clustering
MasterSlave

83. RabbitMQ Clustering
MasterSlave

84. RabbitMQ Clustering
MasterSlave

85. RabbitMQ Clustering
MasterSlave

86. RabbitMQ Clustering
MasterSlave

87. RabbitMQ Clustering
MasterSlave

88. RabbitMQ Clustering
Master
X

89. RabbitMQ Clustering
MasterSlave

90. RabbitMQ Clustering
MasterSlave

91. RabbitMQ Clustering
MasterSlave

92. RabbitMQ Clustering
Master
X

93. RabbitMQ Clustering
Master
XX

94. RabbitMQ Clustering
Master
X

95. RabbitMQ Clustering
Master
X

96. RabbitMQ Clustering
Master Slave

97. Non-persistent
15990 Messages / Second
30 MB/s

98. Persistent
2781 Message / Second
5.5 MB/s

99. Clustered and Persistent
1262 Message / Second
2.5 MB/s

100. Applying RabbitMQ

101. Unreliable High Latency
Connections
Engine
Engine
Engine
Job Write
Service
Remote DC Primary DC
MySQL

102. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Remote DC Primary DC
MySQL

103. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Remote DC Primary DC

104. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Remote DC Primary DC

105. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Remote DC Primary DC
Rabbit
MQ

106. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Remote DC Primary DC
Rabbit
MQ

107. Rabbit can talk to Rabbit
Shovel Plugin
Producer RabbitMQ 1 ConsumerRabbitMQ 2

108. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Remote DC Primary DC

109. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Primary DC
Rabbit
MQ
Remote DC

110. Parallelize
Job Write Service
RabbitMQ
Job Write
Service
Job Write
Service
Job Write
Service
Job A
Job B
Job C

111. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Primary DC
Rabbit
MQ
Job Write
Service
Remote DC

112. Replaced with RabbitMQ
Engine
Engine
Engine
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Primary DC
Rabbit
MQ
Job Write
Service

113. Message Flow

114. Message Flow
Engine
Engine
Engine
Job Write
Service
Primary DC
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ

115. Message Flow
Engine
Engine
Engine
Job Write
Service
Primary DC
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ

116. Message Flow
Engine
Engine
Engine
Job Write
Service
Primary DC
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ

117. Message Flow
Engine
Engine
Engine
Job Write
Service
Primary DC
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ

118. Jobs/minute

119. Jobs/minute from one site
220,000 jobs
6 hours
611 jobs / minute

120. Jobs/minute from one site
251,000 jobs
20 minutes
12550 jobs / minute

121. Rabbit
MQ
Horizontal Scale
Engine
Engine
Engine Job Write
ServiceRabbit
MQ
Job Write
Service
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Rabbit
MQ
Job Write
Service
Job Write
Service

122. Horizontal Scale

123. Horizontal Scale

124. Today 1000 messages / second

125. RabbitMQ 3
2486 Message / Second
5MB/s

126. RabbitMQ Configuration
● Confirmations - Fire and Forget
● Persistent Messages - Durable
● Shoveling - Multi-Data Center
● Mirrored Queues in Cluster - High Reliability

127. Can we do more with RabbitMQ?

128. Aggregation Viewer
Real-time browser-based view of job stream

129. ● Almost real-time
● Exclusive queue
● Transient messages
Aggregation Viewer Architecture
Agg Jobs
Rabbit MQ
Cluster
Agg Viewer
Rabbit MQ
Agg
Viewer
Shovel* SubscribeJobs HTTP Browser

130. Resume Contacts Billing
Pay-per-contact: limited budget

132. Resume Contacts Billing
Original Path
P
a
c
i
f
i
c
Asia DC US DC
Log repoResume Search
MySQL
see http://go.indeed.com/logrepo

133. Resume Contacts Billing
Fast Path
P
a
c
i
f
i
c
Asia DC US DC
Rabbit
MQ
MySQL
Log repo
Rabbit
MQ
Resume Search
X

134. Company Page Edits
User-contributed content about companies

135. Company Page

136. Company Page Edits
Implementation
Writing data AND reading it back

137. Company Page Edits
Single Datacenter
Browser
Web Server
MySQL

138. Company Page Serving
Browser
Web Server
LSM Tree
Asia Datacenter
Memcached
see http://go.indeed.com/lsmtree

139. P
a
c
i
f
i
c
Company Page Edits
Browser
Web Server
RabbitMQ RabbitMQ MySQL
Primary US
Datacenter
Asia Datacenter EU Datacenter
A
t
l
a
n
t
i
c
[Et cetera]
Memcached

140. P
a
c
i
f
i
c
Company Page Reads
MySQL
LSM Tree
Builder
LSM Tree
Primary US
Datacenter
Asia Datacenter
LSM Tree
EU Datacenter
A
t
l
a
n
t
i
c
[Et cetera]

141. Memcached
P
a
c
i
f
i
c
Company Pages System
Browser
Web Server
RabbitMQ RabbitMQ MySQL
LSM Tree
Builder
LSM Tree
Primary US
Datacenter
Asia Datacenter
LSM Tree
EU Datacenter
A
t
l
a
n
t
i
c
[Et cetera]

142. Other applications

143. Company Pages

146. Recap: The jobs must flow
● Durability
● High throughput
● Low latency
● Partition-tolerance
● Efficient use of the database
● Minimal points of failure