Video available at: http://www.youtube.com/watch?v=E1ok08TVxDw Indeed's flagship job search product has evolved over the years to meet new challenges. It began as a single, monolithic web application. This grew larger and increasingly complex as we built new features. To remedy this growing problem, we implemented a service-oriented architecture to improve system availability, scalability, and maintainability. We examined common practices for service-oriented architectures, and we discovered ways to improve on the state of the art. We developed these ideas into a new framework called Boxcar. In this talk, we will discuss the scaling problems we solved, the innovative ideas behind boxcar, and how we built the scalable architecture that we now use throughout our systems. R.B. Boyer is a Software Engineer who has been with Indeed since late 2007. Over the years he has worked on a variety of projects, including distributed storage, authentication, and service architectures.

1. Boxcar
A self-balancing distributed services protocol

2. R.B. Boyer
Software Engineer
Resume

3. I help
people
get jobs.

4. I solve
interesting
problems.

5. Boxcar was the solution
to a problem:

6. Building

7. How we build products
Simple
Fast
Comprehensive
Relevant

8. How we build products
Simple
Fast
Comprehensive
Relevant

9. How we build systems
Simple
Fast
Resilient
Scalable

10. Simple
“I want my application to be
more complicated”
- No one ever

11. Complexity creates
confusion

12. Complexity creates
confusion
Confusion breeds bugs

13. Fast
“I want my application to be
slower”
- No one ever

14. conducted a speed test

15. +500 milliseconds of latency
per search

16. 20% fewer
searches

17. Speed is a feature

18. Resilient
“I want my users to
experience outages”
- No one ever

19. Programs crash

20. Programs crash
Machines die

21. Minimize vulnerability to
any failure

22. Scalable
“My system will only need to
support 10 users”
- No one ever

23. Scale with MORE machines

24. Scale with MORE machines
Not BIGGER machines

25. TL;DR:

27. Indeed
Jobs
Sites
Job
Seekers

28. Aggregation
Jobs
Sites
Job Search
Job
Seekers

29. Job Search
Aggregation

30. Challenge!
Job Search
Aggregation

31. Challenge:
keep this
Simple
Fast
Resilient
Scalable

32. Options:

33. Share data access?

34. Example:
Shared Database

35. Shared Database
Main
Database

36. Shared Database
Main
Application
Main
Database

37. Shared Database
Main
Application
Analysis
Tool
Main
Database

38. Shared Database
Main
Application
Analysis
Tool
Billing
Application
Main
Database

39. Shared Database
Main
Application
Main
Database
Analysis
Tool
Billing
Application
Intern
Project

40. Shared Database
Main
Application
Main
Database
Analysis
Tool
Billing
Application
Intern
Project
Other
Intern
Project

41. Shared Database
Main
Application
Main
Database
Analysis
Tool
Billing
Application
Intern
Project
Other
Intern
Project
Email
Tool

42. Shared Database
Main
Application
Main
Database
Analysis
Tool
This is an anti-pattern
Billing
Application
Intern
Project
Other
Intern
Project
Email
Tool

43. On a long enough timeline...

44. Maintenance Nightmare

45. Share data access

46. Share data access
Insulate data from
consumers

47. Shared Database
Main
Application
Main
Database
Analysis
Tool
Billing
Application
Intern
Project
Other
Intern
Project
Email
Tool

48. Insulated Database
Main
Application
Main
Database
Main
Service
Analysis
Tool
Billing
Application
Intern
Project
Other
Intern
Project
Email
Tool

49. Service?

50. Service
Client
Client
Client
Client

51. Client
Client
Client
Client
NETWORK
Service

52. Client
Client
Client
Client
NETWORK
Service
Icky
Technical
Stuff

53. Service
NETWORK
Databases
Client
Client
Client
Client
Logging
Caches
Business Logic
...
Client API
Icky Technical
Details

54. Client API
Service.getJobs([12345, 62])

55. Icky Technical Details
SELECT * FROM jobs AS j
LEFT JOIN companyinfo AS ci
ON j.id=ci.job_id
LEFT JOIN locations AS loc
ON loc.id=j.location_id
WHERE j.id IN (12345, 62)

56. Service
Oriented
Architecture

57. Service
Oriented
Architecture

59. Boxcar

60. Boxcar is a...
self-balancing
distributed
services
protocol

61. Origin Story

62. There was a life before
Boxcar

63. There were services before
Boxcar

64. Pick one:

65. Doc Service

66. Document Serving Service
aka “Doc Service”
http://go.indeed.com/docservice

67. Doc Service controls
access to JOBS

68. Building Blocks
Webapp
Wants jobs
Doc
Service
Controls access to jobs
Docstore
Stores jobs

69. Build it

70. Webapp

71. Webapp
Doc
Service
Docstore

72. Webapp
Doc
Service
Docstore

73. Mission Accomplished
Webapp
Doc
Service
Docstore

74. But is it good?

75. How we build systems
Simple
Fast
Resilient
Scalable

76. Goodness Metric
Simple deploys
Efficient networking (Fast)
Resilient
Horizontally scalable

77. Webapp
Doc
Service
Docstore

78. Webapp
Doc
Service
Docstore

79. ✘ Resilient
Webapp
Doc
Service
Docstore

80. Add Resilience

81. Webapp
Doc
Service
Docstore

82. Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

83. Front-end Load Balancer
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

84. Siloed Stacks
Front-end Load Balancer
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

85. Siloed Stacks
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
?
Horizontally scalable

86. Scaling Silos
Front-end Load Balancer
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

87. Scaling Silos
Front-end Load Balancer
Webapp
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

88. Scaling Silos
Front-end Load Balancer
Webapp
Webapp
Webapp
Doc
Service
Doc
Service
Doc
Service
Docstore
Docstore
Docstore

89. Need bigger and
bigger machines

90. Vertical Scaling

91. Siloed Stacks
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✘ Horizontally scalable

92. Siloed Stacks
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✘ Horizontally scalable
Services
Version 1

93. Improve scalability

94. Front-end Load Balancer
Webapp
Webapp
Webapp
Doc
Service
Doc
Service
Doc
Service
Docstore
Docstore
Docstore

95. Front-end Load Balancer
Webapp
Webapp
Webapp
Doc Service Load Balancer
Doc
Service
Doc
Service
Docstore
Docstore

96. Per-Service Balancer
Front-end Load Balancer
Webapp
Webapp
Webapp
Doc Service Load Balancer
Doc
Service
Doc
Service
Docstore
Docstore

97. Per-Service Balancer
~
Simple deploys
?
Efficient networking (Fast)
?
Resilient
✓ Horizontally scalable

98. Proxying isn’t free
✘2x Bandwidth
Webapp
Doc Service Load Balancer
Doc
Service

99. Per-Service Balancer
~
Simple deploys
✘ Efficient networking (Fast)
?
Resilient
✓ Horizontally scalable

100. Resilience
Front-end Load Balancer
Webapp
Webapp
Webapp
SINGLE POINT OF FAILURE
Doc
Service
Doc
Service
Docstore
Docstore

101. Need two balancers

102. Need two balancers
...and a way to balance
between them?

103. Load Balancer Balancing
Master / Slave
Share IP address
Heartbeat between nodes
Complex

104. Resilience
Front-end Load Balancer
Webapp
Webapp
Webapp
Doc Service Load Balancer
Doc Service Load Balancer
Doc
Service
Doc
Service
Docstore
Docstore

106. Best explained by our
Operations folks:
“Redundant Array of
Inexpensive Datacenters”
http://go.indeed.com/raid

107. Per-Service Balancer
~
Simple deploys
✘ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable

108. Per-Service Balancer
~
Simple deploys
✘ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
Services
Version 2

109. Reduce network waste

110. Front-end Load Balancer
Webapp
Webapp
Webapp
Doc Service Load Balancer
Doc
Service
Doc
Service
Docstore
Docstore

111. Front-end Load Balancer
Webapp
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

112. Naive Round Robin
Front-end Load Balancer
Webapp
Webapp
Webapp
Doc
Service
Doc
Service
Docstore
Docstore

113. Naive Round Robin
✓ Simple deploys
?
Efficient networking (Fast)
?
Resilient
✓ Horizontally scalable

114. Direct Connections
Webapp
✓1x Bandwidth
Doc
Service

115. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
?
Resilient
✓ Horizontally scalable

116. Server A
Server B

117. Server A
Server B

118. REQUEST
Server A
Server B

119. ✘
REQUEST
Server A
Server B

120. ✘
REQUEST
Server A
Server B

121. REQUEST
Server A
Server B

122. Server A
Server B
REQUEST

123. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable

124. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable

125. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
?
Balanced

126. Slow
Fast

127. Slow
Fast

128. Slow
Fast

129. Slow
Fast

130. Slow
Fast

131. Slow
Fast

132. Slow
Fast

133. Slow
Fast

134. Slow
Fast

135. Slow
Fast

136. Slow
Fast

137. Can’t keep
up
Slow
Fast

138. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
✘ Balanced

139. Naive Round Robin
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
✘ Balanced
NOPE

140. Ensure balance

141. Front-end Load Balancer
Webapp
Webapp
Webapp
Doc Service Load Balancer
Doc
Service
Doc
Service
Docstore
Docstore

142. Front-end Load Balancer
Webapp
Webapp
Webapp
Distribute!
Doc
Service
Doc
Service
Docstore
Docstore

143. Front-end Load Balancer
Webapp
Webapp
B
Webapp
B
B
Doc
Service
Doc
Service
Docstore
Docstore

144. Front-end Load Balancer
Web
App
Web
App
B
Web
App
B
Doc
Service
Doc
Service
Docstore
Docstore
B

145. Boxcar
Front-end Load Balancer
Web
App
Web
App
B
Web
App
B
Doc
Service
Doc
Service
Docstore
Docstore
B

146. Naive Round
Robin
Per-Service
Balancer

147. The Boxcar balancing
algorithm is simple

148. Gist
Servers assign numeric value to
connections
Clients use the connection with the
lowest numeric value to service each
request

149. Server A

150. Server A
Server A

151. Server A
Slot 0
Slot 1
Slot 2
Slot 3
Slot 4
...

152. Slot Numbers
Just numbers
Server A
Slot 0
No limit
Slot 1
Slot 2
NOT a priority
Slot 3
Slot 4
...
ONLY used for balancing

153. LOW slot numbers
are the
BEST slot numbers

154. Server A
Slot 0
USED
Slot 1
He
llo
!
USED
Slot 2
Client 2
Slot 3
Slot 4
USED
...

155. Server A
Slot 0
USED
Slot 1
USED
Slot 2
USED
Slot 3
Slot 4
USED
Slot 2
Client 2
...

156. Client 2

157. Client 2

158. Client 2

159. Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
Slot 57
B

160. long-lived
connections
Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
Slot 57
B

161. Clients are greedy
MINE!
50
2

162. Clients are greedy
Want best connections
MINE!
50
2
Continually look for better
connections
Close worst connections

163. Background thread maintains
the
connection pool

164. Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
Slot 57
B

165. Slot
17
Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
Slot 57
B

166. Slot
17
Server
Client 2
Server
Slot 0
Slot 2
A
B
Slot 12
Slot 29
Slot 30
✘
Slot 57
✘

167. Slot
17
Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
B

168. Slot
17
Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 29
Slot 30
B

169. Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 17
Slot 29
Slot 30
B

170. Server
Client 2
Server
Slot 0
Slot 2
A
Slot 12
Slot 17
Slot 29
Slot 30
Continues forever
B

171. Incoming Requests
Client 2
Slot 0
ACTIVE
Slot 2
ACTIVE
Slot 12
[idle]
Slot 29
ACTIVE
Slot 30
[idle]
Slot 57
[idle]
GetJobs()

172. Incoming Requests
Client 2
Slot 0
ACTIVE
Slot 2
ACTIVE
Slot 12
ACTIVE
Slot 29
ACTIVE
Slot 30
[idle]
Slot 57
[idle]
GetJobs()

173. Connections NOT
established on-demand

174. Requests to Busy Pool
Client 2
Slot 0
ACTIVE
Slot 2
ACTIVE
Slot 12
ACTIVE
Slot 29
ACTIVE
Slot 30
ACTIVE
Slot 57
ACTIVE
GetJobs()

175. Requests to Busy Pool
Client 2
Slot 0
ACTIVE
Slot 2
ACTIVE
Slot 12
ACTIVE
Slot 29
ACTIVE
Slot 30
ACTIVE
Slot 57
ACTIVE
✘
GetJobs()
ERROR!

176. Sizing the pool properly is
imperative!

177. Gist Redux
Servers assign numeric value to
connections
Clients use the connection with the
lowest numeric value to service each
request

178. Balanced load is
emergent behavior

179. Load Balancing
Simulations

180. Server A
Server B
Client X

181. Server A
slot 0
Server B
Client X
0

182. Server A
slot 0
Server B
Client X
0

183. Server A
slot 0
Client X
0
0
Server B
slot 0

184. Server A
slot 0
Server B
slot 0
Client X
0
0

185. Server A
slot 0
slot 1
1
1
Server B
slot 0
slot 1
Client X
0
0

186. Server A
slot 0
Client X
0
0
Server B
slot 0
Steady-state balance

187. Server A
slot 0
Client X
0
0
New Clients Join
Server B
slot 0

188. Server A
slot 0
Client X
0
0
Client Y
Server B
slot 0

189. Server A
slot 0
slot 1
Client X
0
0
1
1
Server B
slot 0
slot 1
Client Y

190. Server A
slot 0
slot 1
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1

191. Server A
slot 0
slot 1
slot 2
Client X
0
0
2
Server B
slot 0
slot 1
slot 2
2
Client Y
1
1

192. Server A
slot 0
slot 1
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1

193. Server A
slot 0
slot 1
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1
Client Z

194. Server A
slot 0
slot 1
slot 2
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1
slot 2
2
2
Client Z

195. Server A
slot 0
slot 1
slot 2
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
2
2

196. Server A
slot 0
slot 1
slot 2
slot 3
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1
slot 2
slot 3
3
3
Client Z
2
2

197. Server A
slot 0
slot 1
slot 2
Client X
0
0
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
2
2
Steady-state balance

198. Server A
slot 0
slot 1
slot 2
Client X
0
0
Server Failure
Server B
slot 0
slot 1
slot 2
Client Y
1
1
Client Z
2
2

199. Server A
slot 0
slot 1
slot 2
Client X
0
Client Y
1
Server B
slot 0
Client Z
2

200. Server A
slot 0
slot 1
slot 2
Client X
0
Client Y
1
Server B
Client Z
2

201. Server A
slot 0
slot 1
slot 2
slot 3
Client X
0
Client Y
1
Server B
3
Client Z
2

202. Server A
slot 0
slot 1
slot 2
slot 3
Client X
0
Client Y
1
Server B
Client Z
2
3

203. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
Client X
0
4
Client Y
1
Server B
Client Z
2
3

204. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
Client X
0
4
Client Y
1
Server B
Client Z
2
3

205. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
5
Client Y
1
Server B
Client Z
2
3

206. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
Server B
Client Z
2
3

207. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
Server B
Client Z
2
3
Steady-state balance

208. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
Server B
Client Z
2
3
Steady-state balance

209. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Server Restored
Client Y
1
5
Server B
Client Z
2
3

210. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
Server B
Client Z
2
3

211. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
Server B
slot 0
0
Client Z
2
3

212. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Server B
slot 0
Client X
0
4
Client Y
1
5
Client Z
2
0<3

213. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Server B
slot 0
Client X
0
4
Client Y
1
5
Client Z
2
0

214. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Server B
slot 0
Client X
0
4
Client Y
1
5
Client Z
2
0

215. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Server B
slot 0
Client X
0
4
Client Y
1
5
Client Z
0
2

216. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Client X
0
4
Client Y
1
5
1
Server B
slot 0
slot 1
Client Z
0
2

217. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
slot 5
Server B
slot 0
slot 1
Client X
0
4
Client Y
1
1<5
Client Z
0
2

218. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
Client X
0
4
Client Y
1
1
Server B
slot 0
slot 1
Client Z
0
2

219. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
Client X
0
4
2
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
0
2

220. Server A
slot 0
slot 1
slot 2
slot 3
slot 4
Client X
0
2<4
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
0
2

221. Server A
slot 0
slot 1
slot 2
Client X
0
2
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
0
2

222. Server A
slot 0
slot 1
slot 2
Client X
0
2
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
0
2
Steady-state balance

223. Server A
slot 0
slot 1
slot 2
Client X
0
2
Client Shutdown
Server B
slot 0
slot 1
slot 2
Client Y
1
1
Client Z
0
2

224. Server A
slot 0
slot 1
slot 2
Client X
0
2
Client Y
1
1
Server B
slot 0
slot 1
slot 2
Client Z
0
2

225. Server A
slot 0
slot 1
slot 2
Server B
slot 0
slot 1
slot 2
Client X
0
2
Client Z
0
2

226. Server A
slot 0
slot 1
slot 2
Server B
slot 0
slot 1
slot 2
Client X
0
2
1
Client Z
0
2

227. Server A
slot 0
slot 1
slot 2
Server B
slot 0
slot 1
slot 2
Client X
0
2
Client Z
0
1<2

228. Server A
slot 0
Server B
slot 0
slot 1
slot 2
Client X
0
2
Client Z
0
1

229. Server A
slot 0
slot 1
Server B
slot 0
slot 1
slot 2
1
Client X
0
2
Client Z
0
1

230. Server A
slot 0
slot 1
Client X
0
1<2
Server B
slot 0
slot 1
slot 2
Client Z
0
1

231. Server A
slot 0
slot 1
Server B
slot 0
slot 1
Client X
0
1
Client Z
0
1

232. Server A
slot 0
slot 1
Client X
0
1
Server B
slot 0
slot 1
Client Z
0
1
Steady-state balance

233. Server A
slot 0
slot 1
Client X
0
1
Client Rejoins
Server B
slot 0
slot 1
Client Z
0
1

234. Server A
slot 0
slot 1
Client X
0
1
Client Y
Server B
slot 0
slot 1
Client Z
0
1

235. Server A
slot 0
slot 1
slot 2
Client X
0
1
2
Client Y
2
Server B
slot 0
slot 1
slot 2
Client Z
0
1

236. Server A
slot 0
slot 1
slot 2
Client X
0
1
Client Y
2
2
Server B
slot 0
slot 1
slot 2
Client Z
0
1

237. Server A
slot 0
slot 1
slot 2
Client X
0
1
Client Y
2
2
Server B
slot 0
slot 1
slot 2
Client Z
0
1
Steady-state balance

238. Why does this Balance?

239. Connections are like
running water
seeking lower ground

240. Slots
Connections
Servers

241. Slots
Connections
Servers

242. Slots
Connections
Servers

243. Slots
Connections
Servers

244. Slots
Connections
Servers

245. Slots
Connections
Servers

246. Slots
Connections
Servers

247. Slots
Connections
Servers

248. Slots
Connections
Servers

249. Slots
Connections
Servers

250. Slots
Connections
Servers

251. Slots
Connections
Servers

252. Roughly Equal
Distribution
Slots
Connections
Servers

253. Edge cases

254. Server A
slot 0
slot 1
Server B
slot 0
slot 1
Client X
0
1
Client Z
0
1

255. Server A
slot 0
slot 1
Client X
0
1
Balanced
but not ideal
Server B
slot 0
slot 1
Client Z
0
1

256. Server A
slot 0
slot 1
Client X
0
1
Server B
slot 0
Client Z

257. Server A
slot 0
slot 1
Client X
0
1
Server B
slot 0
Client Z
EMPTY
POOL!

258. Server A
slot 0
slot 1
Client X
0
1
✘ Resilient
Server B
slot 0
EMPTY
POOL!
Client Z

259. Fix by adding entropy

260. Fix by adding entropy
aka “Table Shaking”

261. Table Shaking
Servers regularly hang up on connections

262. Table Shaking
Servers regularly hang up on connections
Clients expect failed connections

263. Table Shaking
Servers regularly hang up on connections
Clients expect failed connections
Failures are retried on new connections

264. Table Shaking
Servers regularly hang up on connections
Clients expect failed connections
Failures are retried on new connections
Bad configurations are less likely

265. Server A
slot 0
slot 1
Client X
0
1
Table Shaking
turns this
Server B
slot 0
slot 1
Client Z
0
1

266. Server A
slot 0
slot 1
Client X
0
1
Into this
Server B
slot 0
slot 1
Client Z
0
1

267. Server A
slot 0
slot 1
Server B
slot 0
Client X
0
Client Z
1

268. Server A
slot 0
slot 1
Client X
0
YAY!
YAY!
Server B
slot 0
Client Z
1

269. Balancing Tricks:
Handicapping

270. Handicapping is
Server Self-quarantine

271. Handicapping
Exploit slot number assignment

272. Handicapping
Exploit slot number assignment
Unhealthy servers inflate slot numbers

273. Handicapping
Exploit slot number assignment
Unhealthy servers inflate slot numbers
Clients naturally avoid these servers

274. Slots
Connections
Servers

275. Unhealthy
Slots
Connections
Servers

276. Slots
Connections
Servers

277. Slots
Connections
Servers

278. Slots
Connections
Servers

279. Unhealthy
Slots
Connections
Servers

280. Slots
Connections
Servers

281. Slots
Connections
Servers

282. Slots
Connections
Servers
graceful degradation

283. Is Boxcar good?

284. Boxcar
✓ Simple deploys
✓ Efficient networking (Fast)
?
Resilient
✓ Horizontally scalable
?
Balanced

285. Clients are pessimistic

286. Clients are pessimistic
Failure is expected

287. Boxcar
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
?
Balanced

289. Balance Connections
Not Requests

290. Balancing Review:
Naive Round Robin

291. Slow
Fast

292. Slow
Fast

293. Slow
Fast

294. Slow
Fast

295. Slow
Fast

296. Slow
Fast

297. Slow
Fast

298. Slow
Fast

299. Slow
Fast

300. Slow
Fast

301. Slow
Fast

302. Can’t keep
up
Slow
Fast

303. The problem was that
requests (connections)
piled up

304. Boxcar has a fixed number of
connections

305. Boxcar has a fixed number of
connections
there’s nothing to pile up

306. Slow
Server
Fast
Server

307. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

308. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

309. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

310. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

311. Client
7 9
0 1
Slot 7
Slot 9
Slow
Server
Fast
Server

312. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

313. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

314. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

315. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

316. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

317. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

318. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

319. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

320. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

321. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

322. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

323. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

324. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

325. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

326. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

327. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

328. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

329. Client
7 9
Slot 7
Slot 9
Slow
Server
Fast
Server

330. Client
7 9
2 requests
4 requests
Slot 7
Slot 9
Slow
Server
Fast
Server

331. Slow servers handle fewer
requests

332. No overloaded servers

333. All requests are serviced

334. Load balancing is
probabilistic

335. Boxcar
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
✓ Balanced

336. Boxcar
✓ Simple deploys
✓ Efficient networking (Fast)
✓ Resilient
✓ Horizontally scalable
✓ Balanced

337. Good enough for Indeed

338. Services well over a
BILLION requests
every day

339. Fundamental technology

340. Powering over 20
different services

341. In production since 2009

343. Service
Oriented
Architecture

344. Q&A