How we migrated the HubSpot product from legacy hardware to a Mesos-based Platform-a-a-Service system.

1. Building a PaaS at HubSpot
Tom Petr — @tpetr

2. Blogging
SEO
Social Media
CMS
Lead Management
Landing Pages
Calls-to-Action
Marketing Automation
Email
Analytics
CRM Sync

3. Over 100 engineers
Over 1,600 deployables
~300 daily deploys
From git push to Production
in about 10 minutes

4. Engineers own the end-to-end
success of their products.
They make most tech decisions;
they wear the pagers.
We don’t have an “ops” team.

5. 1. Develop locally
2. Provision QA hardware
3. Deploy via local Python script
4. Provision PROD hardware
5. Deploy via local Python script
…
6. Replace hardware at 4am

6. PaaS is all about
empowering engineers.
Give them good tools
and a solid foundation so they
can focus on what they do best.

7. First Mesos cluster
Sept 2013

8. Abstracts away machines.
Promotes homogenous environment.
Ability to scale out specific processes.
Centralized service registry.

9. Singularity
Oct 2013

10. Web services
Background workers
Cron jobs
One-off tasks

16. Thomass-MacBook-Pro:~ tpetr$ logtail -r BlazarService-web
Fetching tasks
ctrl+c to exit
67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1 HTTP/1.0
67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/ HTTP/1.
67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/c
67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/c
67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/j
67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i
67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/%3C/blaz
67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i
67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i

22. Baragon
Load Balancer API
Nov 2013

23. Amazon ELB
NGINX NGINX NGINX NGINX
Baragon
Singularity

24. QA Migration
Dec 2013 to June 2014

25. x ∞

26. Local filesystem state
singleProcessRunning
Stationary hosts
Memory isolation

27. 💥
💥

28. Max heap
+ (Stack size * expected max threads)
+ (GC overhead pct * max heap)
+ JVM Overhead
+ Extra off-Heap Memory
= No More OOMs!

29. Biggest fear: Inconsistency

30. “My service will be the last to
move to Mesos PROD.”

31. Singularity Baragon hubapi.com Agents
“Add host(s) to hubapi.com”
“Request XX is WAITING”
“Update config”
“Applied successfully”
“Status of request XX?”
“Request XX is WAITING”
“Status of request XX?”
“Request XX is SUCCESS”

32. PROD
QA
✨ TEST ✨

33. Rebranding
with “v2”

34. The Amazon
Rebootpocalypse
Sept 2014

35. PROD Migration
Feb 2015

36. #

37. “But does this all work?”

38. Yes, it powers the
HubSpot product.
(and other companies too!)

39. 8,000,000
tasks launched in 2015

40. No critical situations
caused by infrastructure.

41. PROD Machines
0
200
400
600
800
May 2013 Dec 2013 Jul 2014 Feb 2015
App Servers Mesos Slaves
PROD migrated
Singularity
first deploy

42. Rainmaker Users
0
10
20
30
40
50
May 2013 Dec 2013 Jul 2014 Feb 2015
PROD migrated

43. Wormhole
Client-side load balancing
Sept 2014

44. Avoid hitting load balancers
Preemptively translates URLs
into actual hosts

45. Facsimile
Near-realtime email automation + intelligence
July 2015

46. Persistent IMAP connections
to 200k+ inboxes
Spread over 60 instances

47. Ghidorah
Mesos-based Load Balancers
Jan 2015

48. NGINX + Baragon Agent
in a Docker image
Hosted in Singularity
Absorbed 40 standalone
instances in QA

49. Blazar
An out-of-this-world build system!™
July 2015

50. GitHub Webhooks
Singularity
Heroku-like buildpacks

51. TL;DR
We run our product on Mesos,
and you should too.

52. We invested in a
custom scheduler
because we started early.

53. CTA!

54. Try Singularity today!
github.com/HubSpot/Singularity