Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1AN8vD7. Randy Shoup, CTO of Kixeye, shares the history of reactive services and visits key milestones in their evolution: async techniques at Google, real-time search at Ebay, & responsive games at Kixeye. Filmed at qconsf.com. As CTO, Randy Shoup brings 25 years of engineering leadership to KIXEYE, ranging from tiny startups to Internet-scale companies.

1. Concurrency at Scale:
The Evolution to Micro-­‐‑Services
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/reactive-services-scale

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Background
• CTO at KIXEYE
o Real-time strategy games for web and mobile
• Director of Engineering for Google App Engine
o World’s largest Platform-as-a-Service
o Part of Google Cloud Platform
• Chief Engineer at eBay
o Multiple generations of eBay’s real-time search infrastructure

5. Evolution in Action
• eBay
• 5th generation today
• Monolithic Perl à Monolithic C++ à Java à microservices
• Twitter
• 3rd generation today
• Monolithic Rails à JS / Rails / Scala à microservices
• Amazon
• Nth generation today
• Monolithic C++ à Perl / C++ à Java / Scala à microservices

6. Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

7. Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

8. The Monolithic
Architecture
2-3 monolithic tiers
• {JS, iOS, Android}
• {PHP, Ruby, Python}
• {MySQL, Mongo}
Presentation
Application
Database

9. The Monolithic
Application
Simple at first
In-­‐‑process latencies
Single codebase, deploy unit
Resource-­‐‑efficient at small scale
Pros
Coordination overhead as team
grows
Poor enforcement of modularity
Poor scaling (vertical only)
All-­‐‑or-­‐‑nothing deploy (downtime,
failures)
Long build times
Cons

10. The Monolithic
Database
Simple at first
Join queries are easy
Single schema, deployment
Resource-­‐‑efficient at small scale
Pros
Coupling over time
Poor scaling and redundancy (all-­‐‑
or-­‐‑nothing, vertical only)
Difficult to tune properly
All-­‐‑or-­‐‑nothing schema
management
Cons

11. “If you don’t end up regrePing
your early technology
decisions, you probably over-­‐‑
engineered”
-- me

12. Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

13. Micro-­‐‑Services
“Loosely-­‐‑coupled service
oriented architecture with
bounded contexts”
-- Adrian Cockcroft

14. Micro-­‐‑Services
“Loosely-­‐‑coupled service
oriented architecture with
bounded contexts”
-- Adrian Cockcroft

15. Micro-­‐‑Services
“Loosely-­‐‑coupled service
oriented architecture with
bounded contexts”
-- Adrian Cockcroft

16. Micro-­‐‑Services
“Loosely-­‐‑coupled service
oriented architecture with
bounded contexts”
-- Adrian Cockcroft

17. Micro-­‐‑Services
• Single-purpose
• Simple, well-defined interface
• Modular and independent
• More graph of relationships than tiers
• Fullest expression of encapsulation and modularity
• Isolated persistence (!)
A
C
D
E
B

18. Micro-­‐‑Services
Each unit is simple
Independent scaling and
performance
Independent testing and
deployment
Can optimally tune performance
(caching, replication, etc.)
Pros
Many cooperating units
Many small repos
Requires more sophisticated tooling
and dependency management
Network latencies
Cons

19. Google Services
• All engineering groups organized into “services”
• Gmail, App Engine, Bigtable, etc.
• Self-sufficient and autonomous
• Layered on one another
è Very small teams achieve great things

20. Google
Cloud Datastore
• Cloud Datastore: NoSQL service
o Highly scalable and resilient
o Strong transactional consistency
o SQL-like rich query capabilities
• Megastore: geo-scale structured
database
o Multi-row transactions
o Synchronous cross-datacenter replication
• Bigtable: cluster-level structured storage
o (row, column, timestamp) -> cell contents
• Colossus: next-generation clustered file
system
o Block distribution and replication
• Cluster management infrastructure
o Task scheduling, machine assignment
Cloud
Datastore
Megastore
Bigtable
Colossus
Cluster
manager

21. Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

22. Reactive
Micro-­‐‑Services
• Responsive
o Predictable performance at 99%ile trumps low mean latency (!)
o Tail latencies far more important than mean or median
o Client protects itself with asynchronous, non-blocking calls
• Resilient
o Redundancy for machine / cluster / data center failures
o Load-balancing and flow control for service invocations
o Client protects itself with standard failure management patterns: timeouts,
retries, circuit breakers

23. Reactive
Micro-­‐‑Services
• Elastic
o Scale up and down service instances according to load
o Gracefully handle spiky workloads
o Predictive and reactive scaling
• Message-Driven
o Asynchronous message-passing over synchronous request-response
o Often custom protocol over TCP / UDP or WebSockets over HTTP

24. KIXEYE
Game Services
• Minimize request latency
o Respond as rapidly as possible to client
• Functional Reactive + Actor model
o Highly asynchronous, never block (!)
o Queue events / messages for complex work
o Heavy use of Scala / Akka and RxJava at KIXEYE
• Highly Scalable and Productive
o (-) eBay uses threaded synchronous model
o (-) Google uses complicated callback-based asynchronous model

25. KIXEYE
Service Chassis
• Goal: Make it easy to build and deploy micro-services
• Chassis core
• Configuration integration
• Registration and Discovery
• Monitoring and Metrics
• Load-balancing for downstream services
• Failure management for downstream services
• Development / Deployment Pipeline
• Transport layer over REST / JSON or WebSockets
• Service template in Maven
• Build pipeline through Puppet -> Packer -> AMI
• Red-black deployment via Asgard

26. KIXEYE
Service Chassis
• Heavy use of NetflixOSS
• Asgard
• Hystrix
• Ribbon + WebSockets è Janus
• Eureka
è Results
• 15 minutes from no code to running service in AWS (!)
• Open-sourced at https://github.com/Kixeye

27. Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

28. Migrating
Incrementally
• Find your worst scaling bottleneck
• Wall it off behind an interface
• Replace it
• è Rinse and Repeat

29. Building
Micro-­‐‑Services
• Common Chassis / Framework
o Make it trivially easy to build and maintain a service
• Define Service Interface (Formally!)
o Propose
o Discuss with client(s)
o Agree
• Prototype Implementation
o Simplest thing that could possibly work
o Client can integrate with prototype
o Implementor can learn what works and what does not

30. Building
Micro-­‐‑Services
• Real Implementation
o Throw away the prototype (!)
• è Rinse and Repeat

31. “So you are really
serious about this …”
• Distributed tracing
• Trace a request chain through multiple service invocations
• Network visualization
• “Weighted” communication paths between microservices / instances
• Latency, error rates, connection failures
• Dashboard metrics
• Quickly scan operational health of many services
• Median, 99%ile, 99.9%ile, etc.
• Netflix Hystrix / Turbine

32. Micro-­‐‑Service
Organization
• Small, focused teams
• Single service or set of related services
• Minimal, well-defined “interface”
• Autonomy
• Freedom to choose technology, methodology, working environment
• Responsibility for the results of those choices
• Accountability
• Give a team a goal, not a solution
• Let team own the best way to achieve the goal

33. Micro-­‐‑Service
Relationships
• Vendor – Customer Relationship
o Friendly and cooperative, but structured
o Clear ownership and division of responsibility
o Customer can choose to use service or not (!)
• Service-Level Agreement (SLA)
o Promise of service levels by the provider
o Customer needs to be able to rely on the service, like a utility
• Charging and Cost Allocation
o Charge customers for *usage* of the service
o Aligns economic incentives of customer and provider
o Motivates both sides to optimize

34. Recap: Evolution to
Micro-­‐‑Services
• The Monolith
• Micro-Services
• Reactive Systems
• Migrating to Micro-Services

35. Thank You!
• @randyshoup
• linkedin.com/in/randyshoup
• Slides will be at slideshare.net/randyshoup

36. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/reactive-
services-scale