QCon London 2007: When moving to a Service-Oriented Architecture, SOAP and WSDL aren't the only option. HTTP, URIs and well-defined formats are the basis of the Web, and more architects are looking at them as a serious alternative. Beyond the hype of REST vs. WS-* and Web 2.0, it's possible to do some remarkable things with very simple tools. In this talk, I relate our experiences so far using HTTP and the Web for services in Yahoo!'s Media Group, covering our reasons for choosing this approach, the places where it's most effective (and where it's not), the pitfalls we've found along the way and what's still needed to make it developers' first choice for connecting their applications and data.

1. Leveraging the Web
for Services at
Mark Nottingham
<mnot@yahoo-inc.com>

2. Y! Calendar
Y! Mexico
Y! Shopping
Y! Sports Y! China
Y! Local Y! Auctions Y! Personal Finance
Y! Travel Y! Messenger Upcoming
Y! France Y! Small Business Bix
Y! Mail Y! Brazil del.icio.us
Y! Germany Y! Spain Y! Tech
Y! Korea Y! Health Flickr
Y! UK Y! Movies Y! India Y! Food
Y! Australia Broadcast.com HotJobs Y! Podcasts
Yahooligans! Y! Italy Y! Argentina Inktomi Y! Video
Y! Classifieds Y! Entertainment LAUNCH Overture Y! Music
Y! Japan Y! Games Y! Photos Y! Maps Kelkoo Y! Go
Y! Y! Singapore GeoCities Y! Groups Y! Search Y! 360
992 1994 1996 1998 2000 2002 2004 2006 200
100,000 25 million 200 million 500 million
users users users users
49 employees 11,400 employees
65 million 1.3 billion 4 billion
daily page views daily page views daily page views

3. Y! Auctions Y! Mexico Y! Messenger
Y! Germany Y! India
Y! Entertainment Y! China
Bix Y! Travel Inktomi
Upcoming
Y! Sports
Y! Korea
Y! Tech
Y! Small Business
Y! Brazil
Y! Mail del.icio.us
Y! UK
Y! Japan
Y!
Integration Y! Food
Flickr
Y! Go
Nightmare
Y! 360
Y! Spain
Overture
Y! France
Y! Health
Y! Music
Y! Search
Y! Italy
Y! Australia
Yahooligans! Y! Argentina
Y! Chinese
Y! Video
Broadcast.com Y! Games LAUNCH
Y! Photos
Y! Singapore
Kelkoo
Y! Classifieds GeoCities
Y! Maps
Y! Podcasts
HotJobs
Y! Shopping
Y! Groups

4. Between properties
With partners
With acquisitions
Within over time

5. “ Scale”

7. Media Group
Y! Music
Y! TV Y! Personal Finance
Y! Kids Y! Finance Y! Movies
Y! Fantasy Sports Y! Games Y! Health
Y! Sports Y! Astrology Y! Wii
Y! Tech Y! Food
Y! News

9. Front-End
Slave
Database
Master
Database

10. Large Datasets Don’t Push Well

11. Adding Capacity is Expensive
Large Datasets Don’t Push Well

12. Adding Capacity is Expensive
Large Datasets Don’t Push Well
What to do after push failure?

13. User-Generated Content Adding Capacity is Expensive
Large Datasets Don’t Push Well
What to do after push failure?

16. Building New Sites

17. Requirements

18. Requirements
massive scalability
flexible deployment
highly dynamic
separation of concerns
mashability

19. i.e.,
Services

20. Scalability Simplicity
Reuse Interoperability

21. i.e.,
HTTP

22. Front-End
Caching
Back-End
Database

23. Single Source of Truth

24. Single Source of Truth Cache Replicates Naturally

25. Adding Capacity Is Easy
Single Source of Truth Cache Replicates Naturally

26. UGC Pushes Through Adding Capacity Is Easy
Single Source of Truth Cache Replicates Naturally

28. HTTP Caching Intermediaries

29. HTTP Caching Intermediaries
Freshness

30. HTTP Caching Intermediaries
Freshness
Validation

31. HTTP Caching Intermediaries
Freshness
Validation
Metrics
0101011010010101101010011001010101

33. HTTP Caching Intermediaries
Freshness Load Balancing
Validation
Metrics

34. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics

35. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics Negative Caching

36. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics Negative Caching
Collapsed Forwarding

37. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics Negative Caching
Collapsed Forwarding
stale-while-revalidate

38. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics Negative Caching
Collapsed Forwarding
? stale-while-revalidate
stale-if-error

39. HTTP Caching Intermediaries
Freshness Load Balancing
Validation Cache Peering
Metrics Negative Caching
Collapsed Forwarding
stale-while-revalidate
stale-if-error
Invalidation Channels

40. 12,000 req/sec/core

41. Tens of Thousands of Connections

43. pitfalls
REST vs. WS-* wars
theory vs. practice
human-intuitive, but not programmer-intuitive
different deployment/operational concerns
formats are hard
format / interface proliferation
authentication isn’t there yet
tools have a way to go

44. what’s needed
tools
web description language
data-oriented schema language
investment in the Atom stack
HTTP test suite