Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1D2YQGg. John Bunting talks about different services Tumblr has built and how their architecture helps them be fault tolerant as they continue to grow. Filmed at qconsf.com. John Bunting is a pragmatic programmer. Possible cyborg hacker for at Tumblr. Loves Python, Ruby and everything Tech.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/tumblr-services

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Bits to Gifs
Tumblr

5. John Bunting
Senior Engineer
Languages: Ruby, Python, PHP, bit o’ C, some JS
Tumblr: blog.johnbunting.me
Twitter: @codingjester
Email: jb@tumblr.com

6. About Tumblr

7. Original Architecture

8. So what’s different, today?

9. Current Architecture

10. Tech we use
C

11. A place for the bits...
Storage

12. MySQL
→ Percona MySQL 5.5
→ Range sharding on Blog ID
→ Homes most data to their shard

13. How do you manage all those databases?

14. Jetpants!
→ Manages many common DB tasks
→ Can divide 750GB, billion row shards in 6 hours
→ Programmatically interact with your topology
→ http://github.com/tumblr/jetpants

15. Other than MySQL?
→ Memcached
→ Redis/Rediscover
→ HBase
→ HDFS
→ S3 for Image Storage

16. Throwing around bits...
Services

17. What are they built in?
→ Scala
→ C
→ Older services, Finagle
→ Newer services, Colossus
→ HBase
→ Redis
→ Thrift
→ Protobuf

18. Rediscover
→ A thin proxy wrapper for Redis
→ Implements the protocol and hashes keys
→ Has a tool called “Redistribute” for slave promotion
→ Used as storage for many offline jobs

19. Rediscover Architecture
Client Client Client Client
rc-proxy rc-proxy
redis
redis
redis
redis
redis
redis
redis
redis
redis
redis
redis
redis
RedisTribute
Zookeeper

20. Refrog

21. Refrog

22. Firehose
→ Collects, stores and streams Tumblr data to listening client
→ Built with 3 pieces:
→ Kafka
→ Mailman
→ Geyser

23. Firehose Architecture

24. Dynamic Image Resizer
→ Amazon EC2 Servers
→ Fronted by a CDN
→ Custom Nginx C module
→ Resizes a variety of images on the fly

25. Async Requests
→ Gearman
→ Persistent Queues using Redis
→ Multiple clients implemented and available
→ Fast and effective to scale horizontally

26. Bits that move and shake...or not.
Dynamic vs Static

27. Dynamic vs Static

29. Blog Network
→ Blogs are identified by their FQDN
→ Over 23,000 Blog requests per second (at peak)
→ Over 6,500 cache purges at peak
→ Started to outgrow static partitioning
→ Patched HAProxy to use DJB2 hashing on FQDN

30. Static Partitioning to Consistent Hashing

31. Measuring the Bits...
Tools

32. We had no central repository of active servers...

33. Collins - Asset Management

34. Deployment took forever...

35. DUI - Deployment Tool

36. We needed better introspection...

37. Fibr - Graphing

38. More information about the system was needed...

39. Large Scale Batch Processing
Web App/Services/Gearman
Scribe
Hive / HBase
Crons
Primary Datastores

40. The bits coming soon...
The Future

41. The Future
→ Exploring more into real-time data processing
→ Continuing to scale an ever growing dataset
→ More services and optimizing the current ones
→ Tons of projects getting ready to be open-sourced
→ Docker and Virtualization for deployment

42. Thanks!
→ Want a cool job? https://www.tumblr.com/jobs
→ Tumblr: blog.johnbunting.me
→ Twitter: @codingjester
→ Email: jb@tumblr.com

44. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/tumblr-
services