Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Scaling Infrastructure Engineering at Slack

21 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k6S0kO.

Julia Grace was asked to build Slack’s first infrastructure engineering organization in August 2016. The company was two years old and they were approaching the scalability limits of the original infrastructure. Things were starting to break in strange and unpredictable ways. She discusses the architectural and organizational challenges, mistakes and war stories of 2.5 years that followed. Filmed at qconnewyork.com.

Julia Grace is currently a senior director of product engineering at Slack focused on building network effects into Slack through shared channels. Prior to joining product engineering she built the infrastructure team at Slack, growing it from from 10 to 100 engineers in three offices in two years.

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Scaling Infrastructure Engineering at Slack

  1. 1. @jewelia Scaling Slack Infrastructure 🚀 Julia Grace Senior Director of Engineering @jewelia
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-scaling-infrastructure/
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. @jewelia Phase 0: 2015
  5. 5. @jewelia ~2.5M Daily Active Users
  6. 6. @jewelia Phase 1: 2016
  7. 7. @jewelia ~4M Daily Active Users
  8. 8. @jewelia Phase 1: 2016 Slack was originally designed for teams < 150ppl. You make very different architectural decisions when you’re building for a team of 100 people vs 500,000. Before August 2016 we had no Infra team. Original infrastructure built for Glitch worked very well in 2014/2015. ~150 Engineers total. Infrastructure investments would come secondary to feature work.
  9. 9. @jewelia Things were starting to break in strange, unusual ways. 

  10. 10. @jewelia Phase 1: 2016 Example: User Presence Green dot indicaFng online/away/offline. Very few people noFce it, unless it’s broken (people expect it to “just work”). Apps and bots are always online.
  11. 11. @jewelia Phase 1: 2016 User Presence IniFally broadcast all changes to all users (e.g. “Julia Grace is away”) to the whole workspace: O(n^2). Presence was ~80% of all web socket traffic. Peak volume in late 2016: 16 million messages/minute over web socket. Presence messages: 13 million/minute. Rapidly transiFon from broadcast to publish/subscribe.
  12. 12. @jewelia There were many organizational challenges as well. 

  13. 13. @jewelia Phase 1: 2016 How to build engineering-led org in a product- led company? Would we be able to get headcount, budget? How to communicate the value of we are doing to non-technical audiences? How do we interface with sales? Infrastructure as a compeFFve advantage.
  14. 14. @jeweliahUps://www.flickr.com/photos/pocheco/14833391966
  15. 15. @jewelia Phase 1: 2016 Start internal evangelism on day #1. I went on an internal PR campaign: Why our work was important, why we needed to conFnually invest in infrastructure. Make work very visible to execs in other funcFons. Followed existing company process. We did planning, status reporFng, etc. at the same cadence and in the same meeFngs as product engineering. Don’t try to start a new group and invent new process. Identify executive sponsor.
  16. 16. @jewelia Phase 2: 2017
  17. 17. @jewelia Phase 2: 2017 Technology landscape. Hack/PHP monolith on backend, JavaScript with no libraries on frontend. 1 service: presence and real-Fme messaging. Building a second service: Go caching service. These bespoke services each had to handle rate limiFng, traffic management, deployment.
  18. 18. @jewelia Phase 2: 2017 It was time to change our DB sharding strategy. MySQL sharded by team/workspace to Vitess sharded by various keys. Worked great! UnFl we hit scaling limits, significant hotspots. 

  19. 19. @jewelia Monolith Service A Service B
  20. 20. @jewelia Monolith Service A Service B
  21. 21. @jewelia Monolith Service A Service B Who owns this?
  22. 22. @jewelia Communication Risk The more technically complex, nuanced a problem is…
  23. 23. @jewelia Communication Risk The more technically complex, nuanced a problem is… The higher the communication risk.
  24. 24. @jewelia Phase 2: 2017 Immense pressure to hire engineers. Many human SPOFs (single points of failure) because team was so small. Everyone was overextended and overcommihed. We had to figure out how to hire Infra engineers. All our hiring processes were opFmized around hiring generalists: frontend backend, iOS, Android, Ops. We skills do we need and value? How do we test for those skills?
  25. 25. @jewelia Phase 2: 2017 Decided to hire Infra engineering generalists. Created a take home coding exercise designed to test: 1. An understanding of servers, networking, and protocols. 2. An understanding of concurrency, performance, and resource constraints, and an ability to anFcipate future issues and implement soluFons. 3. An ability to write clear, easy to understand code, communicate your approach, and reason about tradeoffs that you have made.
  26. 26. @jewelia Phase 2: 2017 I wore so many hats. Too many hats. Similar to my days as a startup CTO! I was the Engineering Director and Forming strategy, hiring managers and ICs, evangelizing the org. …Product Manager and Internal interface to Product Engineering/PMs building features, externally to customers with quesFons about the integrity of our infrastructure. …Program Manager. Running cross funcFonal iniFaFves.
  27. 27. @jewelia Phase 3: 2018
  28. 28. @jewelia Phase 3: 2018 “0 to 1” was over. Now time for “1 to ∞”. ReacFve to ProacFve. Transition from few teams to an org in 3 offices. Team nearly 100 engineers by end of year. Now included Data, Machine Learning, Search Infrastructure Many orders of magnitude better performance Things were not breaking all the time.
  29. 29. @jewelia Phase 3: 2018 Services model matured significantly. SLAs for services, consistent deployment processes, etc. Mature incident response process. Dividing into sub-teams made sense. Data Stores & Cache Infra, Service Mesh & Web Serving, Distributed Messaging.
  30. 30. @jewelia Phase 3: 2018 Hired Director Specialists… Had to quickly learn how to hire senior leaders whose jobs you haven’t done before. How to do this well: talk to a lot people who currently do the job you’re trying to hire for, deeply understand the talent market. and Product Managers… and did an acquisition.
  31. 31. @jewelia Phase 3: 2018 Challenge: coherency across a large organization. Example: overlap between Machine Learning and Frontend Infra was NULL. Difficult to have a unified vision. Stakeholders were each org were different for each part of the org; Data Infra organizaFon worked closely with G&A (finance), Search Infra did not. I should have done more re-orgs!
  32. 32. @jewelia 2016:
  33. 33. @jewelia 2016: 2017:
  34. 34. @jewelia 2016: 2017: 2018:
  35. 35. @jewelia Today Infra has been around for ~3 years 400M async jobs processed/day to 2.5B 3M DAU (daily active users) to 10M DAU 1M simultaneously connected users to 7.5M 10 to ~100 engineers in SF, NYC, YVR Generalist (ICs, Managers) to specialists 1 amazing team
  36. 36. @jewelia Thank You!
  37. 37. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-scaling-infrastructure/