Recommandé
Contenu connexe
Similaire à Scalable SAAS Meetup - Oct 6th
Similaire à Scalable SAAS Meetup - Oct 6th (20)
Dernier
Dernier (20)
Scalable SAAS Meetup - Oct 6th
- 1. Confidential and Proprietary Backend Engineering Best Practices Learned the Hard Way Mohit Dilawari
- 2. 2 Sr Director, Engineering 15 Years building SaaS products 5 Years at Localytics Mohit Dilawari
- 4. Some background on us...
- 5. 2010
- 6. Early 2014
- 7. 2014
- 8. This is 30% of our architecture Today
- 9. TBs of data ingested daily PBs of data stored
- 10. Our architecture/systems are becoming substantially more complicated
- 12. Ownership means wearing many hats: ● Architect ● Developer ● QA ● Operations ● SRE ● Security ● Support
- 13. Lessons that helped us scale and operate complex system
- 14. “MySQL is the golden hammer” Company #1 Lesson #1
- 15. “Uses the best tool for the job” Company #2
- 16. What company has a larger operational cost?
- 17. Pick a few technologies and operationalize them to point you are world class at running them. Lesson #1
- 19. Few
- 22. Operational cost should be the most important factor in your designs.
- 23. What do I mean by operational cost? ● Code efficiency ● Number of servers
- 24. What do I mean by operational cost? ● Code efficiency ● Number of servers ● Complexity of solution ● Engineer time ● Maintenance cost
- 25. How do you make it easy to make the best choice? Lesson #2
- 26. Invest in getting code to production fast!
- 27. Invest in tests
- 29. Early 2014
- 30. 2014
- 31. Lesson #3
- 32. Cattle > Pets -Adrian Cockcroft
- 33. When we started, we had couple pets
- 34. When we started, we had couple pets Invested a lot to turn the pets to cattle.
- 36. Lesson #5
- 37. QueueingQ
- 38. Product Manager on pager duty! Lesson #6
- 39. We had a few engineers and everyone knew our entire system In the early days... Lesson #7
- 40. HQ Metrics Dashboard ●Every service has one main dashboard ●Merge together system and business metrics
- 43. Codify your SOP Lesson #9
- 45. Learning Org
- 46. 1. Choose few technologies 2. Invest in system 3. Cattle not pets 4. Centralized Logging 5. ABQ 6. Engage product during prod issues 7. HQ Dashboards 8. Use percentiles 9. Codify your SOPs 10.Develop a learning org
- 47. Questions? Mohit Dilawari @mdilawari We are Hiring!
Notes de l'éditeur
- 2 major product lines… Analytics - See how your apps are doing Marketing - setup engagement capaigns and measure the effectiveness or LIFT Process TB of data daily We store PB of data
- This diagram represents 30% of our architecture
- This diagram represents 30% of our architecture
- This diagram represents 30% of our architecture
- We have no dedicated OPS and QA functions or gatekeepers. We have a dedicated tools team, but developers deploy their own code to production We held the line. There has been many times we have hired both roles. The urge typically follows a production issues.
- There are two major points of this presentation. Your total operational costs is the sum of all technologies New technologies have inherent known-unknowns or unknowns-unknowns
- Invest, Invest, Invest in systems to make developer productive and to do the right thing
- Invest in getting code to production fast safely Invest in Fast Tests Invest in tooling around making everyone a force multiplier
- Invest in getting code to production fast safely Invest in Fast Tests Invest in tooling around making everyone a force multiplier
- Invest in getting code to production fast safely Invest in Fast Tests Invest in tooling around making everyone a force multiplier
- When we moved to our microservices strategy, we invested substantially in to help us deploy
- Pets = You name them. They are all special snowflakes. They hold state. Cattle = you care about the total milk production, not about an individual cow
- Pets = You name them. They are all special snowflakes. They hold state. Cattle = you care about the total milk production, not about an individual cow
- When you start off, it is okay to have a couple Pets.
- When you start off, it is okay to have a couple Pets.
- Do not under estimate the cost of logging and monitoring
- Communicate via the queues everywhere Increase costs and increased development time Better guarantees Allows flexibility for downtime
- Communicate via the queues everywhere Increase costs and increased development time Better guarantees Allows flexibility for downtime
- Proactively address production issues that might languish if another team was responsible for maintaining their services in production Help them understand the pain of poor quality and technical debt This culture of ownership yields certain incentives and priorities for the team.