The document discusses designing software for testability in production. It recommends removing staging environments and running automated tests and a subset of code in production to gain confidence in software quality. Specific techniques include using sandbox accounts to test functionality without affecting real users, running integration tests that create and delete sandbox environments, and monitoring databases and logs in real-time to detect issues. Designing systems with isolation, well-defined interfaces, and the ability to restrict access and trace effects can help make testing in production safer and more effective. Tools like API documentation generators and mocks can also help improve quality.

1. Production: Designing
for Testability
Michael Bryzek
mike@flow.io / @mbryzek
Cofounder / CTO Flow
Cofounder / ex-CTO Gilt

2. Think about something you deployed
recently to production…

3. Is It Working?

4. Right now in production?

5. How do you know?

6. Feeling anxious?

7. 7
Let’s remove that anxiety

8. About Me
www.flow.io
www.gilt.com

9. Software Quality is Hard
Think end to end for entire lifecycle of code
Verification in Production is a powerful technique
to help us build quality software

10. True continuous delivery
No staging environments
Don’t run code locally
Life at Flow: Delivering Quality Software

11. True Continuous Delivery
Automated tests / No safeguards
1 way to do something
Assume Continuous Delivery in Design Process
github.com/flowcommerce/delta

12. “I love my staging environment”
Said Nobody Ever

13. No Staging Environments
Bottlenecks
Fragile
Difficult to understand failure
Expensive (30-40% of budget common)
Create the wrong incentives

14. Don’t run code locally
If unsure, write the test!
Learn to trust your tests

15. Quality Through Architecture
Extreme Isolation
• Rich event streams
• Own DNS, load balancer
• Private database
• No consul/zk/shared state
• Stop cascading failures
• “Delay” not “Outage”

16. Let’s look at real examples
Successfully “testing in production”

17. Example: Know That Checkout Works
Bot places an order every few minutes
Identify test orders and immediately cancel

18. Example: Support “Sandbox” Accounts
”SaaS” – even for internal accounts
Mark individual accounts as sandbox
One API Key for all sandbox accounts
"every service is a third party"

19. Example: End to End Integration Tests
Create Sandbox Org
Run tests
Delete Sandbox Org
“Safe and Repeatable”

20. Example: Using Sandbox Account for Test Orders

21. Example: Verifying Proxy Server Works as Expected
https://github.com/flowvault/proxy

22. Operating As Expected

23. But sometimes things go wrong
Even to the best of teams

24. Considerations
Make production access explicit (not the default)
Use defined paths (e.g. API calls)
Restrict sensitive data
Design for side effects

25. Unexpected Benefits

26. Perfect Documentation

27. Capture request/response of API Calls

28. Tooling: API | Builder
(formerly known as apidoc)
28
www.apibuilder.io
Version control for APIs
Backwards Compatibility
High Quality Mocks

29. High Quality Mocks – From Specs
Full Mock Generated
Implement Only What
You Need To Test
www.apibuilder.io

30. Tooling: Real Time DB Monitoring
https://www.vividcortex.com/blog/2014/04/22/visualizing-impact-index-change/

31. Tooling: Super Simple Alerts from Log
Log a prefix
Schedule a real time alert

32. Key Takeaways – Design Production to be Testable
Trust your tests, run subset in production
Invest in continuous delivery
Sandbox accounts are powerful
High quality, trustworthy mocks
Real-time feedback from production

33. Thank You!
We’re hiring: https://www.flow.io/careers
Michael Bryzek
mike@flow.io / @mbryzek
Cofounder / CTO Flow
Cofounder / ex-CTO Gilt