This deck represents our current thinking about the best way to build enterprise SaaS software in 2015 - using a variety of techniques from several disciplines. Since I wrote this I have also become very interested in resilience engineering and the notion that web developers are primarily engaged in the construction of socio-technical systems. When I rewrite this I plan to talk about how we should try to minimize mean-time-to-recover (MTTR) instead of mean-time-between-failures (MTBF), and how continuous deployment grows a safety culture around your operations. I redacted most of the examples that illustrate these points because they use sensitive code examples or URLs. If you want to see the rest of slides, join us!

1. Development Manual: Fast and Good

2. PRIORITIES

3. Priorities
1.Accuracy
2.Timeliness
3.Performance

4. Predictability >> Speed

5. Professionalism >>
Heroism

6. Defense systems <=>
New features &
integrations

7. Ethical Conduct,
Always
When in doubt, ask!

8. THE VISION

9. How do we build software
in a predictable,
professional way?

10. We manage
the risk of change

11. It’s more risky
to NOT change code

12. Let’s make it impossible
to break the app

13. PROCESS
GUIDELINES

14. All bugs are ultimately
process failures
Our systems should reject bad changes
and enforce quality automatically

15. Deliver incremental
results

16. What’s the 10% solution
you could ship today?

17. Have more meetings
• Make decisions
• Hash things out
• Explain your plan
• Brainstorm
• Write pseudocode
• Make diagrams
• Sharpen understanding

18. I, Mike Subelsky,
heartily endorse meetings
IIF they entail decision-making, brainstorming,
mentoring, conflict, fun, excitement

20. Conduct Root Cause
Analysis After Every
Outage
“The Five Whys”

21. Team leaders
send weekly change updates
to all@

22. Prioritize automatic tests and
quality control
over new features/integrations

23. Place one person
on QA / bugfixing duty
each week
Need a cool object to pass around the office

24. PROJECT PLANNING
GUIDELINES

25. Set smaller goals
you can actually achieve
We need more milestones,
hard deadlines, “forcing functions”

26. 3 Layers of Planning &
Goalsetting
• Goals
• Epics
• Stories

27. Goal
• Strategic objective that helps STAQ grow
• Increases revenue or reduces costs, or both
• Expressed as a single sentence, completely in
business terms, associated with a deadline
• “Charge customers for use of custom
connections by 12/01/2015”

28. Epics
• Major, high-level initiative to help achieve a Goal
• Complex effort encompassing changes to multiple
codebases
• Expressed in business terms, in terms of user
capabilities, with a deadline
• “Users can view, create, update, and delete custom
connections on their own by 11/15/2015”
• Constantly reprioritized as the project deadline
approaches

29. Push back
• AMs/TAMs should always push us to deliver our best,
advocating for customer
• We should push back when real constraints exist
• Helps everyone be more creative
• Helps clarify priorities
• 80% / incremental solutions often good enough
• Resist the urge to dive in and save the day

30. Refactoring &
maintenance cycles
• Count on about a week of fixing and polishing
major features post-release
• After an epic or major project finishes, we do this
anyway; let’s plan for it
• Good time to handle chores, smaller feature
tickets & bug fixes

31. CODING GUIDELINES

32. Make small changes

33. Observe the
Single Responsibility
Principle
Most other OOP principles derived from this one

34. - https://en.wikipedia.org/wiki/Single_responsibility_principle
“…every class should have responsibility over a
single part of the functionality provided by the
software.”

35. - https://en.wikipedia.org/wiki/Single_responsibility_principle
Responsibility is “…a reason to change…a
class or module should have one, and only one,
reason to change.”

36. Deploy in AM/mid-day

37. Prefer to deploy heavy/
breaking changes on
weekends
Avoid Sunday morning when collections run

38. Evening deploys OK
but more dangerous
You’ll get lazy / sleepy
Make a rollback plan

39. Use the staging server
• https://staging.staq.com
• https://sites.google.com/a/staq.com/
staqnowledged/home/infrastructure/staging-
server

40. Fork the app
• Good for testing database changes
• Need someone to try & document this
• https://devcenter.heroku.com/articles/fork-app
• May need to fork multiple apps in concert
• Heroku addon attachment feature pretty cool

41. Use feature flags

42. Create parallel
gems/engines/
extractors/tables

43. Design solid code
open to future change
But also beware of YAGNI
(You Ain’t Gonna Need It)

44. Make it easy to test
This is the main point of BDD; produces better designs

45. Lint Ruby and JS Code
• Rubocop
• ruby -c after every save
• Let’s make this a git pre-commit hook
• brew install jsl

46. Keep writing tests
• Unit tests: 100% coverage
• Usually no need to test explicit string contents
• Integration test: all major subsystems
• Including failure modes
• Acceptance test: most features, important failure
modes
• Smoke test: engines/gems integrated into apps

47. Tests must always
pass on CI
Fixing the build should usually
take precedence over other work

48. Keep Classes Small
• “One screenful or less”
• Not counting documentation
• Almost no private methods
• There’s always a better way, look for a hidden
object

49. Write many, many
classes

50. Use clean architectural
patterns
• 12factor
• Dependency injection
• Hexagonal architecture

51. Clear code >>
documentation >>
comments

52. Refactor constantly
Think like a gardener, not like a landscaper

53. Make the change easy,
then make the easy
change

54. Write many,
many gems/engines

55. Continue
peer code review

56. You are only done when…
• Documentation written (YARD tags + wiki)
• Unit tests written to 100% coverage
• Integration tests covering important subsystem
interactions
• Acceptance tests covering important features
and failure modes
• Code reviewed by a peer

57. MONITORING
AND ALERTING

58. Develop a few high-level,
actionable alerts
in staqmonitor

59. Many cool possibilities

60. We are going to build
an immune system
Rejects bad changes

61. Develop indications
and warning indicators

62. QUESTIONS?