HumanOps is a set of principles which focus on the human aspects of running infrastructure. It deliberately highlights the importance of the teams running systems, not just the systems themselves. The health of your infrastructure is not just about hardware, software, automations and uptime - it also includes the health and wellbeing of your team. The goal of HumanOps is to improve and maintain the good health of your team: easing communication, reducing fatigue and reducing stress.

1. Jorge Salamero Sanz <jsalamero@serverdensity.com>
Atmosphere Conference Krakow May 2016
HumanOps - the impact of health on operations

2. Jorge Salamero
@bencerillo
@serverdensity
blog.serverdensity.com

4. www.CloudStatusApp.com

5. ● Infrastructure automation
● Configuration automation
● Continuous testing
● Continuous deployment / delivery
● Monitoring
● Logs, error handling
● Feedback
● Human Ops

6. ● Humans are part of any system
● Initial design, ongoing improvements
● Maintenance
● Upgrades
● Issues, Incident response

7. ● System issues = error rates + SLA + ...
● Human issues = alerts out of hours + interruptions + .
● System issues = Human issues

8. ● Downtime = loss of users, reputation, revenue
● Downtime caused by unreliable systems
● Unhealthy teams reduce reliability
● Unhealthy teams = loss of users, reputation, revenue

9. ● Slip
● Lapse
● Mistake
● Violation
● (Always, again, again)

10. What can we do?

11. ● Prepare and practice
● Respond
● Postmortem

12. Real example
(small war story, won’t be long)

13. ● Power failure to half of our servers
● Automated failover unavailable
(known failure condition)
● Manual DNS switch required
● Expected impact: 20 min
● Actual impact: 43min

15. Lessons learned?

16. ● Unfamiliarity with the process
● Pressure of time sensitive event
(panic effect)
● Escalation introduces delays

17. Handling the Human factor

18. ● First responder, acknowledge alert
● Load incident response checklist
● Log into #ops-war-room in Slack
● Log incident into JIRA
● Begin investigation

19. 1. Extended use of checklists
2. Not to follow blindly, use knowledge
and experience
3. Independent system
4. Searchable
5. List of known issues and
documented workarounds/fixes

20. ● The “limits of human memory and
attention”
○ Complexity
○ Stress and fatigue
○ Ego
● Pilots, doctors, divers:
Bruce Willis Ruins All Films
(BCD, weights, releases, air, final)

21. 1. Extended use of checklists
2. Not to follow blindly, use knowledge
and experience
3. Independent system
4. Searchable
5. List of known issues and
documented workarounds/fixes

22. ● Realistic replica environment
● or mock command line
● Record actions and timing
● Multiple failures
● Unexpected results

23. Results

24. ● Team and individual test of response
● Run real commands
● Training the people
● Training the procedures
● Training the tools

25. ● Increase confidence
● Reduce panic
● Better coordination
● Trust relationships
● Improves time to resolution

26. ● Review
● Suggestions for improvements
● Do it again
● Scenario evolves
● People forget

27. ● On call rotation design
● Alert prioritization
● Notification optimization

28. Human Ops

29. 1. Humans are part of the system
2. Humans impact systems
3. Humans impact business
4. Human issues count as system issues

30. meetup.com/humanops-london/
humanops.com

31. serverdensity.com/conferences
DEVOPSDAYS

32. Jorge Salamero
@bencerillo
@serverdensity
blog.serverdensity.com