DevOps in the Enterprise: How “Swarming” can fix the problem of becoming a 3rd-Line Support Team

1. DevOps in the Enterprise:
How “Swarming” can fix the problem
of becoming a 3rd-Line Support Team
Jon Hall
Principal Product Manager, BMC
@jonhall_
DevOpsDays Riga 2018

2. “The enterprise space doesn’t move slowly because
they’re stupid, or they hate technology.
It’s because they have users”
—Luke Kanies, Puppet Founder, Configuration Management Camp 2015, Belgium.

3. DevOps growth in established enterprises
“Startup-like”
visionaries
New services.
Ad-hoc support
Services outgrow ad-hoc support.
Enterprise support channel “adopts”.
@jonhall_

4. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Classic “Tiered” Support Structure
@jonhall_

5. Escalation
Escalation
Deconstructing the “Tiered” Support Structure
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
@jonhall_

6. …when the answer is here… …or here.
Issues may spend time here
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
@jonhall_

7. LEVEL 1 SUPPORT
LEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS
When tickets
eventually
escalate…
…they frequently
bounce back
@jonhall_

8. LEVEL 1 SUPPORT
LEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS
LEVEL 2 SUPPORT
SUBJECT MATTER EXPERT
The system creates “heroes” (not in a good way)
@jonhall_

9. involves removing the tiers of
support, and calling on the collective
expertise of a “swarm” of analysts.
https://www.serviceinnovation.org/intelligent-swarming/
Swarming…
@jonhall_

10. Swarming
Network
Collaborative
Dynamic, loopy
Measured by value creation
Tiered support
Siloed, hierarchical
Directed
Linear, rigid
Measured by activity
@jonhall_

11. 24 hours, 365 days.
500 specialists, 3000 years of experience
200,000+ incidents addressed each year
Communication skills are a hiring focus
BMC Contact Centres
Support Centres
Support Centres Co-located with R&D
Pleasanton/
Sunnyvale
Houston
Austin
McLean/
Herndon
Lexington
Sao Paulo
Buenos Aires
Spain
Dublin
Winnersh
Amsterdam
Paris
Tel Hai
Pune
Singapore
Shanghai
Beijing
Seoul
Dalian
Tokyo
Melbourne
Houston, TX, USA
Dublin, Ireland
Dalian China
BMC Customer Support
@jonhall_

12. Swarming at BMC
Dispatch SwarmSeverity 1 Swarm Backlog Swarm
@jonhall_

13. Severity 1 Swarm
Prioritise
Swarming Process at BMC
@jonhall_

14. • Rapid responders
• Three agents on a scheduled one-week rotation
• Primary focus: Provide immediate response, and resolve as soon as
possible
Swarm lead
Communications
Other members
Research, coordinate, test
Severity 1 Swarm
@jonhall_

15. Severity 1 Swarm
Local Dispatch Swarm
Prioritise
30% solved here
Swarming Process at BMC
@jonhall_

16. • “Cherry pickers”
• Meet every 60-90 minutes
• Primary focus: Can new tickets be resolved immediately?
• Also: Validation of ticket details before assignment to specialists
Experienced analyst Less-experienced analyst
Dispatch Swarm
@jonhall_

17. Local Product-Line
Support Teams
Severity 1 Swarm
Local Dispatch Swarm
Prioritise
Swarming Process at BMC
@jonhall_

18. Local Product Line
Support Teams
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Local Product Line
Support Teams
Swarming Process at BMC
@jonhall_

19. Local Product Line Support Teams Local Product Line Support Teams
Backlog Swarm Backlog Swarm Backlog Swarm
Swarming Process at BMC
@jonhall_

20. • Global fixers of troublesome tickets
• Meet regularly (varies from multiple times per day, to weekly)
• Primary focus: Challenging tickets brought by local support teams
• Take the place of individual subject-matter-expert escalation
Experienced analysts R&D Engineers
Backlog Swarms
@jonhall_

21. • Guidelines, not rules – encourage good decisions
• Metrics had to change (Swarming breaks traditional ones)
• Some people needed help to become newly customer facing
• Direct escalations to subject matter experts were banned
• Increased use of chat and mobile tools
Making it work at BMC Customer Support
@jonhall_

22. • 25% median resolution time improvement
• Customer satisfaction up 8 points
• More issues closed in <2 days
• Significant reduction in backlogs
• Halved onboarding time
• Freed resources for innovative offerings
Results at BMC
@jonhall_

23. So… what does this all have to do with DevOps?
@jonhall_
How organisations spend their time (State of DevOps Report, 2018):

24. “IT organisations that have tried to adjust
current tools to meet DevOps practices have a
failure rate of 80%”
DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified (IDC, 2014)
So… what does this all have to do with DevOps?
@jonhall_

25. • More developers, less “off the shelf” software
• More resolution by coding
• Increased change frequency
• Disconnect in tooling (ITSM vs SDLC)
DevOps challenges ServiceDesk norms…

26. • Industrial-scale user base
• Customer/business context
• Complexity of the enterprise environment
• Adaptation to life ”on call”
• What to prioritise? Fix bugs or build new stuff?
…but enterprise realities challenge DevOps
DevOps challenges ServiceDesk norms…

27. • Work-in-progress queues
• Asynchronous communication
• Single role teams
• Individual over-exposure
• Lack of knowledge sharing
How to really really annoy a DevOps practitioner
@jonhall_

28. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Where have we seen those things before?
@jonhall_
You are here?

29. Swarming aligns really well to DevOps
• Autonomy and self-organisation
• Knowledge transfer and skills development
• Aversion to work queues
• ChatOps
• Protection of individuals from burnout
@jonhall_

30. • May increase costs (even if other measures improve)
• Difficult to evaluate individual contribution
• Some individuals may dominate
• Finding the right people for a swarm is difficult
It’s not all positive – some issues reported by
Swarming practitioners
@jonhall_

31. “The most frustrating job I had was in a very large company.
When I stepped outside the box, my colleagues really
appreciated it. But management did not.
The people that excel and create the most value are the
ones that step outside their box”
Hank Barnes, Gartner: “Playing Outside Your Box” May 2018.

32. “I have probably doubled my knowledge of
the products in a year because of Swarming,
and I have been here a long time”
- Senior Support Analyst, BMC
@jonhall_

33. medium.com/@jonhall_serviceinnovation.org/intelligent-swarming
Some more information
@jonhall_