Discussion of Swarming and Cynefin as means to better align devops functions into enterprise support channels.

1. Swarming, Cynefin…
and avoiding the problems of
becoming a third-line support team
Jon Hall
Principal Product Manager, BMC
© Copyright 2019 BMC Software, Inc.
@jonhall_
Velocity 2019, Berlin

2. 2© Copyright 2019 BMC Software, Inc.
Enterprise services are complex
Digital Services
Service
Subscribers
Tools
Processes
Services
Vulnerabilities3rd Party
Providers
Compliance
Environments

3. 3© Copyright 2019 BMC Software, Inc.
Enterprise Services are Complex
Compliance
Environments
Private
Cloud
Mainframe
Hybrid
Cloud
Data
Centers
Public Cloud
Digital Services
Service
Subscribers
3rd Party
Providers
Tools
Processes
Services
Vulnerabilities
Waterfall
Agile
Lean
SLAs
DevOps
DevSecOps
ITIL
End Users
Agents
IT Ops
Security
Applications

4. “IT organizations that have tried to custom-
adjust current tools to meet DevOps practices
have a failure rate of 80%”
DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified (IDC, 2014)
@jonhall_

5. • New services and applications suddenly appear
• Lost visibility when issues go to developers
• Lack of knowledge sharing
• New kinds of customer, especially external
DevOps challenges Service Management orthodoxies…
@jonhall_

6. • Scaling customer support
• Understanding the context of an issue
• Adaptation to life “on call”
• What to prioritise? Fixes or features?
• How to process alerts, particularly if noisy/low-quality.
…but enterprise realities challenge DevOps
@jonhall_

7. “The enterprise space doesn’t move slowly because
they’re stupid, or they hate technology.
It’s because they have users”
—Luke Kanies, Puppet Founder, Configuration Management Camp 2015, Belgium.

8. 2018 State of DevOps Report
There is room for Tech Support to bring value to DevOps
@jonhall_

9. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Classic “Tiered” Support Structure
@jonhall_

10. Escalation
Escalation
Deconstructing the “Tiered” Support Structure
LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 1 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
@jonhall_

11. @jonhall_
The Org Chart Problem
• Formal Structure
• The organisation chart
• Informal Structure
• Realm of influence between individuals
• Value Creation Structure
• How work actually gets done

12. • Work-in-progress queues
• Asynchronous communication
• Single role teams
• Individual over-exposure
• Lack of knowledge sharing
How to annoy a DevOps practitioner
@jonhall_

13. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT
LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS
LEVEL 1 SUPPORT
Classic “Tiered” Support Structure
@jonhall_
You are here?

14. Swarming
Network
Collaborative
Dynamic, loopy
Measured by value creation
Tiered support
Siloes and hierarchies
Directed
Linear, rigid
Measured on activity
@jonhall_

15. involves removing the tiers of
support, and calling on the collective
expertise of a “swarm” of analysts.
https://www.serviceinnovation.org/intelligent-swarming/
Swarming…
@jonhall_

16. “I have probably doubled my knowledge of the
products in a year because of Swarming…
and I have been here a long time”
Senior Support Analyst, BMC
@jonhall_

17. Severity 1 Swarm
Prioritise
Swarming Process at BMC
@jonhall_

18. • Rapid responders
• Three agents on a scheduled one-week rotation
• Primary focus: Provide immediate response, and resolve as soon as
possible
Swarm lead
Communications
Other members
Research, coordinate, test
Severity 1 Swarm
@jonhall_

19. Severity 1 Swarm
Local Dispatch Swarm
Prioritise
30% solved here
Swarming Process at BMC
@jonhall_

20. • “Cherry pickers”
• Meet every 60-90 minutes
• Primary focus: Can new tickets be resolved immediately?
• Also: Validation of ticket details before assignment to specialists
Experienced analyst Less-experienced analyst
Dispatch Swarm
@jonhall_

21. Local Product-Line
Support Teams
Severity 1 Swarm
Local Dispatch Swarm
Prioritise
Swarming Process at BMC
@jonhall_

22. Local Product Line
Support Teams
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Severity 1
Swarm
Local Dispatch Swarm
Prioritise
Local Product Line
Support Teams
Swarming Process at BMC
@jonhall_

23. Local Product Line Support Teams Local Product Line Support Teams
Backlog Swarm Backlog Swarm Backlog Swarm
Swarming Process at BMC
@jonhall_

24. • Global fixers of troublesome tickets
• Meet regularly (often several times a day)
• Primary focus: Challenging 3rd-line tickets
• Replace reassignments and individual assignments
Experienced analysts R&D Engineers
Backlog Swarms
@jonhall_

25. • Guidelines, not rules
• Metrics had to change (because old ones broke!)
• Some people needed help to became more customer facing
• Banned ticket tennis and direct escalations to experts
• New tooling practices, particularly mobile and chat
Things we had to do to make it work
@jonhall_

26. • 25% median resolution time improvement
• Customer satisfaction up 8 points
• More issues closed in <2 days
• Significant reduction in backlogs
• Halved on-boarding time
• Freed resources for innovative offerings
~1 year results at BMC
@jonhall_

27. “Swarming works better.
I am able to get multiple experiences from swarm
attendees of similar cases they have worked.
If there are no experiences, then it’s perspectives:
Decades of experience”
Senior Support Analyst, BMC
@jonhall_

28. Swarming at a Global Telco: Expert support for service desk
@jonhall_
CUSTOMER CHAT SESSIONS
Service Desk Agents
CHAT
CHANNEL
Subject Experts
CHAT
CHANNEL
Subject Experts
CHAT
CHANNEL
Subject Experts
• Regional chat-based service
desk at a global Telco
• Agents can put customer
on-hold for 3 minutes
• Subject experts wait in
“always-on” chat channels

29. Swarming example: Auto manufacturer’s connected cars team
@jonhall_
Engineering Team A
• First responder initiates and
coordinates swarms for big issues
• Other teams have 1 person on
rotation for swarming
• Swarms may also involve 3rd parties
(e.g. Amazon, Microsoft)
• Swarm grows and shrinks as needed
Engineering Team B 3rd Party Suppliers
First Responder
Challenge: how to scale support to millions of new vehicles every year.

30. • Perceived increase in “per record” cost
• Difficult to evaluate individual contributions
• “Cradle to grave” ownership across time zones
• Dominant individuals
• Finding the right people for a swarm is difficult
Issues reported by Swarming early adopters
@jonhall_

31. Swarming aligns really well to DevOps
• Autonomy and self-organisation
• Knowledge transfer and skills development
• ChatOps, not email
• Prevention of accumulation of queued work
• Protection of individuals from burnout
@jonhall_

32. ITSM is accelerating its learning from DevOps
ITIL® 4 Foundation (2019)
VeriSM – A service management
approach for the digital age (2017)
...and it must change in many areas to enable new thinking

33. Complex systems fail in complex ways
@jonhall_
“10% of nodes enter a simultaneous crash loop cycle, about five times
a day, at unpredictable intervals. It clears up before we can debug it”
“We run a platform, and it’s hard to distinguish between problems that
users are inflicting on themselves, and problems in our own code,
since they all manifest as the same errors or timeouts”.
“I have 20 microservices and three datastores across three regions, and
everything seems to be getting a little slower over the past 2 weeks
…but nothing has changed that we know of.
Latency is usually back to the historical norm on Tuesdays”
So… who do you assign to?
Charity Majors
Observability for emerging infra
Config Management Camp, Ghent 2019

34. • Wait… What?
• Pronounced “kuh-nev-in”
• Developed by Dave Snowden while at IBM in 1999
• “Decision support framework which comes from a
mixture of complexity theory and cognitive science…
the opposite of a one-size fits all model”
Swarming as a means of delivering Cynefin?
@jonhall_

35. @jonhall_
• Obvious and Complicated domains:
• Repeating relationship between cause and
effect
• With Complicated you need to do analysis
to find that relationship
• Complex domain:
• Understanding the problem requires
experimentation and analysis.
• May, over time, be able to move to
Complicated
• Chaotic domain:
• Dramatic and unconstrained
• Focus on damage limitation, try to move to
another domain

36. “Obvious” Domain
@jonhall_
• “Sense, Categorise, Respond”
• Can apply best practice
• Template/knowledge-driven resolution
• Self service

37. “Complicated” Domain
@jonhall_
• “Sense, Analyse, Respond”
• Good practice.
• Dispatch-type swarm – pair up agents with varied
experience
• Capture detailed knowledge for organizational learning

38. Identify
“coherent”
hypotheses
Cynefin approach in a Complex system
@jonhall_
• “Sense, Analyse, Respond”
• Identify multiple hypotheses
• Gain understanding of the system by interacting with it.
• Create predictability, increase constraints, hopefully to
enable move to Complicated
Convene
“safe to fail”
experiments
Observe and
monitor impact
Amplify good
patterns,
dampen bad

39. “Complex” Domain
• “Probe, Sense, Respond”
@jonhall_

40. “Complex” Domain
• “Probe, Sense, Respond”
@jonhall_

41. “Chaotic” Domain
• “Act, Sense, Respond”
• Sub-swarms
• Deal with the acute situation
• Try to discover sufficient
information to move to complex
@jonhall_

42. • Service Management needs to evolve its practices and
tooling deliver specific new value to you.
• We need to listen to how support is affecting your role,
as your impact grows in your enterprise.
• You are agents of change in enterprises, with a good
opportunity to influence thinking.
What next?
@jonhall_

43. serviceinnovation.org/intelligent-swarming
@jonhall_
http://www.bosslevelpodcast.com
/dave-snowden-on-complexity-
theory-and-astrology/
http://medium.com/@jonhall_
https://youtu.be/fOdtgHu_KeA
Consortium for Service Innovation:
Intelligent Swarming
Boss Level Podcast:
Dave Snowden on Cynefin
Long-form blog on why
Swarming works better for DevOps
Charity Majors at #cfgmgmtcamp
Observability for emerging infra
@mipsytipsy
@matthewpskelton
@manupaisable
@NielsPflaeging