Tempo. Most people are familiar with it in the musical sense. It’s the speed, cadence, rhythm that the music is played. It drives the music forward - and pulls it back. But there’s more to tempo than a musical beat. In war, like in business, tempo - the speed at which you can transition from one task to the next - is a critical component for victory.
No single person nor department owns tempo. Somebody can’t just shout, “I now control the tempo,” and take charge. If you operate at a faster tempo than your cycle time allows, then you’ll get thrashing. The rate of tempo emerges organically as companies move around that action loop of sensing, deciding and acting.
Tempo emerges from the convergence of architecture, infrastructure, organization, and mindset. All these things have to align to achieve tempo. None of them can be changed in isolation.
In this talk, we will look at different models for transforming an organization to high tempo and high performance. We'll see how that can get derailed and what to do about it.
68. Michael T. Nygard
Thank You!
Want the slides?
Email mtnygard@zip.sh, Subject: “tempo”
@mtnygard
www.michaelnygard.com
Notes de l'éditeur
my poor defenseless system and (click)
a horde of developers carrying (click)
these, except for one
brave sysadmin standing like Horatius defending the Tiber. I’d tell you about how I put them all in a chroot jail and wouldn’t
let them sudo. But! We’re not at that kind of conference, and this isn’t that kind of story. No, this is more like ...
it’s more like a medical drama, where we all have to work together heroically to save the patient but nothing makes sense
until a random personal comment causes a sudden flash of insight then we cure the patient in the last 10 minutes with just enough time left over for a tidy denouement in that tense personal situation.
the victims in our drama turn out to be the bags of money. based on our capacity projections heading into the holiday peak season, it looked like we’d need about $10M in new app server hardware to handle load… assuming linear scaling.
we draw a connection like this.
some of these weren’t even on the logical diagram. what looked like front vs back actually just goes out eth0/eth1 (bonded).
back end is the monitoring network. monitoring network reserved for monitoring & admin. high I/O on monitoring switch was unexpected.
amusing myself with iostat and vmstat, noticed high usage on back-end NIC. what do i mean by back end?
we draw pictures like this, but what we really deal with is this.
ethereal. now wireshark… riff on the vibe of “wire shark” vs. “ethereal”. new-agey hippie gentle ethereal. wire shark. hard edged. Kind of thing that you love if you’re a VC that reads too much Tom Clancy.
trace showed a lot of TCP cxn setup/teardowns from app to app. sending 1 packet per connection.
app-to-app traffic on monitoring network. specifically, cache invalidations.
hundreds per second per server. each app opening sockets to each other app instance. (about 100)
p-2-p cache inval not visible in dev. not observable in QA. killer in production.
Displaying a product changed a transient attribute, but caused cache invalidation. Knocked item out of cache on every other server. It doesn’t help a lot to have a cache that only lets an item exist in one place!
I found the problem, but wasn’t in pos’n to fix it. I could have gone up the ops management chain, filed an SCR (a hostile act) & waggled my finger all around. Instead, I called a tech lead I knew on the dev team. We sat down over the problem and figured out a fix. He started implementing the fix whilst I started change control processes to get the fix released.
Operator who spoke code. Dev who understand data center networking. Both could talk in terms of the cost of the bug. ($10M!)
Operator who spoke code. Dev who understand data center networking. Both could talk in terms of the cost of the bug. ($10M!)
Change in controls faster than controls take effect.
When feedback is acted on faster than controls take effect.
Difficult person to like. Abrasive, confrontational. Combative. (I know, sounds like every fighter pilot ever.) 40-second Boyd.
Get inside opponent’s turn radius.
Also autodidact. Took degree in aero eng. to develop understanding of why some planes win dogfights. Not top speed that matters, or max altitude. Not even the airspeed/altitude envelope. “Fast transients”
At Eglin, stole computer time to run calculations. Compared all planes in US service to all planes in USSR service. Embarrassed the USAF, FB-111 (proud new aircraft) strictly inferior to all USSR fighters under all conditions.
“fighter mafia”. f-16, f-15, f-18. pilot egress ladder. small things matter. f-16 thrust to weight ratio. blunt, small wings. high drag: counterintuitive! Seems inefficient. Edge of instability.
Don’t focus on destroying enemy equipment & people. Make it impossible for enemy to bring force to bear. Maneuvering is difficult. 6 hours for green unit, 1 hour for experienced. Requires practice. (Similar to deployment!)
Edge of instability. OODA, later development. anything with this many feedback loops will be unstable. outside this talk. just need EM for today.
How do we create a system that will exhibit emergent properties that we like, when it’s based on local action?
Old thinking: work by destroying the forces. Attrition until one side cannot continue.
Old thinking: work by destroying the forces. Attrition until one side cannot continue.
Maneuver says: don’t commit (get pinned) to only one objective. See where progress can be made, push harder there. Dislocate rather than destroy
Let progress “pull” more resources to that point. Opposite of usual corporate approach (pour more into failing ones
Let progress “pull” more resources to that point. Opposite of usual corporate approach (pour more into failing ones
I like “disposable infrastructure” rather than “immutable infrastructure.”
Probably what you think of as tempo. takt time from TPS/Lean. But takt time requires standardized work.
Control tempo == set pace of market releases. Intense but relaxed. If rushed, you are reacting.
Take initiative == force opponent to react to you. Nullify their advantages, but break out in different directions.
Ambiguity is threatening. Don’t let opponent know what you are doing. Dislocate opponent. Can't bomb, no machine guns. But can spoil features so they waste energy. Must implement as quickly, cheaper than, opponent (why Clojure!)
Also has tempo and initiative.
Time to set up camp for experienced unit 1/6 the time of a green unit.
"The army way" matters. Uniformity. Know where to find mess hall, latrines. Know where not to walk (microwave line-of-sight beam). Willing to change both methods and equipment to achieve higher tempo.
doesn’t really work like that. tempo is the result of your ability to maneuver. i’ve spent last 5 years helping paralyzed companies start to move faster. here’s what i’ve learned.
Fingerspitzengefühl, lateral visibility, instinct for “right action” in local context.
slow comms. strictly following the chain of command when time is critical. bad news. (air strike comes too late, team has moved into position.)
(Or: finally get all applications puppetized, but now devs have moved on to docker)
Direct comms during the action, with after-action review.
manager on the right hears about it later… that’s what he gets for not flattening out his org.
Crucial pre-requisite: common language, trust, common priorities.
Impediments: lack of uniform training, language. Dev fashion & underlying tech change causes geologic strata in management. (Except the higher you go, the older the sediments are.)
Key tool for vertical integration. Intent flows from theater to battalion to company to squad
S - Enemy Forces, Friendly Forces, Attachments/Detachment, Civil/Terrain considerations
M - Who, What(Tactical Task), Where, When, and (most importantly) Why?
E - Center of Gravity, Critical Vulnerability, Exploitation Plan, Desired Endstate, Scheme of Maneuver, Fire Support Plan, Tasks, Coordinating Instructions
A - Administration - "Bad Guys & Bandages” / Logistics - "Beans, Bullets, & Batteries”
C - Signal (pri, alt, second, emerg), Command (leaders, succession)
Initiative.
Sente = move that demands response (has initiative, determines rate of territory change)
Gote = must respond
Emergent properties. Keep the focus on independent autonomous action. Enable “right” action as the easy path.