My presentation at Monktoberfest 2016. Video at https://www.youtube.com/watch?v=4PaWFYm0kEw

1. Oral tradition in software engineering:
Passing the craft across generations
CTO
bryan@joyent.com
Bryan Cantrill
@bcantrill

2. Oral tradition
• Long before humans developed systems of writing, we were
leaving legacies of information via oral tradition
• Oral tradition has been used to:
• Preserve and explain cultural values
• Convey skills and wisdom
• Entertain as well as enlighten
• Oral tradition continues to play an important role in post-
literate societies — including in software engineering

3. Oral tradition in software engineering
• In the beginning, software was subservient to hardware
• Software rarely enjoyed the levels of design documentation
found in computer design
• Like true oral tradition, it relied primarily upon oral
retransmission — and much of this is lost today
• Fred Brooks’ Mythical Man Month captures the tribal wisdom
of this antediluvian era of software

4. Oral literature
• When oral tradition is distilled into clear narrative and reliably
repeated, it becomes oral literature
• For us in software engineering, the early Internet allowed a
way to reliably share stories via USENET
• USENET is filled with incredible and interesting narrative, but
most survives only in archive
• But one story from USENET does survive — and it has
become the epic of software engineering…

5. Our epic, as penned by Ed Nather
A recent article devoted to the macho side of programming
made the bald and unvarnished statement:
Real Programmers write in FORTRAN.
Maybe they do now,
in this decadent era of
Lite beer, hand calculators, and “user-friendly” software
but back in the Good Old Days,
when the term “software” sounded funny
and Real Computers were made out of drums and vacuum tubes,
Real Programmers wrote in machine code.
Not FORTRAN. Not RATFOR. Not, even, assembly language.
Machine Code.
Raw, unadorned, inscrutable hexadecimal numbers.
Directly.

6. …and our epic hero, Mel Kaye
Lest a whole new generation of programmers
grow up in ignorance of this glorious past,
I feel duty-bound to describe,
as best I can through the generation gap,
how a Real Programmer wrote code.
I'll call him Mel,
because that was his name.

7. Our Odyssey
• The Story of Mel was recorded in the early 1980s, but dates
from the earliest days of software in the 1960s
• Another set of stories recorded in the late 1970s at Data
General has become software engineering lore…
• Tracy Kidder’s Soul of a New Machine has wisdom that has
long survived the team and company it describes
• If Mel is our Illiad, Soul is our Odyssey

8. The 1980s and 1990s
• In the deeply proprietary era of the 1980s and 1990s, oral
tradition was locked on proprietary islands — tribes privately
passed on their own traditions and practices
• Collections of oral tradition like sgi.badattitude were the stuff
of legend — but they too have been lost
• The canonical folklore from this era may be John Browne’s
beautiful The Bug Count Also Rises, the winner of an internal
Microsoft Hemingway parody contest (!)

9. Dawn of a new era: November 7, 2005

10. Oral tradition post-YouTube
• With the rise of Internet video, story — orally delivered and
aurally received — could travel unencumbered
• Stories could be suddenly retold simply by sharing a link —
with all of the force and authority of the original
• Oral tradition can spread so quickly, we explain it in terms of a
pandemic: stories that become viral
• We in software were particularly lucky in that Google Tech
Talks managed to capture much oral history and tradition
before participants recognized what it meant to be recorded

11. For me personally, a watershed

12. For me personally, a watershed (ahem!)
DTrace Review

13. Oral tradition post-YouTube
• Post-YouTube, our oral tradition is suffering from overload —
we have an embarrassment of riches!
• The true oral tradition has shifted from the content to the
meta-content, e.g. this evergreen topic on HN:

14. Oral tradition closer to home
• Every source base is awash in narrative: of origin, of struggle,
of conflict, of rewrite, of success and of failure
• These narratives ultimately convey not only the history but
also the values of a source base
• In the proprietary era, these were purely oral tradition —
something to regale new hires with over lunch
• Every source base is loaded with these; several that are
relevant to mine… (UNIX Unix 6th Edition 7th Edition SunOS
System III SVR4 Solaris OpenSolaris illumos/SmartOS!)

15. I: The untimely death of Joe Ossanna
• In 1973, Joe Ossanna at Bell Labs wrote troff in PDP-11
assembler for the Graphic Systems CAT typesetter
• In 1975, Ossanna ported it to C — with the intent of a later
clean rewrite to make the macro language less alien
• In 1977, Ossanna died suddenly, of either a heart attack, a
car accident or (most graphically) being struck by a car
• The code he left behind — which remains in our source base
— is inscrutable; Brian Kernighan failed to rewrite it!
• Code survives us

16. II: “You are not expected to understand this”
• Perhaps the most (in)famous comment in Sixth Edition:
/*
* If the new process paused because it was
* swapped out, set the stack level to the last call
* to savu(u_ssav). This means that the return
* which is executed immediately after the call to aretu
* actually returns from the last routine which did
* the savu.
*
* You are not expected to understand this.
*/
if(rp->p_flag&SSWAP) {
rp->p_flag =& ~SSWAP;
aretu(u.u_ssav);
}
• That which we can’t explain, we don’t understand!

17. III: Logical XOR
• I have always been enraged disappointed by C’s lack of
logical XOR, having had a hard time holding my tongue, e.g.:
/*
* Logical XOR, where are you?
*/
ASSERT((vstate->dtvs_nglobals == 0) ^ (vstate->dtvs_globals != NULL));
• I became involved in a heated argument with Roger Faulkner,
who insisted that Dennis didn’t add logical XOR because “it
can’t short-circuit” — which is obviously stupid
• Important difference between Roger and me: Roger had
worked at Bell Labs with Dennis…

18. III: “Isn’t that much worse to write” (!)
From dmr@plan9.bell-labs.com Sun Aug 6 23:33:54 2000
To: roger.Faulkner@eng.sun.com
Subject: Re: history question
Hi Roger,
back from some vacation; here's a belated response.
^^ was omitted because it is of little use, particularly
because the "shortcut evaluation" guaranteed for || and &&
doesn't help--both sides have to be evaluated no
matter what. When you do need the effect,
(A!=0) ^ (B!=0) isn't that much worse to write.
Dennis

19. IV: “You think this is cheesy?”
• A favorite comment in the operating system is one that Jeff
Bonwick and I wrote together in a moment of crisis
• We had (finally!) figured a narrow path by which we could
solve a devastating problem that seemed insurmountable
• The details are highly technical, but documented thoroughly
in code comments — the context around it in an abstruse
chapter I wrote in Beautiful Code…

20. IV: “You think this is cheesy?”
/*
* The turnstile hash table is partitioned into two halves: the lower half
* is used for upimutextab[] locks, the upper half for everything else.
* The reason for the distinction is that SOBJ_USER_PI locks present a
* unique problem: the upimutextab[] lock passed to turnstile_block()
* cannot be dropped until the calling thread has blocked on its
* SOBJ_USER_PI lock and willed its priority down the blocking chain.
* At that point, the caller's t_lockp will be one of the turnstile locks.
* If mutex_exit() discovers that the upimutextab[] lock has waiters, it
* must wake them, which forces a lock ordering on us: the turnstile lock
* for the upimutextab[] lock will be acquired in mutex_vector_exit(),
* which will eventually call into turnstile_pi_waive(), which will then
* acquire the caller's thread lock, which in this case is the turnstile
* lock for the SOBJ_USER_PI lock. In general, when two turnstile locks
* must be held at the same time, the lock order must be the address order.
* Therefore, to prevent deadlock in turnstile_pi_waive(), we must ensure
* that upimutextab[] locks *always* hash to lower addresses than any
* other locks. You think this is cheesy? Let's see you do better.
*/

21. Code comments as oral tradition?
• The best comments seem to have an aspect of oral tradition
to them: they not only serve to explain what is going on, but
provide a narrative as to why
• Comments live with the code but do not otherwise affect its
correctness — allowing for broad scope and artistic license
• Relevant, correct software systems will survive long past their
creators; comments are the art within the timeless machine
• Comments are how we speak to future generations!

22. Oral tradition in software engineering
• Software engineering has changed radically in the last
decade: it’s now open, transparent and distributed
• This has forced a revolution in oral tradition, which has
increasingly become recorded, yielding oral literature
• Oral literature allows us to expose and preserve the narrative
behind the artifact — to explain the why of the thing
• Software engineering teams should be encouraged to record
their oral tradition: it attracts technologists who share values!
• Our oral tradition is part of our legacy to future generations!