DOES15 - Mike Bland - Pain Is Over, If You Want It
•Download as PPTX, PDF•
3 likes•1,265 views
Mike Bland, Practice Director, 18F Technology is always the easiest part of any problem. This was true of Google in 2005, when Mike Bland joined the Testing Grouplet’s effort to drive adoption of automated testing throughout a highly successful company as its organization and systems increased in complexity at an alarming and unstoppable rate. This was true in late 2013, when the Healthcare.gov crisis led to a stunningly successful recovery after private industry experts were given clearance to fix the technical issues. It is also true of the U.S federal government today, as Mike has joined 18F as part of the effort to modernize how software is developed and procured, and to steer the culture towards maximum transparency, autonomy and collaboration. This talk will outline Mike’s experiences at Google that shaped his outlook and honed his organizational skills, and describe his efforts to capitalize on the opportunity produced by the Healthcare.gov recovery to effect broad cultural change throughout the federal government.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to DOES15 - Mike Bland - Pain Is Over, If You Want It
Similar to DOES15 - Mike Bland - Pain Is Over, If You Want It (20)
More from Gene Kim
More from Gene Kim (20)
Recently uploaded
Recently uploaded (20)
DOES15 - Mike Bland - Pain Is Over, If You Want It
- 1. Pain Is Over, If You Want It by Mike Bland Practice Director, 18F 2015-10-19 Slide deck: http://goo.gl/CrCUii This presentation is licensed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. This work is derived from Large Scale Development Culture Change: Google and the US Government, which is Copyright 2014 Mike Bland, licensed under a Creative Commons Attribution 4.0 International License; and from Solving the Total Problem of Software Quality and Government Services, which is licensed under a Creative Commons CC0 1.0 Universal Public Domain Dedication.
- 2. October 2013
- 4. April 2014
- 5. So now what?
- 6. How I Learned To Stop Worrying and Love the Bomb...Again
- 7. Google 2005
- 8. Inexperience
- 9. Code gets added. Tools get slower. Builds take longer. Tests take forever. Code goes untested. Dependency cruft builds.
- 10. Large, infrequent changes frequently conflict. Builds break overnight. Emergency pushes common. Fear is the mind killer.
- 12. InertiaEnormous early success Overconfidence, arrogance, Impostor Syndrome Insecurity Inexperience, “My code is too hard to test” Ignorance Old tools, “I don’t have time to test.” Friction
- 13. (After-the-fact: goto fail; and Heartbleed) Impact of testing is impossible to measure a priori
- 14. Priority Structure If it can’t be measured, (e.g. more clicks) it doesn’t matter. (i.e. won’t get me promoted) Ignorance/Communication Breakdown
- 15. How does culture change?
- 16. Not like this…
- 17. Or like this… Beware of heroes, echo chambers Cultivate mythology as a useful model
- 18. What did we have to work with?
- 19. Transparency Employee directory, project database, wiki/Sites Freedom to experiment, 20% time Autonomy Grouplet system, startup ethos Collaboration
- 21. GWS tech lead Bharat Mediratta believed automated testing would help… …and it did.
- 22. Started by Bharat Mediratta and Nick Lesiecki Volunteers pooling 20% time to drive adoption of automated testing Testing Grouplet
- 23. Testing on the Toilet (TotT) Test Certified (TC) Test Mercenaries Ubiquitous, incremental exposure Clear, tangible path via measurement, policy, goals Hands-on help, tool adoption and advocacy
- 24. Company-wide events, usually one day long Address “important but not urgent” backlog Focus, motivation, concrete goals, free stuff Fixits
- 26. Rainbow of Death: Testing Grouplet Intervene Validate Inform Inspire EmpowerMentor Dependent Independent Fixits Test Certified Build Orbs Lectures TotT CodelabsTool development (w/ Testing Tech, Build Tools) Test Mercenaries Tech Talks Testing Grouplet All projects Test Certified Level 3 Revolution Fixit (build tools) Test Certified Mentors TAP Fixit (CI platform)
- 27. Google Stats 2013 via Eran Messeri 15,000 developers, working on 4K projects All code is checked into one source tree 5,500 code commits/day 75 million test cases are run daily
- 28. Power and knowledge to do the right thing Thorough automated testing now the norm Most breakages fixed before clients notice Less fear, more confidence, flow, and joy The Value to Developers
- 30. 18F 2015
- 31. 18F Open-source development, Agile methodologies Educate, reform procurement, not replace vendors Savings as a natural side-effect Founded March 2014 by Presidential Innovation Fellows
- 32. USCIS
- 33. Every Kid in a Park
- 37. Consulting
- 38. limiting perceived risk meeting regulatory requirements job security Internalization: Don’t rock the boat Priority Structure
- 39. Inertia No quality incentives, PCSRA, “successful company” people Avoid risk/“accountability”, “gov’t can’t attract talent” Insecurity Waterfall is familiar, testing is someone else’s job Ignorance Outdated tools/procedures, vendor lock-in of code, data Friction
- 40. Policy often mandated by nontechnical people Development teams disconnected from end users They don’t know what they don’t know Ignorance/Communication Breakdown
- 41. Employee directory Code browser Project data base Wiki EngEDU Codelabs First Day at Google, August 29, 2005 Tech Talks Snippets Objectives and Key Results 20% time Grouplets
- 42. Where are the docs? Who do I ask? What do I need to know? How do I get access to everything? Who’s on my team? Who’s working on what? How can I contribute? First Day at 18F, November 3, 2014
- 44. Working Groups and Guilds
- 45. 18F Pages 18F Guides 18F Edu
- 46. Rainbow of Death: 18F Intervene Validate Inform Inspire EmpowerMentor Dependent Independent 18F Consulting Success stories on 18F blog Hub 18F Delivery Discovery sprints 18F Guides 18F Edu Workshops 18F Blog: Useful Mythology Positive user experiences Digital Coalition (18F, USDS, CFPB…) Working Groups/Guilds Onboarding Revamp 18F Pages Gov’t-wide Hub Cross-agency collaboration Team API
- 47. TransparencyThe Hub, Team API, .about.yml 18F Pages, 18F Guides, 18F Edu Autonomy Working Groups and Guilds Collaboration
- 49. “Never doubt that a small group…” “Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it’s the only thing that ever has.” Margaret Mead
- 50. Will it succeed?Yes, if we want it to.
- 51. How can you help us?
- 53. None More Black https://18f.gsa.gov https://github.com/18F Slides: https://goo.gl/CrCUii
Editor's Notes
- Problem Healthcare.gov (2m) October 2013: [click] heathcare.gov is crushed under the load. On the one hand, there was high demand for the service. On the other, health care reform was in danger of falling apart because of a website? ---------------------------------------------------------------- Does that seem right to you? Image: http://b-i.forbesimg.com/theapothecary/files/2013/10/healthcare.gov-crash-1.png From: http://www.forbes.com/sites/theapothecary/2013/10/01/healthcare-gov-crashes-during-first-day-why-massachusetts-never-had-this-problem/ Any Strongbad fans? http://homestarrunner.com/sbemail45.html
- November 2013: [click, point at the knee in the graph, which is when the recovery team was brought in] Doors open to bring in industry experts to lead the recovery effort. ---------------------------------------------------------------- Original source: http://www.hhs.gov/digitalstrategy/sites/digitalstrategy/files/pdf/healthcare.gov-progress-report.pdf Image cropped from: http://cdn.arstechnica.net/wp-content/uploads/2013/12/aca-fixes.png Article: http://arstechnica.com/information-technology/2013/12/healthcare-gov-sort-of-fixed-good-enough-for-vast-majority/
- April 2014: [click, point at huge differential between November and December] Enrollment not only surpassed the revised 6 million target, or the original 7 million target, but it eventually reached 8 million. The website recovery was a roaring success, thanks to a dramatic change in the healthcare.gov tech culture. Once the administration decided it wanted to fix the problem, rather than adhere to decades of conventional contracting and procurement wisdom, it fixed the problem. ---------------------------------------------------------------- Image: http://www.newrepublic.com/sites/default/files/u18524/slide1_18.jpg Source: Jonathan Cohn, “Obamacare Signups Hit 8 Million...”, New Republic, April 17, 2014
- How does healthcare.gov stay fixed? How can the momentum of the recovery effort help to reform Government IT development culture in general? What could the federal government learn from the grassroots automated testing adoption effort at Google? And how did I get involved? I had left Google and the tech industry in September 2011, and was enrolled in Berklee College of Music when… ---------------------------------------------------------------- The healthcare.gov website recovery shook up the status quo of government technology. It produced both the opportunity and momentum to make a lasting improvement in how government IT development is done, by enacting a similar culture change across the whole of government.
- ...my former colleague Jason Huggins, who some of you may know by way of Selenium and Sauce Labs, had just found me on LinkedIn. He wrote: “As you may or may not be aware, I helped late last year with the healthcare.gov web site rescue…. “It's going to take awhile before the government gets good at software development (and testing) -- and they're going to need a culture change…the White House ‘gets’ it now, that fundamental change is needed in how they create and test software, and --more importantly-- that change imposed ‘top down’ is likely to fail…. “Again, no pressure…I respect your desire to focus on other things…. But, there's this whole ‘your country needs you’ thing. :-) So I figured I'd just put that out there for you to think about.” Being from a military town, he hit my big red button. I dropped out of Berklee and eventually joined 18F in November 2014. To understand why Jason recruited me... ---------------------------------------------------------------- “As you may or may not be aware, I helped late last year with the healthcare.gov web site rescue…. “It's going to take awhile before the government gets good at software development (and testing) -- and they're going to need a culture change. (You can start to guess why I'm writing you. :-) Luckily, the White House ‘gets’ it now, that fundamental change is needed in how they create and test software, and --more importantly-- that change imposed ‘top down’ is likely to fail. It needs to be grass-roots, with the appropriate amount of air cover from officials when necessary…. “There's lot's to chat about, and I know you're semi-retired, but I'm wondering if you'd ever be interested in talking about your Google experience to the teams now embedded on the various hc.gov projects. I'm specifically interested in bootstrapping something of a Tech Talk program for people with great experiences like yours to present to peers working on these gov projects…. “Again, no pressure. You're quite clear in your LinkedIn bio about jobs and networking. I respect your desire to focus on other things. (I've got my own art projects I desperately want to spend more time on...) But, there's this whole ‘your country needs you’ thing. :-) So I figured I'd just put that out there for you to think about.” (- Jason Huggins, private email, 2014-05-14) ---------------------------------------------------------------- Image from the film Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb: http://www.imdb.com/title/tt0057012/?ref_=nv_sr_1 Image: http://www.filmcaptures.com/wp-content/uploads/2013/08/Dr_Strangelove_8.jpg From: http://www.filmcaptures.com/dr-strangelove-or-how-i-learned-to-stop-worrying-and-love-the-bomb/
- Problem Google 2005 (5m, 28m remaining) let’s go back to when I joined Google in 2005. Everyone could see how successful the company was. Everybody wanted to work there. We had to be doing something right, and in a lot of ways, we were. But...
- …there was a side no one outside the company could see or really appreciate. We were at risk of collapsing under our own success. ---------------------------------------------------------------- Image: http://groundedpsyche.com/wp-content/uploads/2015/01/Iceberg.png From: http://groundedpsyche.com/iceberg/
- The size and complexity of our code base and products exploded with our growth. Though we did our best to hire the best developers we could and provide them with a supportive environment, no amount of brainy heroics could overcome the challenges of scale. ---------------------------------------------------------------- Quoth myself: https://mike-bland.com/2011/12/02/coding-and-testing-at-google-2006-vs-2011.html
- We were at risk of holding ourselves back, slowing ourselves down, once we reached the point... [click] …when fear of change and all the things that might go wrong stifles courage and innovation, and leads to missed opportunities, bureaucracy, petty empire-building, mediocrity, and irrelevance. There was no project more visibly susceptible to these forces than… ---------------------------------------------------------------- “Fear is the mind killer.” is of course a quote from Dune by Frank Herbert.
- …the Google Web Server team (GWS for short, pronounced “gwiss”, like “Swiss”), the team responsible for the google.com home page. I didn’t work on the GWS team, but was close with many of its members, for reasons that will become clear. The thing was, in the beginning, GWS was not such a glamorous assignment. It was a dumping ground for lots of unrelated changes as search features were developed by different teams. Imagine if you break google.com…or allow it to be broken. Search results might be broken, or unacceptably slow to return. Thousands of queries per second turn to thousands of unfulfilled promises, and the business loses not only revenue, but trust. However, at the same time… ---------------------------------------------------------------- Source: http://blogoscoped.com/archive/2006-04-21-n63.html http://blogoscoped.com/files/google-com-history/2005.jpg
- …to many folks, things seemed to be working, for the most part. [click] Some may have believed their own hype, and latecomers like me were scared as hell that we’d be revealed as the frauds that we were. [click] Also remember, automated testing was not the common sense practice many of us take for granted today. People didn’t know how to write tests or testable code. [click] And the development tools we were using at the time were reaching their limits.
- But by far, the biggest force working against us was the notoriously difficult problem of measuring negative impact, or the value derived from avoiding problems rather than fixing them as they arise. [click] We can point in retrospect to notable examples where automated testing discipline might’ve avoided catastrophic bugs. But people tend to underestimate the chances of such things happening to them… ---------------------------------------------------------------- Yes, I claim that it is “impossible”, not just “difficult”; I have a section of a blog post explaining the futility of collecting “productivity” metrics. Of course, I’d be delighted to be proven wrong somehow, someday, so call this the “Bland Conjecture” if you want.
- …especially when the core of their value system relies almost entirely on objective measurements. By “priority structure” I’m talking about cultural norm of what’s most important to an organization, it’s “corporate religion”, if you will. Now, Google’s obsession with data-driven decision making is extremely valuable, and has obviously served the company (and society) extremely well. But it made it extremely difficult to convince people that an investment in automated testing would pay dividends. And we couldn’t really blame them. [click] People for the most part had no experience with testing outside of the slowness and brittleness of the status quo, and were under constant delivery pressure. Given our inability to communicate using the language of data, they couldn’t understand what we were on about… …and who could blame people for not trying to test when they couldn’t afford the time to learn? We had to find other ways. ---------------------------------------------------------------- This is a popular misquote of W. Edwards Deming, the management expert who turned post-WWII Japan into an industrial giant, who actually said something like: “Even what cannot be measured must be managed.” From Gene Kim’s notes: (Google’s measurement focus actually made it more difficult to improve testing! “show us the data that testing will improve things”)
- Before getting into specifics, let’s examine how cultures change in general. One thing we know...
- There’s nothing worse than Cartman with authoritah. That is to say, mandates will fail, no matter how good the idea. Google executives learned this at Microsoft, Digital, Bell Labs, Sun, etc. It also doesn’t happen like this… ---------------------------------------------------------------- Image: http://ecx.images-amazon.com/images/I/41l-SWkjXEL.jpg From: http://shop.southparkstudios.com/SOUTH-PARK-CARTMAN-POSTER-You-will/A/B00302A3OI.htm
- No rockstar guru ninja is gonna come save the day. It is wasteful at best and dangerous at worst to assume change is possible only through the magic and charisma of a selected few. Power and mythology in and of themselves are not bad things, but they require deliberate and constructive cultivation. A useful mythology is repeatable at some level, in other words… [click] …a model. We can’t repeat the exact steps others have taken in the past, but we can use their stories to inspire our present course. ---------------------------------------------------------------- Image taken from http://ethunter1.blogspot.com/2012/05/whoops.html
- Solution Google 2010 (10m, 23m remaining) Despite the limitations and challenges we faced, we had a lot going for us by virtue of the existing culture and environment within Google.
- [Take more time] We had access to information, and tools that facilitated discovery. We were empowered to develop and pursue a vision. And there was enormous freedom to combine forces with like-minded folks to pursue a common goal. We didn’t realize it at the time, but we were also adhering to a very common model…
- …described by Geoffrey A. Moore in Crossing the Chasm. Different people react differently to different forces and stimuli, and to different methods of persuasion. Innovators and Early Adopters are like-minded seekers. The Early Majority is sympathetic to change, but need accessible tools and procedures. The Late Majority will go along with whatever seems to work for the Early Majority. Laggards are hopeless; disregard them. Getting the right messaging to the right people in the right order is key, and crossing the “chasm” between the Early Adopters and the Early Majority is what makes or breaks an initiative. Once the chasm has been crossed, adoption will spread to the other segments of the population. We’ll see shortly what this bridge across the chasm is comprised of. But first, let’s revisit the GWS example. ---------------------------------------------------------------- Image by Catherine Laplace based on other illustrations of the “Crossing the Chasm” model and Albert Wong’s framework image, discussed in the upcoming slides.
- The tech lead of the GWS team, Bharat Mediratta, believed automated testing would go a long ways towards curing the ills of complexity and fragility. So the team took a hard line: No changes would be accepted into GWS without accompanying tests. They set up a continuous build and religiously kept it passing. They set up coverage monitoring and ensured that their level of coverage went up over time. They wrote a policy, and testing guides, and insisted that contributors both inside and outside the team followed them. Despite the initial unpopularity this policy, the GWS team held firm... [click] …and eventually turned a corner. They became one of the most productive teams in the company, integrating large numbers of changes from different teams per week while maintaining a brisk release schedule. New team members made productive contributions to this complex system quickly, thanks to good test coverage and code health. Ultimately, this radical policy enabled the google.com home page to expand its capabilities rapidly and thrive in an amazingly fast-moving and competitive technology landscape. It goes without saying that GWS was a model of automated testing effectiveness, but GWS was still a relatively small team in a large and growing company. We had to amplify its message somehow, and build that bridge across the chasm. ---------------------------------------------------------------- Gene’s notes: Google.com: team lead was testing grouplet lead; death march (GWS) took hard line: no changes that didn’t have tests over time, better test coverage led Grouplet: model dev team (GWS) went from fragile/fear: fewer rollback, more changes, could accept more changes everyone needed to integrate their changes into frontend SOLUTION (Quoth me: http://martinfowler.com/articles/testing-culture.html) Over time, unit test coverage and development momentum went up, while defect, production rollback, and emergency release counts went down. New team members found themselves becoming productive far more quickly because the tests allowed them to gain a deeper perspective on a system one unit at a time, and to begin contributing changes with the confidence that the existing tests would likely detect any unexpected side-effects. Any tests they caused to fail in the course of their early efforts accelerated their grasp of the system. Experienced members of the team, who had grown cautious of making changes and accepting changes from contributors, were able to make and accept changes quickly for the same reason and no longer had to rely primarily upon large and expensive system or manual tests with feedback cycles on the order of hours or days. Adding more new developers actually allowed the team to move faster and do more, avoiding the scenario described by Brooks's Law in which "adding manpower to a late software project makes it later".
- Bharat partnered with Nick Lesiecki to form the Testing Grouplet, the automated testing Instigators within Google. I eventually became one of the co-leads of the grouplet as well. We had very little budget and zero authoritah, but we had the freedom to explore creative solutions to the problems facing automated testing adoption, from many different angles, often relying on the GWS experience as a model. We worked closely with EngEDU, our in-house training organization, to develop in-house training materials, a new hire lecture and lab, and tech talks. We worked closely with tools teams to reduce the friction that produced the “I don’t have time to test” excuse. Our biggest breakthrough, however, was... ---------------------------------------------------------------- 20% time was a tradition whereby any developer could spend roughly one day a week working on a Google-related project other than his/her primary project. Grouplets (aka Intergrouplets) were teams volunteering their 20% time to solve company-wide problems together. We operated under our own direction; there were no marching orders from the top, but there were no explicit constraints placed upon us, either. Image Source: http://www.flickr.com/photos/dullhunk/3712840085/ Image License: CC BY 2.0 https://creativecommons.org/licenses/by/2.0/ Logo and slogan by Johannes Henkel; small contribution from yours truly in the form of tragedy/comedy mask suggestion.
- TotT, our weekly testing periodical. By publishing an episode each week in nearly every bathroom in nearly every office worldwide (outlined in my TotT blog post) we were able to gradually raise the degree of testing knowledge and sophistication throughout the company. It’s doubtful an online-only publication would’ve involved people to the same degree. I chose this episode for this slide not just because I wrote it (ahem), but because it happened to encapsulate two other significant Testing Grouplet initiatives. The first of which being... [click] Test Certified, which was a roadmap designed to do two things: to hack the measurement-focused priorities of the culture (by providing measurable tasks, levels, and a “ladder” of teams); and to overcome the first, scary obstacle of not knowing where or how to start. Level one: rapidly establishing baseline metrics Level two: setting a policy and reaching for early coverage goals Level three: striving towards suitable long-term coverage goals We also provided volunteer “mentors” to each team who would provide advice and ultimately “validate” their progress in climbing the “TC Ladder”. And it became our goal to get every team to TC Level 3--whether they were enrolled in the program or not. [click] The Test Mercenaries, of which I was also a member, were “internal consultants” that would help teams improve their testing practices, applying our tools and techniques to the team’s own code, using Test Certified as both a guide and a goal. Our close collaboration with the tools teams, providing feedback as we applied new tools to challenging projects, fueled the innovation that would culminate in the tool suite that removed the “I don’t have time to test” excuse. The Testing Grouplet also organized a series of large-scale events called… ---------------------------------------------------------------- TotT was the most powerful and successful communication channel, and continues to be so to this day. In fact, thanks to friends on the inside, I recently became the first “outsider” to publish not one, but two TotT episodes: one for “goto fail”; and one for Heartbleed. It helped eliminate the “echo chamber” effect. (Ironic, considering the reflective nature of bathroom surfaces.) Image taken from http://www.flickr.com/photos/dullhunk/1019220380/ of an episode by yours truly. ---------------------------------------------------------------- TotT helped standardize the use of Small/Medium/Large and other concepts throughout the company, and the feedback we received from readers was critical to ensuring our message was relevant and useful to the company as a whole. As Antoine Picard points out, TotT injected automated testing into everyone’s awareness from day one, even before they had a chance to attend the unit testing lecture and lab. Nick Lesiecki: “TC levels ‘hack’ the internal priorities of the culture. Testing wasn't measurable, so we created [snip “5”] measurable milestones that people could engage with and brag about at perf time. Sure the measure wasn't as smooth as ‘clicks’ but what wasn't measurable became measurable.” Nick Lesiecki: “The Grouplet advocated and got funding for a staffed team of internal consultants. This [was] an important step. Management was getting involved, but not with edicts. They were demonstrating the importance of testing by funding a consultancy to train people.” We were the “boots on the ground” practicing what the Grouplet preached, and we were also able to influence internal tool development. Once the Testing Tech and Build Tools teams produced promising new tools the Test Mercenaries could then drive adoption of these improved tools and the practices they enabled.
- These were neither mandated nor approved by the executive hierarchy; they were entirely self-organized affairs. They addressed “important but not urgent” issues such as writing tests, or fixing broken tests, or moving up the Test Certified ladder. They were also an incredibly powerful and efficient means of rolling out new development tools to the whole company all at once. The power of Fixits came from the fact that they were focused on specific goals within an ambitious time frame. This urgency helped generate a critical mass of effort that ratcheted up the state-of-the-art in tools and techniques and drove the culture change mission to a new plateau each time. Plus, they were a lot of fun, and we got to give out a bunch of free stuff. Googlers love free stuff. ---------------------------------------------------------------- My scan of a physical copy of an episode by Antoine Picard announcing the second company-wide Fixit I organized. That said, when business-critical emergencies do arise, the execs will order a focused emergency effort, but that’s a wholly separate phenomenon from grassroots events such as these. The power of Fixits came from the fact that they made it easy for both organizers and participants to take specific action, in advance and during. Plus there was plenty of free stuff, recognition, and perf material: Three things Googlers cherish. At a higher level, Fixits provided missions at points in time that generated excitement and energy, which helped advance the state-of-the-art. The long-term culture change mission reached a new plateau with every big, visible effort.
- Let’s take the pieces of the Testing Grouplet puzzle that we built over the course of five years and see how we built a bridge across the chasm.
- This is a model I borrowed from fellow Ex-Googler/current VA Digital Service member Albert Wong, which he derived from a two-week sprint with the US Citizenship and Immigration Services in July of 2014. The actual framework is all Albert’s, but I gave it the cute name. :-) It nicely delineates the different functions necessary to migrate an initiative from one side of the chasm to the other, in terms of the needs of the Early and Late Majorities. [click] The progression of concepts is also pleasantly linear, as the activities serve to transition people from dependency on experts towards empowerment. Bear in mind that many of the Testing Grouplet activities that I’m about to reveal spread across different functions, not just the ones I’ve listed them under here. [click 4x] Build Orbs were “information radiators” that would provide a conspicuous visual indication of the state of a team’s continuous build. [click 2x] The Revolution Fixit in January 2008 served to overcome the “I don’t have time to test” problem by empowering developers with new tools that could build our code more quickly (i.e. it removed friction). The Test Automation Platform (TAP) Fixit in March 2010 built upon these tools to produce a continuous integration platform that could test every change and clearly indicate the source of a breakage, often so fast that breakages were already reported and fixed before most people noticed. ---------------------------------------------------------------- Testing Grouplet members would often hold “orb-building parties” to construct these devices, which we’d eventually award to teams achieving Test Certified Level One status. T-shirts are the fru-its of the dev-eel. Applying a suite of build tools to set up a continuous build system for my first Test Mercenaries engagement inspired the “Revolution Fixit”, the event that spread adoption of these tools throughout Google and provided the foundation for the development of the Test Automation Platform (TAP), Google’s company-wide continuous integration system. The core Testing Grouplet members built relationships that lasted well beyond any fixit or Test Certified mentorship or Test Mercenary engagement.
- Eran Messeri, GOTOcon Aarhus, ‘What Goes Wrong When Thousands of Engineers Share the Same Continuous Build?’, (2013), http://scribes.tweetscriber.com/realgenekim/206.
- That’s not to say there isn’t plenty of room for improvement; but instead of arguing whether or not developers should be writing automated tests, the debate (as I understand it now) is over how best to do it. But it is true that the fear is largely gone, and Googlers today enjoy seeing tangible progress towards exciting new milestones without being held back by chronic outbreaks of high-priority bugs. After years of grassroots teamwork… ---------------------------------------------------------------- Quoth myself: http://martinfowler.com/articles/testing-culture.html Furthermore, the mitigation of fear led to the expansion of their joy in programming, as they could see tangible progress being made towards exciting new milestones without being held back by chronic outbreaks of high-priority bugs. The impact on productivity of high morale, based on the ability to remain in a state of creative flow, cannot be overstated. While I was at Google, the GWS Team exhibited the ideal testing culture, integrating an enormous number of complex changes from outside contributors while making their own constant improvements.
- …we done the impossible, and that made us mighty. I find Caravaggio’s depiction of the tale of David and Goliath particularly compelling and relevant to this story because it is a self-portrait of the artist...as Golaith. There was no outside force resisting us, and the technical problems were the easier ones to solve. We had to provide the knowledge and power necessary to change perceptions and provide experience, because often the greatest obstacle to the change we wish to see in the world is the way we, as individuals, teams, and entire organizations, already see it. And that brings us back to... ---------------------------------------------------------------- “David with the Head of Golaith”, Michelangelo Merisi da Caravaggio, c.1610 Image: http://upload.wikimedia.org/wikipedia/commons/6/60/Caravaggio_-_David_con_la_testa_di_Golia.jpg From: http://en.wikipedia.org/wiki/David_with_the_Head_of_Goliath
- Mission At 18F (10m, 13m remaining) First, a little about how we came to be, who we are, and what we do.
- 18F was created within the General Services Administration to capitalize on the momentum generated by the Healthcare.gov recovery to reform how the government builds and buys software. We’ve grown from about seventy members at the time I joined to about 150 today. As for the name...after dozens of ideas that were already trademarked, the founders just went with the cross streets of the GSA headquarters. Image: https://18f.gsa.gov/images/logo-18f.png
- 18F is working with the US Digital Service and the Citizenship and Immigration Services team, under the leadership of last year’s speaker Mark Schwartz, to improve the USCIS system’s software architecture, delivery process, and user experience for prospective citizens.
- Every Kid in a Park, which we developed for the Department of the Interior, is notable for providing a well-researched website experience for kids. For example, you won’t find tons of social media buttons on this site, since nine-year-olds aren’t eligible for social media accounts.
- College Scorecard makes large quantities of data from the Department of Education accessible and useful to prospective college students, so they and their families can make informed decisions about where to attend. The Web Design Standards, a joint project between USDS and 18F, aims to provide for a much improved user experience across US government websites by transforming them from special snowflakes…
- …as represented by these actual buttons from government websites...
- ... by providing design elements and a style guide for a common look and feel.
- Our Consulting wing provides partner agencies a taste for agile processes within their own organizations by performing discovery sprints, delivering recommendations and prototypes, and hosting agile workshops. We’re off to a great start, but how can we sustain our own own momentum, and avoid becoming another promising experiment that didn’t pan out? First, let’s identify the organizational forces challenging our mission. ---------------------------------------------------------------- We’re doing a lot of good work on many fronts, showing the government how effective technology requires a shift in perspective from traditional procurement vehicles and waterfall-style project schedules to a modern, agile approach in which the government itself is an informed and engaged participant. In the process, how can we share our tools and working models so other digital service teams within the government can find a solid footing participate in driving this whole movement forward?
- A premium is placed upon compliance with the existing rules, rather than focusing on the quality of products and services.
- There are no real personal incentives like there are in the private sector, and the Pendleton Civil Service Reform Act of 1883, one big source of job security, serves to attract what Jamie Zawinski might call “people who want to work for a successful company, not people who want to work to make a company successful.” [click 3x…] All of these processes and obstacles are a response to the fear of something going wrong and being held accountable, which people believe translates to getting fired. And as we’ve observed before, fear is the mind killer, leading to missed opportunities, mediocrity, irrelevance… Consequently...
- …on every side of this situation, from the policy makers to the administrative staff to the developers, nobody has access to the full information needed to ensure a quality product and experience for end users. [click] This is not just ignorance, but also a communication breakdown, since these isolated groups lack a common goal, a common vocabulary, and common understanding of everyone’s needs and objectives. This is the abject absence of transparency, autonomy, and collaboration. If 18F is to overcome these challenges, we need to build an organization as robust as Google, to withstand these pressures and to communicate a better way of working. However, we have a long way to go. To illustrate this, let’s examine...
- …the experience of my first day at Google. I was a bright-eyed new hire ready to jump at the chance to start doing some good in the world. At Google, I felt like I had the world already at my fingertips; I describe it as jumping in the fire to drink from the hose. On my first day at 18F, however…
- …I had to really dig and ask around for answers. Not only did that use up my time and energy, but the time and energy of the folks that had to stop and answer me. I realized that everything I did at Google was based upon a combined platform of values and the technology to support those values that already existed before I joined. Everything the Testing Grouplet did was built on a foundation that we mostly took for granted. We needed a similar foundation at 18F, and it didn’t exist yet. So what did I do? I began to steal all the best ideas from Google that I could, and adapted them to suit this new environment. ---------------------------------------------------------------- That Google got this part right early on is a woefully undervalued, undersung component of its overall success story.
- The Hub organizes our team-wide documentation and enables us to explore the connections between people, projects, and skill sets. As an amalgam of systems I had experience with at Google, it has made the point that effective documentation and dissemination of information is vital to our continued success as an organization. [click] The Team API is the data engine extracted from the Hub that takes metadata about our people and activities and joins it all together in a cross-linked JSON API, the very core of our graph between people, projects, and skill sets. [click] It gets a big chunk of its data from our .about.yml files, which are metadata files that live in each repository on GitHub. We’ve begun automatically harvesting updates into the Team API, from which it gets published on our project Dashboard, eventually the Hub, and who knows where. By scaling up these systems and making them more accessible to 18F team members, I aim to create the space where Instigators can discover one another, take initiative and band together to create… ---------------------------------------------------------------- Up to this point the hub has been a 100% static site built using Jekyll, nginx, and the bitly/oauth2_proxy for authentication. https://github.com/18F/hub Public 18F Hub: https://18f.gsa.gov/hub/
- Working groups and guilds, our version of “grouplets”. I’ve organized and co-organized three: the Documentation WG, the Testing Grouplet, and the Working Group Working Group (to help cultivate Working Group and Guild guidelines and tools). To make it easier for our team members to document their knowledge and share it with the rest of 18F, as well as with the broader public… ---------------------------------------------------------------- The primary distinction are that guilds have official endorsement from our leadership team, and as such are directly responsible for delivering outcomes, whereas working groups can spin up or down without explicit sponsorship. They empower people across the team to self-organize in order to address the issues they’re most passionate about, pooling their insights and experiences for dissemination across the team, contributing to the overall health of the organization.
- …we’ve launched 18F Pages, a GitHub Pages-like publishing system that runs on government infrastructure, and 18F Guides, a documentation series highlighting how we work. 18F Guides are works-in-progress, but have sparked discussion both within the team and with members of the broader government tech community. Via GitHub, we’ve received lots of feedback and direct contributions from outside our team that have improved our materials and made our dream of increased government engagement come true. 18F Edu is our newest initiative, just coming online, that I hope will encompass the same scope as Google’s EngEDU when it comes to training materials and programs, and become the permanent stewards of our 18F Guides and other materials. Now let’s see how these pieces we’ve developed so far fit together…
- Some other items to call out here that I haven’t yet mentioned are: we’ve completely overhauled our onboarding process to meet the demands of our rapid growth our network with the US Digital Service team (part of the White House), the Consumer Financial Protection Bureau, and other agency teams continues to grow, and I dream of one day surfacing this entire community via a government-wide Hub, linking together our individual Team APIs and Guides to maximize opportunities for cross-agency knowledge-sharing and collaboration I’ve only been able to scratch the surface, here, as there’s a lot more going on beyond what I’m personally involved in. But the point of developing these tools and processes and telling you about them is that…
- ...the insights, methods, and products generated from the combination of transparency, autonomy, and collaboration are what empower a team to produce a product or deliver a service that not only satisfies expectations of customers and society at large, but exceeds them.
- The right people are in the right place at the right time doing the right things for the right reasons. ---------------------------------------------------------------- Image http://media2.s-nbcnews.com/i/MSNBC/Components/Video/__NEW/tdy_beatles_140127.jpg from http://www.today.com/entertainment/tag/the-beatles.
- [click] Charles Worthington, Former Senior Advisor to the US CTO: “If you think there is a problem with how government does tech, and that you could help government do tech better, then the question is what are you doing to improve it? It's not going to get fixed on its own, and the people in charge have never been more open to new ideas. “If we don't try, who else are we expecting to?” ---------------------------------------------------------------- Charles W.’s entire comment: “The Government is not this big monolithic other thing that just exists in a big building closed off to everyone. It is made up of Americans who have chosen to work in public service... i.e. to help make other American's lives better. “If you think there is a problem with how government does tech, and that you could help government do tech better, then the question is what are you doing to improve it? It's not going to get fixed on its own, and the people in charge have never been more open to new ideas. “If we don't try, who else are we expecting to? “Some percentage of America's top lawyers choose to work at the Dept. of Justice prosecuting bad actors instead of corporate law firms. “Some percentage of America's top scientists choose to work at the NIH/CDC fighting infectious diseases instead of at a big pharma company. “Where are the top technologists working to build awesome digital services for the American people?” Michael Byrne, CFPB: “i think here might be a good place to add challenges still a head. one of those key things is technology for technologies sake doesn't do anything in government. in the private sector success is determined by sale of units. not so in the government. there aren't the same clear metrics in government. success is not a pretty web page. if the ACA had a working web site at day 1, but the issue was about say, how far off the atomic clock is from an iPhone clock, is it the same success? success in government is 100 pct about getting the policy right. in the ACA case, the right policy is a combination of policy about healthcare AND getting people to sign up for it. not every high goaled policy effort in government has the same implementation and we need to recognize that workable web sites don't solve every problem. challenges a head help keep our efforts in check. what does the next administration look like? what does the next congress look like? what if there is a new great recession? what if the virus (e.g. ebola, or cpog or something else) causes a new world scare? what if there is a new 911? what if there is a student revolt to FARM, or common core? these policy issues do not have the same approaches to tech development as ACA, so we need to make sure we are agile enough to know what methods support the policy. in many many cases the policy answer is so unclear that the debate needs to be fostered (e.g. education about a plan), rather than here is the implementation. in all cases, there isn't the same metric (7 million people signed up only works for ACA). i might say that a core issue here is gov metrics is WAY hard.”
- Gene told me that it’s customary to ask the DevOps Enterprise audience for assistance in solving our most pressing issue. Now, as a government employee, it’s unethical for me to directly solicit unpaid effort from the private sector; but I can ask that you…
- Validate our efforts, Inform the government of their value, and Inspire change by helping shine a light on the great work 18F and other teams are starting to do across the government. Write blog posts and articles that highlight what we’re doing right, and how we might do better. Comment on and contribute to our GitHub repositories if so inspired. Follow the 18F blog and Twitter account and give us a shout-out whenever we get something right. Help amplify the voice of our small team so that we may build that bridge across the chasm. All of that will help empower 18F and the broader Digital Coalition to make government of the people and by the people work better for the people.