Starts with lean product development for cars, in Japan, circa 1990. Then exposes the details of continuous delivery and service architectures, which are boomeranging back to Japan from the US tech scene.Includes some of the lean philosophy that makes it such a great way to deliver a valuable product. Presentation at from a Perforce / Toyo event in Tokyo, April 2015. Includes English and Japanese. You can just skip the Japanese slides.
3. What we can learn from Japanese Lean Manufacturing
• Lean principles
• Continuous Integration
What Japan can learn from Silicon Valley
• Continuous delivery
• Continuous agile product development
• Building large systems
The future
• Automation of human programming tasks
• Ecosystems and Testbeds
• New type of Keiretsu
23. Scrum Sprint
Validate and
Sort Deliverables
Select a
Sprint Plan
Completed
= observed
velocity
Expand stories
and estimate
tasks
Collect
Ideas
Close Sprint
Into next
sprint
Work on
tasks
25. Scrumban Iteration
Validate and
Sort Deliverables
Pull Deliverables
When Ready
Collect
Ideas
Stabilize
Release
Into next
release
Feature
Freeze
Triage
29. Code Contribution Patterns
Manage code if possible. People are hard to manage and
can’t be automated. They want to contribute.
• Centralized continuous delivery
– No branches, finds and fixes problems as early as possible
• Distributed continuous delivery
– Release every change with its own branch and test
• Temporary branches
– Combines benefits of centralized and distributed
• MAXOS
– Use centralized continuous integration to manage a
massively scalable IT system
37. The big question: How to test?
• We release software in batches so that we can test it.
• In continuous delivery, we might get as little at 10
minutes to test a release candidate
39. Test Layering
Monitor your released software: Errors, Usage
volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to
ask for test scripts.
Continuous integration: Run automated tests
before using human review time
Unit tests in the development environment
Switch new features and architecture
More frequent releases can increase quality
45. Feature Switch and Unveil
Hidden
Programmer sees a change locally.
Change is tested in the main
version but not seen.
Test
Story Owner and testers see the
change on test systems.
Beta
Insiders see it and use it. Story
Owner can show it to selected
users for feedback or A/B testing.
UNVEIL!
The big event. Communicate with
all users. Measure reaction.
Onecodeversion
Nospecialtestbuilds
Nolong-runningbranches
47. Role: Developer
• Developers have more power and responsibility.
• Developers have more responsibility for testing.
• Developers (not QA or PM) decide when to release.
This is a strong finding.
• Incentives are correct. Developer might have to
come back from Friday night beers to fix a problem.
This provides a motivation to make good decisions
and automate testing.
• Features can be released but hidden. Product
Managers and Marketers will unveil when they are
ready. Unblock!
Programmers approve releases, not QA
53. Role: IT Vendor
Old Agreement
• Fixed Price
• Fixed deliverables
• Large probability of late delivery
New Agreement
• Fixed price
• Fixed Time
• Variable deliverables
– Small guaranteed deliverable
– Weekly meetings to agree on
improvements
Advantages
• No late delivery
• Deliverable is better
than original
specification
• Start sooner and finish
sooner
59. Scale it like Google
• 15,000 developers, 5,000 projects, one current version of
the code (2013). They can go from an idea to a release in 48
hours
• Vast Internet system divided into thousands of "services"
• Most programming done by teams of 3-4
• Centralized process with single version of the test system –
run 100 million test cases daily
• Before developers release a change, they test it with the
most recent version of all the other services. If a test script
finds conflicts, it tells developers who to contact to resolve
them
61. Matrix of Services - MAXOS
Prioritized
Backlog
Current
Work
Each team
releases
when ready
Hundreds
of releases
per day
Service team
Production
service
Service team
Production
service
Service team
Production
service
Feedback on speed, errors, usage, and requests
Test as one system
Integration
test env
Integration
test env
Integration
test env
63. Coordinate without big meetings
Continuous Integration between
latest dev version of each service
• Continuous integration
helps teams coordinate.
• See dependencies
between “producers”
and “consumers”
• Errors and conflicts show
related team contact info
• Meetings and changes
negotiated between two
teams, not many
Prioritized
Backlog
Current
Work
Service team
Service team
Service team
Integration
test env
Integration
test env
Integration
test env
Machines can replace layers of management
65. Teams are largely self-managing
Prioritized
Backlog
Current
Work
Service team
Integration
test env
Up to 50% of work
from backlog
At least 50% of work is self-planned
Problems get fixed quickly
Production
service
Production
service
Production
service
Feedback: quality, reliability,
speed, user support
Production
service
Production
Server
Sense, respond, self manage
minimize planning
67. Hubspot – Great at Mid-scale
• Transformed a monolithic app to 200 services over one
year
• 3-person programming teams. Each of 20 teams is
responsible for about 10 services
• Dev teams responsible for design, programming,
testing, release, monitoring, and responding to
production problems. No full-time QA. Shared PM
and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy,
manage, and monitor their services
• Feedback from customer support also grouped by team
69. Team Building
Prioritized
Backlog
Current
Work
Each team
releases
when ready
Service team
Production
service
Service team
Production
service
Add capacity fast by building around
Single-function programmer/tech leads
Integration
test env
Integration
test env
Teams are not permanently multifunctional
71. Brooks and Mythical
Man Month
– Hypothesis: Scaling
problems comes from
n^2 communication
explosion
– Solution: Cells and
hierarchy to contain
communication
Internet reality
– Scaling problems come
from dependencies
– Solution: more
sharing, more
communication
Mythical Man Month is wrong40 years of awesome insights, but
75. Ways to Scale
Scrum + SAFe
• Add more hierarchy
• Complex
multifunction teams
• Hold big meetings and
teleconferences
• Block everyone into
one cadence
• Coordinate big
releases
Top Tech Companies
Automate management,
as well as testing and
deployment.
Dev-lead teams
Communicate peer to
peer
Unblock! teams to move
as fast a possible
Release more frequently
77. Competing with MAXOS
The secret weapon that Silicon Valley is using to
disrupt and destroy competitors
• Retailer X deploys changes to their monolithic online
ordering app once every six weeks. Ops holds for three
weeks to make sure the complete system is stable.
• Amazon has thousands of services and more than 1000
service teams. They release something about once every
11.6 seconds. In the time that Retailer X takes to try
one new release, Amazon has made 100,000 changes.
• Amazon hosting competitor: “It’s an emergency”.
97. Automated Programming
Each team
releases
when ready
Hundreds
of releases
per day
Automated Programming or
Machine Learning
Service Team
Service Team
Integration
test env
Integration
test env
Integration
test env
Production
Service
Production
Service
Production
Service
99. Monitor your released software: Errors, Usage
volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to
ask for test scripts.
Continuous integration: Run automated tests
before using human review time
Unit tests in the development environment
Switch new features and architecture
Automated Code Generation and Machine Learning
106. 1) Practice
We can get more efficient at anything with practice and
optimization
This effect explains almost all of the productivity
difference between Waterfall and Agile. In the agile
process, we do more cycles, so we get better at each
step.
Any periodic release cycle causes a lot of stress around
releases. Continuous delivery removes this stress by
doing releases more frequently and optimizing down to a
button push.
108. 2) Do the right thing
Users ignore at least 50% of the new stuff you
try. If you can measure usage or value and
figure out what to ignore in development, you
can increase development productivity by 100%
for zero extra cost.
Measurement is very important.
More frequent releases = more measurements