Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2f6rtOx. Adrian Trenaman discusses how HBC Digital has applied a blend of microservice / serverless architectures, continuous deployment, and cloud technology to make it easy to push code swiftly and safely and operate it reliably in production. He also talks about the organisational tools like team self-selection, internal startups, etc, that allow them to decentralise and decouple high-performing teams. Filmed at qconnewyork.com. Adrian Trenaman is SVP Engineering at HBC Digital, where he leads the engineering and infrastructure teams for Gilt in New York and Dublin. He is an experienced, outspoken software engineer, communicator and leader with over 20 years of experience working with teams throughout Europe, US and Asia in diverse industries such as financial services, telecoms, retail, and manufacturing.

1. Fitter, Happier,
More Productive.
Removing Friction in the Developer
Experience
Q-Con New York, June 27th 2017
Ade Trenaman, SVP Engineering, HBC Digital
t: @adrian_trenaman
http://tech.gilt.com
t: @hbcdigital fa: @hbcdigital in: hbc_digital

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
hbc-digital-deployment

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. What are you doing
while in the US?
I’m speaking a
software conference.
Sir.

5. What’s your official
title?
I’m SVP Engineering
at Hudson’s Bay Co.

6. What’s your talk
about?
Ummm.
Improving the way we
do software
engineering.

7. So, how do you do that?
<fear induced pause>
Provide unfettered access to cloud
computing resources and remove all
things that block engineers from getting
software to production.

8. It’s hard, huh, all that red tape and
bureaucracy?
Yup.
Well, hope you solve it! :)
Welcome to America.

9. minimise the distance between
“hello, world” and production.
scala> println ("hello, world.")
hello, world.
production

10. minimise the distance between
a good idea and production.
productiona good idea

11. For great dev-ex:
“... build an organisation and
architecture that allows you
to deploy change frequently,
swiftly and safely to
production, and own the
impact of that change”

12. Laptop. Wifi. VPN. Seat.
Standing Desk. Screen.
Warmth. Light.
Fuzbol. Beanbags.
Free-food.
Pet-creche.
Self
Actualise:
Get stuff
done and
have cool
stories that
impress your
friends.
Perks.
Basics. You must have
these.
This is the most important thing.
DEVELOPER HIERARCHY OF NEEDS

13. code first
Teams: 5±2 in size
Departments: 20±4
#leadersnotmanagers
#leaderswhocode: 85%, 60%, 15%
IC & Lead tracks
#devops
#ownership
#opensource

14. SAY “AUTONOMY, MASTERY, PURPOSE”
ONE MORE TIME

15. M
G
FApplied
Friction*
* reactive force resisting motion
… motivation: autonomy, mastery,
purpose
The change we want to make...
… work is hard.
… all the things that slow
us down or block us.

16. f : Staging/Testing Environments
Prefer to test in production. #srsly

17. Dev, QA & Test environments are high-friction places to
write code.
PROD
DEVDEVDEV
DEVDEVDEV
DEVDEVDEV
QA
QA
QA
PRE
PROD

18. Lack of Flow, Excessive Bending, Kneeling,
Reaching
Increased Waste = Lower Productivity, Safety
Opportunities
Borrowing from
lean / six-sigma

19. MOTION STUDY – “SPAGHETTI DIAGRAMS”
Spaghetti Diagrams make poor layouts
and wasted motion obvious

20. Spaghetti diagram of movement and handover within the software
delivery process.
PROD
DEVDEVDEV
DEVDEVDEV
DEVDEVDEV
QA
QA
QA
PRE
PROD

21. Overproduction
▪ Encourages fewer ‘big
bang’ releases
Waiting
▪ Can’t get my stuff
deployed
Motion
▪ Commit deploy test
commit deploy test
commit deploy test...
Transportation
▪ Multiple handoffs
between Engineers,
QA & Ops
Inventory
▪ Lots of commits held up
in the pipeline.
Rework
▪ Works in one
environment, not in
another
Overprocessing
▪ Tickets tested and
rested in different
environments.
Intellect
▪ Spending time building
and debugging
environments instead
of adding value
Muda - “Waste” in
the software
software delivery
process

22. Prod
Core idea #1: test in prod with dark canaries, canaries, release, roll-back.
Dark Canary
1.0.0
Instance_0
1.0.0
Instance_1
1.0.0
Instance_n
1.0.0
Dark Canary
1.0.1
Canary
1.0.1
Instance_0
1.0.1
Instance_1
1.0.1
Instance_2
1.0.1

23. github.com/gilt/nova- deployment patterns
Instance_0 - v1.0.0
Instance_1 - v1.0.0
Instance_2 - v1.0.0
Live Traffic
Instance_3 - v1.0.0 Canary
Instance_4 - v1.0.0
Dark
Canary
Elastic Load
Balancer (ELB)
http://hello-world-nova.common.giltaws.com
Elastic Load
Balancer (ELB)
http://hello-world-nova-dark.common.giltaws.com

24. github.com/gilt/nova - creating environments
nova.yml
templates
$> nova stack create production
CloudFormation
CodeDeploy

25. github.com/gilt/nova- deployment
Instance_0 - v1.0.0
Instance_1 - v1.0.0
Instance_2 - v1.0.0
Live Traffic
Instance_3 - v1.0.0 Canary
Instance_4 - v1.0.0
Dark
Canary
Elastic Load
Balancer (ELB)
live
Elastic Load
Balancer (ELB)
dark
$> nova deploy common DarkCanary
1.0.1
Instance_4 - v1.0.1
$> nova deploy common Canary 1.0.1
Instance_3 - v1.0.1
$> nova deploy common Production
1.0.1
Instance_0 - v1.0.1
Instance_1 - v1.0.1
Instance_2 - v1.0.1
CodeDeployS3bundle

26. Core idea #2: your teams are startups providing services
to other development teams
prod
contract
sandbox

27. Core idea #3: exploit multi-tenant design for confident
testing in production
api-brand-favapi-brand-favapi-brand-fav api-brand-fav
dark canary
https://...hbc.com/saks/favourites/...
https://...hbc.com/bay/favourites/...
https://...hbc.com/test/favourites/...

28. Core idea #4: give your teams secure, unfettered control
over their own infrastructure. Segregate and apply
command-and-control where you need it most.
Master AWS Account
Web & Shared Services
ML & Algos Mobile
Services
Data
INFRA

29. f : Forced technology choices.
Prefer voluntary adoption.

30. adopt
hold
trial
assess
std.
https://github.com/gilt/standards
CodeDeploy
Docker
CodePipeline
ION Roller
ECS
sbt-code-deploy
AWS
Lamda
Docker Hub
(Open Source)
CloudFormation
NOVA
ECR

31. Philosophical note: choose your abstractions &
frameworks carefully.
adoption by rule
centralised
uniform
efficient
go
voluntary adoption
decentralised
diverse
effective
Scala, Java,
Ruby, Swift, JS,
Node, ...
Steer towards classroom size
consensus

32. f : Fear of Breaking All The Things
Adopt µ-services. Adopt λ.
Maximize code-to-cruft-ratio.

33. A minimalist abstraction of our architectural evolution
2007
Monolith
2010
Service
Oriented
2012
µ-Services
λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ
λ λ λ λ λ λ λ
2016
Rise of λ

34. Lots of Small Apps (LOSA) - AKA “micro-frontends”.
<<monolith>>
swift:jsp
<<traffic manager>>
:zxtm
~15 more small, simple,
isolated web apps that
share the same look
and feel.

35. I hear ya.

36. We have marketing URLs like:
https://gilt.com/loveaws
We need to look up the slug ‘loveaws’ and
change that to a ‘pkey’ for our login, so we can
redirect with 302 to:
https://gilt.com/login?pkey=loveaws&...
Existing solution routed to legacy Ruby on Rails
app:
● not scalable.
● not ‘symmetric’
A small-but-important problem: marketing redirects
Rails
Postgres
Zeus
:( customer-facing
traffic on Rails :(

37. λ-based solution
Replace with solution using API-Gateway +
Lambda + KMS
● 3/80 LOC (cruft/code) .js
● KMS used for encrypted DB credentials
● Response cached
● No longer hits Rails! λ
Postgres
Zeus
Tiny λ
API
Gateway

38. It’s just code.

39. f : Forced team choices.
Prefer self-selection.

40. Self Selection
Product Mgr, Tech Lead &
Project Mgr ‘pitch’ to engineers.

41. “I love the team
I’m on right
now!”
Imagine the power of a
fully-aligned team who want to
work together.

42. f : Distractions.
Reinforce the notion that coding is the
primary activity.

43. RED HOT ENGINEER

44. Work your meetings
5@4 (~3w, by location)
Tech Huddle (weekly, by location)
All Hands (monthly, global)
Team KPI meetings: 2-4 weeks
Quality Review
Team meetings? Up to them.
Ask: “was this meeting valuable?
should we meet again?”

45. ~ 2.75 - 5 hrs a week

46. Measure It.

47. POps Mission
To build and maintain the best
product development teams in the
world through establishing the
models around how we staff and
organize our teams, how we plan
and execute our work, and how we
develop our people and our
culture.
Reduce the Friction in the
Employee Experience!

48. Team Health Check - Trends
Baseline
9/27/2016
Current
11/23/2016

49. #thanks @adrian_trenaman @gilttech @hbcdigital
Seek out and remove friction in your engineering process.
Give freedom-of-choice & freedom-of-movement to your engineers.
Code is the primary artifact.
Minimize the distance between “hello, world” and prod.

50. Overproduction
▪ Routinely exceed
customer needs
("gold-plating")
▪ Exceeding scope of
SLAs
Waiting
▪ Idle time during
automated program
runs
▪ Waiting between
assignments
Motion
▪ Interruptions leading to
context switching,
mental motion
▪ Lack of or sub-optimal
Standard Operating
Procedures (SOP)
Transportation
▪ Multiple handoffs of
incidents, changes
▪ Sub-optimal dispatch
and routing
▪ Insufficient use of
remote diagnosis
Inventory
▪ Large number of
servers due to a low
server utilization
▪ System-generated
alerts clogging ticket
queues
Rework
▪ Misrouted tickets
▪ Inadequate testing
before
production
▪ Poor
change-window
planning
Overprocessing
▪ Unnecessary system
replacement,
patching
▪ Backup/defrag runs
earlier than needed
▪ Excessive
documentation
Intellect
▪ Mismatched work
functions with skill sets
▪ Lack of best prac-
tice sharing across
groups
Muda - “Waste” in
manufacturing
process

51. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
hbc-digital-deployment