A presentation I gave at Agile Australia 2013 on Continuous Delivery

1. Continuous
Delivery
The Path to Rapid Experimentation
Paul Coia

2. Deploy
Experiment
Example
Lessons

4. Global platform for
independent artists
and designers to sell
their work more
easily, more oen.

8. 10M designs on 40M products
500,000 artists all over the world.
40M pageviews from 7M Users/month

9. Deploy
Experiment
Example
Lessons

10. 2000 releases 4 years
700 in 2012
Average 3 per day

11. cap production deploy

13. 1. Continuous Integration success
2. Deploy code via Capistrano
3. Monitor vital signs

14. Jenkins Continuous
Integration Server

15. New Relic
Application Monitoring

16. New Relic
Key Transaction Monitoring

17. Airbrake
Error Reporting

18. Circonus and Statsd
Business Metrics Monitoring

19. StatsdClient.increment("Payment OK")

20. if enrolled_in?(“new”).variant?
// show variant
else
// show normal
end

22. Minimal cost of deploying
Rapid pace of change possible
Confidence make lots of changes
Try ideas quicker
Why we like it

23. Deploy
Experiment
Example
Lessons

24. 1.Observe
2.Form a hypothesis
3.Devise an experiment to test it
4.Draw conclusions
Scientific Method

25. Quantitive Research - What?
Qualitative Research - Why?

26. “If the SEARCH box was bigger, then
users will be more likely to search”

31. Split users at point of variation
Isolate experiments
Persist the decision across visits
Exclude crawlers and ‘old’ users
Enrollment

32. // ON LANDING PAGE
if new_visitor()
@guinea_pig = true
end
// ON EXPERIMENT PAGE
if @guinea_pig && !retrieve_enrolment()
experiment = enrol_in_experiment(“new”)
persist_enrolment(experiment)
end
if enrolled_in?(“new”).variant?
// experiment logic
else
// normal logic
end

33. RUNNING_EXPERIMENTS =
{
...
new: {description: 'New feature',
enrolment_percent: 20},
...
}

34. Google Analytics
Data Collection

35. ‘R’ Statistical Language
Google Analytics API

37. Respect Statistics
Establish your test duration up front
Wait for the test to complete
Results

38. Model your Product funnel
Greater volume earlier in the funnel
Higher conversion rate between steps
Add lots of instrumentation
Useful metrics

39. Homepage
Sign up page
Signed up

40. 1%
Homepage
Sign up page
Signed up

41. 1%
Homepage
Sign up page
Signed up
20%
5%

42. ‘Landing Page’ Optimizers
Client-side changes only
Simple experiments

43. Trivial deployment
Feature toggle
Experimentation platform
= Trivial feature experimentation

44. Deploy
Experiment
Example
Lessons

45. 22% less likely to Add to Cart
50% less likely to Checkout
Mobile Visitors

47. No change to Checkout Completion

49. Improvement in Checkout Completion

51. Increase in Add to Cart rate

53. Further Increase in
Add to Cart rate

54. Multiple lightweight experiments
A handful of weeks
Verified 30% increase in Purchasing

55. Deploy
Experiment
Example
Lessons

56. Invest in your deployment tools

57. Experiment on the small scale

58. Expect to be wrong

59. Build on the successes

60. Design for incremental change

61. Thank You
Paul Coia
paul@redbubble.com
@pjcoia
h p://www.redbubble.com/jobs