1. Strategies In Continuous Delivery
Learn the “magic” behind the scenes
Aviran Mordo
Server Group Manager @ Wix
@aviranm
http://www.linkedin.com/in/aviran
21:34

2. About Wix
21:34

3. Wix in Numbers
• Over 35,000,000 users
– Adding over 1,000,000 new users each month
• Static storage is over 150TB of data
– Adding over 1TB of files every day
• 3 Data centers + 2 Clouds (Google AE, Amazon)
– Around 300 servers
• Over 40,000,000 Server API calls per day
• ~400 people work at wix
– Over 100 people in R&D
21:34

4. Is Doing Multiple
Deployments A Day Takes
Guts ?
21:34

5. 21:34
From Jan’ 2013 – Jun’ 2013
• 1500
• 470
• 200

6. 21:34
From Jan’ 2013 – Jun’2013
• 1500 Deployments
• 470 A/B Tests
• 200 Feature Toggles

7. Continuous Delivery at Wix
• Abandon “VERSION” paradigm – move feature
centric life
• Make small and frequent release as soon as
possible
• Automate everything – TDD/CI/CD
• Measure Everything
–A/B test every new feature
–Monitor real KPIs (business, not CPU)
21:34

8. Continuous Delivery at Wix
• Empower the developer
–The developer is responsible from product idea to 1M
active users
–Remove every obstacle in the developer’s path
–Big cultural change from waterfall – affects the whole
company
21:34

9. Test Driven Development
• No new code is pushed to Git without being fully tested
• Before fixing a bug first write a test to reproduce the bug
• Cover legacy (untested) systems with Integration tests
21:34

10. Test Driven Development
• What people think is the impact on development
–TDD slows down development
–With TDD we write more code (product + test code).
• Current Test Count (U-Tests + IT-Tests) – over
10,000
21:34

11. Test Driven Development
• Actual impact on development
–We develop products faster
–Removes fear of change
–Easier to enter some one else’s project
–Do we really need QA? (Yes, they code tests)
–Writing a feature is 10-30% slower, 45-90% less bugs
–50% faster to reach production.
–Considerably faster time to fix bugs
21:34

12. Guidelines for successful TDD
• Tests should run on project checkout to a random
computer.
• Tests that cannot be debugged on a developer
machine will never consistently run for any period
of time
• Tests should run fast
• Tests have to be readable – They are the project’s
specs
21:34

13. Feature Toggles
21:34

14. Feature Toggles
• Everyone develops on the Trunk
• Every piece of code can get to production at
anytime
21:34

15. Feature Toggle to the rescue
• Unused new code can go to production – no harm done
• Operational new code goes with a guard – use new or old code by
feature toggle
21:34

16. Usage example
Simple “if” statement in your code
21:34

17. Feature Toggle Strategies (gradual expose users)
• Company employees
• Specific users or group of users
• Percentage of traffic
• By GEO
• By Language
• By user-agent
• User Profile based
• By context (site id or some kind of hash on site id)
21:34

18. Feature Toggle Override
• By specific server
– Used to test system load
– New database flows/migration
– Refactoring that may affect performance and memory usage
• By Url parameter
– Enable internal testing
– Product acceptance
– Faking GEO
• By FT cookie value
– Testing
– When working with API on a single page application
21:34

19. Feature Toggles Management
21:34

20. A/B Tests
21:34

21. A/B Test
• Every new feature is A/B tested
• We open the new feature to a % of users
– Define KPIs to check if the new feature is
better or worse
– If it is better, we keep it
– If worse, we check why and improve
– If we find flaws, the impact is just for % of our
users (kind of Feature Toggle)
21:34

22. An interesting site effect on product
• How many times did you have the conversion
“what is better”?
– Put the menu on top / on the side
• Well, how about building both and A/B Testing?
21:34

23. Marking users with toss value in a cookie
• Anonymous user
– Toss is randomly determined
– Can not guarantee persistent experience if changing browser
• Registered User
– Toss is determined by the user ID
– Guarantee toss persistency across browsers
– Allows setting additional tossing criteria (for example new users only)
– Only use this for sections that a user has to be authenticated
21:34

24. • Do not mix anonymous and registered tests
• AB test parentage of users with optional filters
– New Users Only (Registered users only)
– By language
– By GEO
– By Browser
– user-agent
– OS
– Any other criteria you have on your users
21:34

25. A/B Test Features
• A/B Test Override
– Allows setting a value of a test for validation
– Helps support experience what users experiencing
• Override methods
– Via URL parameter
– Via cookie
• Start/Stop Test
• Pause tests
• Bots always get “A”21:34

26. Manage your tests
21:34

27. Refactoring
21:34

28. Refactoring
• Before refactoring make sure everything is covered with
tests
– Legacy code usually covered by IT tests
• Refactor from inside out
– Small iterations with tests
– Refactor small methods - make sure the tests don’t break
– Deploy often
• Re-write from the outside in
– Write from scratch
– Code duplication sometimes needed (temporary)
– Protected by Feature Toggle21:34

29. Code branch
21:34
New Code Old Code
FT
Opene
d
Yes No

30. DB Schema Changes Without
Downtime
• Adding columns
– Use another table link by primary key
– Use blob field for schema flexibility
• Removing fields
– Stop using. Do not do any DB schema
changes
21:34

31. New DB schema
• Plan a lazy migration path controlled by feature toggle
1. Write to old / Read from old
2. Write to both / Read from old
3. Write to both / Read from new, fallback to old
4. Write to new / Read from new, fallback to old
5. Eagerly migrate data in the background
6. Write to new / Read from new
21:34

32. Gradual Deployment
21:34
• Assume two components
• We shutdown one and install on it the
new version. It is not active yet
• Do self test
• Activate the new server it is passes self test
• Continue deploying the other servers,
a few at a time, checking each one with
self test
A 1.1 B 1.1
A 1.1
B 1.2
A 1.1A 1.1 B 1.1B 1.1
A 1.1A 1.1
B 1.1
B 1.2
A 1.1
B 1.2
A 1.1A 1.1
B 1.1
B 1.2
A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.2

33. Self Test / Post Deployment Test
After each server deployment run a self test before deploying the
next server.
• Checking server configuration and topology
– Make sure database is accessible (DB connection string)
– Is the schema the one I expect
– Access required local resources (data files, other config files, templates,
etc’)
– Access remote resources
– RPC / REST endpoints reachable and operational
• Server will refuse requests unless it passes the self test
• Allow a way to skip self test (and continue deployment)21:34

34. Tools - App-info
21:34

35. Backward and Forward compatible
• Assume two components
• We release a new version of one
• Now Rollback the other…
21:34
A 1.1
B 1.2
A 1.2
B 1.1A 1.1A 1.1
B 1.1
B 1.2
A 1.2A 1.1
B 1.1B 1.1
A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.1
A 1.0
A 1.2A 1.1 B 1.2B 1.1
B 1.2 A 1.2
A 1.2A 1.1 B 1.2B 1.1
B 1.0

36. Production Visibility
21:34
Prepare for a rainy day

37. How does it work – CD Practices
• Test driven development
• Small Development Iterations
• Backwards and Forwards compatible
• Gradual Deployment & Self-Test
• Feature Toggle
• A/B Testing
• Exception Classification
• Production visibility
21:34

38. Where are we today?
• We have re-written our flash editor product as an
HTML 5 editor
– In just 4 months
• Introduced Wix 3rd party applications (developers API)
– In just 6 weeks
• We are easily replacing significant parts of our
infrastructure
• And we are doing ~10 releases a day!
21:34
0
5
10
15
20
25
30
35
1/1/2013
1/4/2013
1/8/2013
1/11/2013
1/15/2013
1/18/2013
1/22/2013
1/27/2013
1/30/2013
2/3/2013
2/6/2013
2/10/2013
2/13/2013
2/18/2013
2/21/2013
2/25/2013
2/28/2013
3/4/2013
3/7/2013
3/12/2013
3/15/2013
3/19/2013
3/24/2013
4/3/2013
4/7/2013
4/10/2013
4/15/2013
4/19/2013
4/23/2013
4/28/2013
5/1/2013
5/6/2013
5/9/2013
5/13/2013
5/19/2013
5/22/2013
5/26/2013
5/29/2013
6/1/2013
6/4/2013
6/7/2013
6/11/2013
6/16/2013
(blank)
Total
Total

39. 21:34

40. 21:34
Aviran Mordo
@aviranm
http://www.linkedin.com/in/aviran
Read more: The Road To Continuous Delivery:
http://goo.gl/K6zEK
http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery