Apr 24, 2012 Deployment Pipeline for Magento Enterprise in the Cloud. The Talk covers the Amazon Cloud Infrastructure; Scaling and Autoscaling in the Cloud, the Deployment Pipeline used to do continuous deployments...

1. Tuesday, April 24, 2012

2. Tuesday, April 24, 2012

3. Tuesday, April 24, 2012

4. Tuesday, April 24, 2012

5. Angry Birds Shop
Cloud Deployment
Presented by
Kian T. Gould and Daniel Pötzinger
Tuesday, April 24, 2012

6. Tuesday, April 24, 2012

7. Tuesday, April 24, 2012

8. Irregular Traffic Behaviour
Tuesday, April 24, 2012

9. The Angry Birds Magento Universe
DRM Server
Mail ServiceProducts +
Warehouse Inventory
Storefronts Reporting
Ordermgmt
Manage Warehouse Fullfillment
Giftcards
Braintree
Magento Store
Shipwire
Tuesday, April 24, 2012

10. Angrybirds Magento Universe
DRM Server
Mail ServiceProducts +
Warehouse Invetory
Storefronts Reporting
Ordermgmt
Manage Warehouse Fullfillment
Giftcards
Braintree
Magento Store
Shipwire
Shipwire
Find best way for
order Fullfillment
Manages global
warehouses &
inventory
Tuesday, April 24, 2012

11. Why Amazon Cloud ?
Tuesday, April 24, 2012

12. ✓Scales quickly on demand
✓Pay only what you need
Tuesday, April 24, 2012

13. Powerfull AWS Services
Rightscale Services
Amazon Webservices & Amazon Console
Cloud
Front
Elastic
Cache
S3EC2 RDS ELB R53
Tuesday, April 24, 2012

14. Cloud Drawbacks?
Complexity Plan for EC2 Failover
Plan Security
Need to adjust
Application
Lack of Service
Tuesday, April 24, 2012

15. Cloud Drawbacks?
Complexity Plan for EC2 Failover
Plan Security
Need to adjust
Application
Lack of Service
Architecture Requirements
High availability (Multiple Failover
and Loadbalancing)
Magento Admin Backend
independent from Shop-Frontend
Efficient and independent
background task management
Intensive Caching
Asset management in the cloud
Tuesday, April 24, 2012

16. Tuesday, April 24, 2012

17. Our Cloud Architecture
Tuesday, April 24, 2012

18. Tuesday, April 24, 2012

19. AutoscalingVarnish Array
Frontend Array
Autoscaling
Backend Array Worker Array
Architecture
Cloud Front
CDN for Skin
ELB R53
DNS
RDS
mySQL
S3
Deployment
Elastic Cache
Cache Backend
Cloud Front
CDN for Assets
S3
Assets Storage
FE FE FE
Tuesday, April 24, 2012

20. Scaling is about finding
the Bottleneck
Tuesday, April 24, 2012

21. Scaling and Tuning
Frontend Array
Autoscaling
RDS
mySQL
Cloud
Front
Elastic Cache
Cache Backend
Varnish
Array
Bottleneck
CPU
Load
Balancing
FE FE FE FE FE FE
Bottleneck
Database
Varnish
FE
Bottleneck
Bandwidth
Varnish
Bottleneck
CPU
FE FE FE FE FE FE
FE FE FE FE FE FE
Bottleneck
Database
Tuesday, April 24, 2012

22. Cloud Front
CDN for Assets
Frontend
Image Caching
Frontend Backend
S3
Assets Storage
Image
resized?
Read and save
resized images
Upload new original
images to S3
Tuesday, April 24, 2012

23. The 6 secrets of caching
1. achieve a high hit rate
2. don‘t flood your cache
3. choose your cache backends wisely
4. never loose all cache at once
5. clean up old cache entries
6. be able to clean/refresh on demand
Tuesday, April 24, 2012

24. Magento Performance Pointers
1. Asyncronous Cache Clearing Queue (Aoe_AsyncCache)
2. Take care of Cache Cleaning (Aoe_CacheCleaner)
3. Disable Magento Logs
4. Make page cacheable in Varnish (Aoe_Static)
5. Prefix bundeled JS and CSS with timestamp
6. and... Profile, Profile, Profile:
•xdebug + kcachegrind
•Newrelic
•xhprof
Tuesday, April 24, 2012

25. Magento Cloud-Adjustments
1. Compress Javascript + CSS on deploy time
2. Use Deployment Name as Cache prefix
3. Cache Warmup Script as part of Deployment
4. Health Check for Loadbalancer
5. Automatic Varnish Purging
6. Use E-Mail Service
7. Establish Asset handling
Tuesday, April 24, 2012

26. Tuesday, April 24, 2012

27. Lessons learned
1. 404s are too expensive
=> cache or redirect
2. Cloudfront with S3 backend
cannot handle gzip
=> deploy time bundeling/compression
3. Don‘t hit „Reports“ in the backend :-)
4.You cannot forsee everything
=> But you can be prepared to fix things quickly
=> continuous deployment
Tuesday, April 24, 2012

28. Tuesday, April 24, 2012

29. Continuous
Deployment
Tuesday, April 24, 2012

30. Tuesday, April 24, 2012

31. Don‘t be scared of
releasing to
production
Tuesday, April 24, 2012

32. Definition of Continuous Deployment
We want on-demand deployment
of tested features and fixes
in a quick and reliable way
Tuesday, April 24, 2012

33. Automating everything
You can achieve this by
Build
Testing
Deployment
Tuesday, April 24, 2012

34. doing it frequently
..and
Tuesday, April 24, 2012

35. Tuesday, April 24, 2012

36. The Deployment
Pipeline
Tuesday, April 24, 2012

37. Tuesday, April 24, 2012

38. Selenium
Acceptance Tests
Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
„latest“
Performance
Tests
Angry Birds Deployment Pipeline
Feedback to Developers
Tuesday, April 24, 2012

39. ‣ Filesystem
‣ Database
‣ Install Binaries
‣ settings.php
Magento Installation Package
‣ htdocs [Magento Source]
‣ media
‣ .modman [Custom Packages]
‣ changelog.txt
1.set maintenance mode
2.Delta Package prepare
3.apply Database
4.sync files
5.run modman deploy-all
6.adjust Settings
7.Post install:
fix p. / cache clean / ...
Tuesday, April 24, 2012

40. Angry Birds Deployment Pipeline
Install on
Deployment
System
Copy to S3
Cloud
Deployment
Staging
Integration
Tests
AOE + Rovio
Cloud
deployment
production
Tuesday, April 24, 2012

41. Deployment „Version 89“
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
Start new release by
✓ cloning old deployment
✓ adjusting release number
RDS
mySQL
S3
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
ELB
WFE FE FE BE BE BE
V V VV V V
WFE FE FE BE BE BE
R53
DNS
Tuesday, April 24, 2012

42. Tuesday, April 24, 2012

43. The real life
Tuesday, April 24, 2012

44. Tuesday, April 24, 2012

45. Day 1 - Release 1
Tuesday, April 24, 2012

46. Day 2 - Release 2 - „get ready for Space“
Tuesday, April 24, 2012

47. New release
deployed
Space game
launched
Array scales up
USA wakes up
Day 3 - „People want it so badly“
Tuesday, April 24, 2012

48. 40% average
CPU usage.
1-2 FE Servers
are enough
Day 4 - Release 3 „Performance Improvement“
Tuesday, April 24, 2012

49. Tuesday, April 24, 2012

50. Thank you
Follow us on Twitter:
AOE: @aoemedia
Daniel P: @typo3media
Tuesday, April 24, 2012