The rise of utility computing platforms such as Amazon EC2 has made it more feasible to build turnkey hosted solutions on top of open source software. Learn how we built PondCMS, a turnkey CMS deployed to Amazon's EC2. This presentation discusses the advantages of hosting Plone sites in the elastic computing cloud and some of the challenges we faced. Watch the video from this talk at http://pycon.blip.tv/file/1949202/

1. Plone in the Cloud
A CMS hosted on Amazon EC2
Nate Aune (Jazkarta.com)
PyCon 2009
Chicago, IL
Tuesday, April 7, 2009 1

2. • President of Jazkarta (based in Boston)
• 3 Full-time and 10 freelance developers
• Python-based open source web applications
• Members of core Plone framework team
• Member of the Plone Foundation Board
Tuesday, April 7, 2009 2

3. In the next 30 min...
• Amazon Web Services (AWS)
• AWS with Python
• Plone with AWS
• Tools for launching & remote deployment
• Challenges of using AWS
• Monitoring and maintaining EC2 instances
Tuesday, April 7, 2009 3

4. What is
Amazon Web Services?
Tuesday, April 7, 2009 4

5. What is Amazon Web Services?
• Suite of elastic IT
infrastructure services
• Requisition computing
power and storage
• Run any application
you want on any OS
Tuesday, April 7, 2009 5

6. Benefits of AWS
• Metered pricing (per hour / per GB)
• no large up-front capital investments
• only pay for what you use
• Scaleable hosting architecture
• Redundancy for high availability
Tuesday, April 7, 2009 6

7. AWS Architecture
• EC2 = Elastic Computing Cloud
• S3 = Simple Storage Solution
• EBS = Elastic Block Storage
• Elastic IP = Static IP addresses
Tuesday, April 7, 2009 7

8. Elastic Computing Cloud (EC2)
amazon.com/ec2
• Use an existing AMI (Amazon Machine Image)
to launch a new instance
• Or create your own AMIs (even paid ones)
• Attach EBS volumes to your EC2 instance
Tuesday, April 7, 2009 8

9. Elastic Block Storage
(EBS)
Tuesday, April 7, 2009 9

10. So how does Python
play into all this?
Tuesday, April 7, 2009 10

11. Enter Boto
Tuesday, April 7, 2009 11

12. What is Boto?
• Python package that provides interfaces to
Amazon Web Services
• Thanks to author Mitch Garnaat!
• Boto supports:
• S3 via the REST API
• EC2 via the Query API
• and more...
Tuesday, April 7, 2009 12

13. Install Boto & Bototools
$ sudo easy_install boto
$ svn co http://bototools.googlecode.com/svn/trunk/ bototools
$ cd bototools
$ sudo python setup.py install
$ sudo python ./src/bototools/postinstall.py -p /usr/local/bin
• This will put these commands in /usr/local/bin:
• ec2start, ec2stop, ec2ip, ec2ls, ec2img
• s3ls, s3rm, s3cp
Tuesday, April 7, 2009 13

14. Using Bototools
• Set these environment variables:
• AWS_ACCESS_KEY
• AWS_SECRET_ACCESS_KEY
• EC2_SSH_KEY
• Can also put in ~/.boto or /etc/boto.cfg
Tuesday, April 7, 2009 14

15. Find an AMI
• We will use the Ubuntu 8.10 Intrepid (32-bit)
from Alestic.com - ami-7cfd1a15
• More can be found at:
http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=171
Tuesday, April 7, 2009 15

16. Launch an instance
$ ec2start -a ami-7cfd1a15 -k gsg-keypair
Reservation:r-0f26a666
[Instance:i-8e3154e7]
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 pending None 2009-03-25T01:49:24.000Z
...wait...
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 running None 2009-03-25T01:49:24.000Z
ec2-174-129-238-8.compute-1.amazonaws.com
Tuesday, April 7, 2009 16

17. Associate a static IP
$ ec2ip -g
Address:174.129.238.8
$ ec2ip -l
174.129.238.8 ->
$ ec2ip -a 174.129.238.8 i-8e3154e7
True
$ ec2ip -l
174.129.238.8 -> i-8e3154e7
• Note: once you assign a static IP, you won't be
able to access the machine using the original IP.
Tuesday, April 7, 2009 17

18. Log in
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 running None 2009-03-25T01:49:24.000Z
ec2-174-129-238-8.compute-1.amazonaws.com
$ ssh -i ~/id_rsa-gsg-keypair root@ec2-174-129-238-8.compute-1.amazonaws.com
Linux domU-12-31-39-00-E5-14 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686
root@domU-12-31-39-00-E5-14:~#
• Time to play!
Tuesday, April 7, 2009 18

19. Stop the instance
$ ec2stop i-8e3154e7
Stopping ['i-8e3154e7']
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 terminated None 2009-03-25T01:49:24.000Z
• Be sure to stop the instance or
you will be charged!
Tuesday, April 7, 2009 19

20. Use the AWS Console
Tuesday, April 7, 2009 20

21. Or use ElasticFox
• Firefox extension to start/stop EC2 instances, assign
IPs, associate EBS volumes and create keypairs.
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609&categoryID=88
Tuesday, April 7, 2009 21

22. Case study
Plone on EC2
Tuesday, April 7, 2009 22

23. What is Plone?
• Open source content management system (CMS)
• Built on top of Zope
• Written in Python
• Large international community
• Used by Oxfam, NASA, Novell, Akamai, FSF, etc.
• more listed at http://plone.net/sites
Tuesday, April 7, 2009 23

24. Why Plone on EC2?
+
• Too few Zope hosting providers
• Avoid renting rackspace in co-lo
• Flexible and scalable architecture
• Complete control of servers
Tuesday, April 7, 2009 24

25. What is PondCMS?
PondCMS = Plone On Demand
• Hosted and supported version of Plone
• Eliminate the complexities of hosting
• Turnkey - automated and repeatable
• Bundled with certified add-ons
www.jazkarta.com/pondcms
Tuesday, April 7, 2009 25

26. PondCMS Technology stack
HTTP proxy
Caching proxy
0%1&2
Load Balancer
+
*#quot;+
PondCMS ,-./+ add-ons
*#quot;+
Web App Server ,-./+
Operating System *#quot;+
,-./+
Virtual Servers !quot;#$%&!'()
Tuesday, April 7, 2009 26

27. One server deployment
Apache
HTTP
(HTTP)
Varnish
(Caching)
Pound
(Load balancing)
1 2 n
Zope Client #1 Zope Client #1 Zope Client #n
Zeo Server
Zeo Backup
Tuesday, April 7, 2009 27

28. Add Zeo clients as
demand increases
Tuesday, April 7, 2009 28

29. Multi server deployment
Website visitor
HTTP
Elastic Block Storage
public IP: 72.44.50.237
Apache
Frontend server Apache logs
Varnish
Pound
Application Servers
1 2 n
zope-1.web.private:8222 zope-2.web.private:8222 zope-n.web.private:8222
Zope client #1 Zope client #2 Zope client #n
Elastic Block Storage
Database server
zeo-1.web.private:8221
Data.fs
Zeo server #1
Data
Data.fs Backup files
Simple Storage Solution (S3)
Tuesday, April 7, 2009 29

30. Goal: automate everything
Scripts for repeatable deployment
1. Bootstrap the server
2. Install dependencies
3. Install PondCMS (includes Plone + add-ons)
4. Configure backups
5. Create the Plone site and Apache vhosts
Tuesday, April 7, 2009 30

31. 1. Bootstrap server
1. Create the EC2 instance
2. Create an EBS volume
3. Associate volume with EC2 instance
4. Acquire IP address
5. Associate IP with EC2 instance
Tuesday, April 7, 2009 31

32. 2. Install dependencies
1. Install packages with apt-get
• zope, python, varnish, apache2, pound, postfix
2. Initialize EBS volume and mount as ext3 fs
3. Store Apache log files on EBS volume
4. Register server with Landscape
5. Set up Pound to start automatically
6. Set up Pound to point to Zeo clients
Tuesday, April 7, 2009 32

33. 3. Install PondCMS
1. Make a 'zope' user
2. Create a virtualenv
3. Make Zope data dirs on EBS volume
4. Download PondCMS tarball release (incl. Plone)
5. Install PondCMS into virtualenv
Tuesday, April 7, 2009 33

34. 4. Configure backups
• Configure logfile rotation
• Setup backup scripts (using repozo)
• Setup sync of backups to S3 (using Duplicity)
• Create cron jobs for backups (daily, weekly)
Tuesday, April 7, 2009 34

35. 5. Create Plone site & vhosts
1. Use Fabric (remote deployment tool) to:
• Make Plone site incl. CacheFu setup
• Make vhost for accessing site on port 80
2. Create Apache vhosts for:
• supervisor.domainname.com (start/stop Zope)
• admin.domainname.com (ZMI access)
Tuesday, April 7, 2009 35

36. What is Fabric?
Tuesday, April 7, 2009 36

37. What is Fabric?
http://www.nongnu.org/fab/
• Simple remote deployment tool
• Similar to Capistrano (Ruby on Rails)
• Deploy applications to multiple similar hosts
Tuesday, April 7, 2009 37

38. Setup site Fabric script
@require('host', 'site_id')
@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename'
def createplonesite(host=None, user=None, key=None, site_id=None,
public_host=None):
quot;quot;quot;Create a new plone site quot;quot;quot;
if not site_id:
site_id = host
installzcml(host=host, user=user, key=key, package='pondcms.basi
restart(host=host, user=user, key=key)
put('createplonesite.py', '/tmp/createplonesite.py')
s = SERVICE+'/bin/utility run /tmp/createplonesite.py -a
pondcms.basicapp -s '+site_id
if public_host:
s += ' --site-url='+public_host
run(s)
Tuesday, April 7, 2009 38

39. Setup vhost Fabric script
@require('host', 'vhost_name')
@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename'
def setupvhost(host=None, user=None, key=None, vhost_name=None):
quot;quot;quot;Setup an apache2 vhostquot;quot;quot;
site = VHOST_TEMPL % {'vhost_name': vhost_name}
tmpfile = '/tmp/'+get('fab_timestamp')+'_vhost'
f = open(tmpfile, 'w')
f.write(site)
f.close()
targetf = vhost_name+'.conf'
target = '/etc/apache2/sites-available/' + targetf
put(tmpfile, target)
run('a2ensite vhost_name')
run('/etc/init.d/apache2 reload')
Tuesday, April 7, 2009 39

40. Running Fabric scripts
$ cd fabric
$ source bin/active
(fabric)$ cd fabs/pondcms
(fabric)$ fab
createplonesite:host=test.pondcms.net,user=root,key=~/id_rsa-gsg-
keypair,site_id=test.pondcms.net,public_host=http://
test.pondcms.net
...
(fabric)$ fab setupvhost:host=test.pondcms.net,user=root,key=~/
id_rsa-gsg-keypair,vhost_name=test.pondcms.net
Tuesday, April 7, 2009 40

41. Challenges
• Persistent data must be stored on EBS
• Static IPs
• Mail sent from EC2 instances flagged as spam
• Monitoring and maintenance of VMs
• Landscape
• ZenOss
• Nagios
• Hyperic
Tuesday, April 7, 2009 41

42. Tuesday, April 7, 2009 42

43. AWS Status history
Tuesday, April 7, 2009 43

44. Landscape
monitoring / updates
• http://landscape.canonical.com
Tuesday, April 7, 2009 44

45. Landscape update pkgs
Tuesday, April 7, 2009 45

46. Thank you!
Questions?
Follow me on Twitter:
natea
Or email me at:
natea (at) jazkarta (dot) com
Tuesday, April 7, 2009 46

47. More info
• Amazon EC2: http://amazon.com/ec2
• Boto: http://code.google.com/p/boto
• Bototools: http://code.google.com/p/bototools/
• Fabric: http://www.nongnu.org/fabric
• Fabulatr: http://code.google.com/p/fabulatr/
• Plone: http://plone.org
• PondCMS: http://jazkarta.com/pondcms
Tuesday, April 7, 2009 47

48. Other resources
• Ubuntu on EC2:
http://groups.google.com/group/ec2ubuntu
• Ubuntu AMIs: http://www.alestic.com
• Beta program: http://www.ubuntu.com/ec2
• Landscape: http://landscape.canonical.com
• vm-builder-ec2:
http://packages.ubuntu.com/intrepid-updates/utils/python-vm-builder-ec2
• Enomaly's Elastic Computing Platform:
http://src.enomaly.com/
Tuesday, April 7, 2009 48

49. Photo credits
• http://www.flickr.com/photos/docman/66137352/
• http://www.flickr.com/photos/thomashawk/126534906/
• http://www.flickr.com/photos/moeview/498544386/
• http://www.flickr.com/photos/shellysblogger/
2464969383/
• http://www.flickr.com/photos/mhw/56386662/
• http://www.flickr.com/photos/mrtopf/1541527579/
Tuesday, April 7, 2009 49