PaaS Comparison 2014
comparing and contrasting leading PaaSes on multiple dimensions
and feature sets to see where CF leads and lags
dr.max @maximilien
maxim@us.ibm.com
IBM Cloud Labs
!
and
!
James Bayer
jbayer@gopivotal.com
Pivotal
v0.8.0
10 June 2014
credit: http://www.teacherspayteachers.com/

Agenda
• Background
• Limitations and threat to validity
• Methodology - briefly
• Results - along comparison main axes
‣ Workloads, Tooling, Integration services
‣ SLAs, Datastores, Programming Models,
Management, and Misc.
• Insights
• QoE - quality of experience test
• Potential next steps

Background
• Based IBM Research study (circa 2010)
• Comparing & contrasting 6 leading PaaSes
‣ Microsoft’s Azure
‣ Google App Engine (GAE), GAE4Business
‣ Amazon AWS+, i.e, EC2, Beanstalk, + services
‣ Heroku from SalesForce.com
‣ OpenShift OSS PaaS from RedHat
‣ CloudFoundry OSS PaaS
• Started at IBM but revised with Pivotal in ‘14

Limitations and validity threat
• Challenges in comparing PaaSes
‣ what is claimed may not equal day-to-day reality!
‣ no standard measuring stick for most features!
‣ “signal vs. noise” problem measuring some features
• Limitation of approach
‣ non-scientific, i.e., subjective rather objective data!
‣ potentially biased (limited numbers of evaluators)!
• Value is providing a framework for own analysis
• Goal is measure PaaS QoE (quality of experience)

Overview of comparison
• Found 50 features of comparisons
• Eight groups
‣ (1) workloads, (2) tooling!
‣ (3) integration services, (4) SLAs!
‣ (5) datastores, (6) programming models!
‣ (7) management, and (8) miscellaneous
• Question: to what extent does the PaaS support
each of the features in group?
0: none, 1: basic, 2: strong, and 3: leading

workloads & tooling

integration services & SLAs

datastores & programming models

management

misc.

Quality of Experience
Scenario: Run (deploy, scale, update, troubleshoot)
a java web application with a database
1. OpenShift (OpenShift Online)
2. Heroku
3. Amazon Beanstalk
4. Google AppEngine
5. Microsoft Azure
6. Cloud Foundry (Pivotal Web Services)

AWS Beanstalk
Does well:
• Super well integrated into all AWS Services and feels like eases
IaaS friction
• OOTB Monitoring is best-in-class
!
Not so well:
• Responsiveness - VM orchestration takes a really long time
• Treats frameworks differently
• Must understand many AWS concepts to use effectively
• Extensibility – only fixed number of options

Google AppEngine
Does well:
• Super well integrated into many Google Services like Cloud
SQL and other APIs
• Events and runtime info well displayed
!
Not so well:
• Java support limited (limitations on what you can do with
threads, requests, SQL driver, services integration)
• Forces you to learn too much about Google Cloud
• Limited ability to customize container
• No debug/shell access in standard AppEngine

Azure Websites
Does well:
• Many options for frameworks
• Well integrated web UX
• Impressive list of SCM integration options
!
Not so well:
• Windows default option instead of Linux
• Very much an IaaS / VM feel
!
Did not finish yet

OpenShift
Does well:
• Excellent container ssh and port forwarding UX
• SCM and CI directly integrated as an option (did not try
Jenkins cartridge but it is available)
Not so well:
• No auto-detection, must explicitly choose a cartridge
• Out of date dependencies like JDK, Tomcat version, etc
• No Status, Health Management
• Scaling and managing many app instances limited
• Host network connections visible with $netstat -a
• WAR file support requires SCP

Heroku
Does well:
• Best-in-class polish all around on UX
• Run command, pipelines, stats in logs, marketplace
• Holds request at router when updating, unpausing apps
!
Not so well:
• Java seems to be a 2nd or 3rd class citizen
• Must use git, does not support war file directly
• Teams and managing multiple apps

Cloud Foundry
Does well:
• Java build tools best in class with maven, gradle, Eclipse and
many buildpacks to choose from
• Scales and responds quickly
• Logs (both app and audit events)
• Super easy Services integration
Not so well:
• Shell access without workarounds
• SCM / CI integrations without workarounds
• Provide app metrics
• Build in zero downtime updates

Insights and lessons learned
1. Difficult to extract comparison axes across PaaSes
2. Many comparison points are supported in all
3. AWS+ seem to be most mature PaaS (advertised?)
4. Might need to define a benchmark for PaaS
a. should contain common apps
b. should allow PaaS apps to be tested openly
5. Make spreadsheet and full presentation OSS
6. Crowdsource ratings to remove some subjectivity?
7. Involve members of each other PaaS community
8. Iterate, iterate, iterate

Potential next steps
• Socialize, update comparison matrix
• QoE test expanded to thorough scenario
• Could we develop a PaaS benchmark?
‣ Sample apps that can be ported for each PaaS
‣ Develop data collection tools and spreadsheets
‣ Collect data for each PaaS for benchmarking
‣ Making sense of the results data
• Iterate benchmark results for new PaaS releases

Thank you
(Q&As)

workloads & tooling
• Workload classes representing cloud apps
• Five types of workloads
1. Analytics - Hadoop-style workload
2. Transaction - DB-driven (with TX) workload
3. Media - workloads with large static files (read)
4. Web2 - workloads with read/write intensive
characteristics but no transactions and not sensitive to
read-coherency
5. Mobile - workloads for mobile apps and backend
• Tooling support includes IDE, CLI, APIs, git

integration services & SLAs
• Integration services
‣ includes BSS-types of services
‣ also includes other types of cloud services
• Microsoft seems to dominate this space
• Service level agreements
‣ most IaaS and PaaS providers advertise SLAs
‣ most advertised SLAs are coarse grained
‣ no monetary guarantees advertised
‣ most give more time as compensation
‣ no aggregate stats yet

datastores & programming models
• Datastores
‣ relational models support
‣ so-called No-SQL databases, e.g.,
Cassandra
‣ Column-store vs. Row-store
• Programming models
‣ support for various scripting languages
‣ support for different frameworks
‣ customized frameworks for cloud
‣ shared-runtimes?

misc.
• Misc. (other)
‣ pattern (or template) -based deployment
‣ support for old-style (mature enterprise) platforms
‣ support for multiple target IaaS clouds