Managing Amazon AWS Costs

7 195 vues

Publié le

Managing costs on the Amazon AWS cloud

Publié dans : Technologie, Business
2 commentaires
11 j’aime
  • You may want to check out this webinar - Quota and User Management for AWS ( from Convirture. The webinar will show how you can:
    • Centrally manage all your AWS accounts
    • Control user access via role-based access control (RBAC)
    • Define and enforce quota limits to manage costs and usage
    • Give your business customers self-service access to their AWS resources
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • In order to control your AWS costs you will need to manage it manually at least or get some online solution that can provide you this option. i can forward you to this resource - and I think that you will find some great other resources there. You also welcome to test drive your AWS costs using our online small tool at
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
Aucun téléchargement
Nombre de vues
7 195
Sur SlideShare
Issues des intégrations
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

Managing Amazon AWS Costs

  1. 1. Cloudy Mondays Joe Kinsella, VP of Engineering, Sonian Twitter: @joekinsella, Blog: November 2011 Managing AWS Costs
  2. 2. Agenda <ul><ul><li>Overview of AWS Bill </li></ul></ul><ul><ul><li>Available Tools </li></ul></ul><ul><ul><li>Top Reasons For Excessive Bill </li></ul></ul><ul><ul><li>5 Steps To Reducing Bill </li></ul></ul><ul><ul><li>Cloud vs. Physical </li></ul></ul><ul><ul><li>Metrics To Manage </li></ul></ul><ul><ul><li>Cloud Control Demo </li></ul></ul>
  3. 3. Early Warning Sign: Cloud Dropouts <ul><ul><li>Each month we see more cloud dropouts </li></ul></ul><ul><ul><li>E.g. </li></ul></ul><ul><ul><ul><li>Company: Mixpanel </li></ul></ul></ul><ul><ul><ul><ul><li>Real-time analytics platform launched in 2009 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Funded by Sequoia & Y Combinator </li></ul></ul></ul></ul><ul><ul><ul><li>Started using Rackspace cloud </li></ul></ul></ul><ul><ul><ul><ul><li>Low initial cost </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Fast deployment times </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Consumption-based pricing </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Cheap CPU performance </li></ul></ul></ul></ul><ul><ul><ul><li>Issues as cloud usage scaled </li></ul></ul></ul><ul><ul><ul><ul><li>One size fits all </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Lack of access to high-end hardware </li></ul></ul></ul></ul><ul><ul><ul><ul><li>At mercy of neighbors </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Costs </li></ul></ul></ul></ul><ul><ul><ul><li>Solution </li></ul></ul></ul><ul><ul><ul><ul><li>Rebuilt on dedicated infrastructure hosted at SoftLayer </li></ul></ul></ul></ul><ul><ul><li>Dropouts are early warning sign not everyone successful in the cloud </li></ul></ul>
  4. 4. Amazon AWS Bill Is Complex <ul><ul><li>AWS bill details costs by </li></ul></ul><ul><ul><ul><li>Service – e.g. EC2, S3, RDS </li></ul></ul></ul><ul><ul><ul><li>Region – e.g. US East, US West </li></ul></ul></ul><ul><ul><ul><li>Service item – Linux m1.large compute hours </li></ul></ul></ul><ul><ul><li>Each service has multiple components to pricing </li></ul></ul><ul><ul><li>All pricing consumption-based </li></ul></ul><ul><ul><li>Pricing model very complex </li></ul></ul><ul><ul><li>Pricing changes periodically </li></ul></ul>
  5. 5. Example: EC2 Pricing <ul><ul><li>EC2 Compute </li></ul></ul><ul><ul><ul><li>Pay per compute hour by instance type and OS </li></ul></ul></ul><ul><ul><ul><li>List and reserved pricing </li></ul></ul></ul><ul><ul><ul><li>Spot pricing </li></ul></ul></ul><ul><ul><ul><li>Data transfer out or within regions </li></ul></ul></ul><ul><ul><li>EC2 Block Store </li></ul></ul><ul><ul><ul><li>Per GB of provisioned storage </li></ul></ul></ul><ul><ul><ul><li>I/O requests </li></ul></ul></ul><ul><ul><li>EC2 Other </li></ul></ul><ul><ul><ul><li>Elastic IP addresses by hour/remap </li></ul></ul></ul><ul><ul><ul><li>CloudWatch by metric per month </li></ul></ul></ul><ul><ul><li> </li></ul></ul>
  6. 6. Limited AWS Tools For Managing Costs <ul><ul><li>Four primary tools available from AWS </li></ul></ul><ul><ul><ul><li>Activity Report </li></ul></ul></ul><ul><ul><ul><li>Usage Report </li></ul></ul></ul><ul><ul><ul><li>Cost Calculator </li></ul></ul></ul><ul><ul><ul><li>Cloud Watch </li></ul></ul></ul><ul><ul><li>Tools provide useful raw data that can be used in managing costs – but not complete solution </li></ul></ul><ul><ul><li>Tools sufficient for managing small deployments </li></ul></ul><ul><ul><li>Medium to large deployments require more than AWS currently provides </li></ul></ul>
  7. 7. AWS Activity & Usage Reports <ul><ul><li>Activity report </li></ul></ul><ul><ul><ul><li>Details costs by service </li></ul></ul></ul><ul><ul><ul><li>By service and account </li></ul></ul></ul><ul><ul><ul><li>Current and historical </li></ul></ul></ul><ul><ul><li>Usage report </li></ul></ul><ul><ul><ul><li>Export of usage by hour or day </li></ul></ul></ul><ul><ul><ul><li>Allows filtering on usage type </li></ul></ul></ul><ul><ul><ul><li>Minimal documentation on how to decipher usage information </li></ul></ul></ul><ul><ul><li>Using reports with accounts </li></ul></ul><ul><ul><ul><li>AWS supports designating single account for consolidated bill </li></ul></ul></ul><ul><ul><ul><li>Common practice to designate empty account for consolidated bill </li></ul></ul></ul><ul><ul><ul><li>Multiple accounts often used to work around lack of sub-accounts </li></ul></ul></ul>Activity Report Usage Report
  8. 8. Other AWS Tools <ul><ul><li>Monthly calculator </li></ul></ul><ul><ul><ul><li>Simple calculator to project costs based on usage </li></ul></ul></ul><ul><ul><ul><li>Good for getting back of napkin estimates on costs </li></ul></ul></ul><ul><ul><li>CloudWatch </li></ul></ul><ul><ul><ul><li>Real-time and trended metrics on usage of provisioned infrastructure </li></ul></ul></ul><ul><ul><ul><li>Basic metrics provided for free </li></ul></ul></ul><ul><ul><ul><li>Allows understanding of actual consumption of provisioned infrastructure </li></ul></ul></ul>
  9. 9. 3 rd Party Tools <ul><ul><li>New and emerging market for cloud asset & cost management </li></ul></ul><ul><ul><li>Dominated by early stage startups </li></ul></ul><ul><ul><li>Products immature and take very different approach to managing costs </li></ul></ul><ul><ul><li>E.g. </li></ul></ul><ul><ul><ul><li>Ylastic, Cloudability, CloudVertical, Sensible Cloud, RaveId, Cloud Cruiser </li></ul></ul></ul>
  10. 10. Top Reasons For High AWS Bills <ul><ul><li>Lack of understanding of what software actually costs to operate </li></ul></ul><ul><ul><li>Lack of understanding of infrastructure & its usage </li></ul></ul><ul><ul><li>Over optimism on infrastructure lifespan </li></ul></ul><ul><ul><li>Not taking advantage of all available AWS options to manage costs </li></ul></ul><ul><ul><li>Lack of infrastructure standardization </li></ul></ul><ul><ul><li>Inattention to the pricing outliers </li></ul></ul><ul><ul><li>Human error </li></ul></ul>
  11. 11. 5 Step Process To Managing Your Bill <ul><ul><li>Gain visibility </li></ul></ul><ul><ul><li>Define blueprint </li></ul></ul><ul><ul><li>Manage capacity </li></ul></ul><ul><ul><li>Rightsize </li></ul></ul><ul><ul><li>Optimize </li></ul></ul><ul><ul><li> </li></ul></ul>
  12. 12. Step 1: Gain Visibility <ul><ul><li>Decide on tool to provide near real-time visibility into your infrastructure, e.g. </li></ul></ul><ul><ul><li>E.g. AWS console, custom application, Ylastic, Excel spreadsheet </li></ul></ul><ul><ul><li>Ensure tool supports correlating infrastructure and application </li></ul></ul><ul><ul><li>Cannot understand costs without knowing how application uses infrastructure </li></ul></ul><ul><ul><li>Adopt tool that allows customization </li></ul></ul>
  13. 13. Step 2: Define Blueprint <ul><ul><li>Define reference architecture for application(s) </li></ul></ul><ul><ul><li>Goal: use as close to 100% of provisioned infrastructure without impacting availability & performance </li></ul></ul><ul><ul><li>Reference architecture should capture all required infrastructure by functional cluster </li></ul></ul><ul><ul><ul><li>E.g. node sizes, attached storage, production vs. non-production </li></ul></ul></ul><ul><ul><li>Create cost model for reference architecture </li></ul></ul><ul><ul><li>Perform R&D to optimize cost model </li></ul></ul>Sample Reference Architecture
  14. 14. Step 3: Manage Capacity <ul><ul><li>Define capacity management policy for each functional clusters </li></ul></ul><ul><ul><ul><li>Identify all metrics for scale (e.g. # concurrent users, # transactions per second) </li></ul></ul></ul><ul><ul><ul><li>Identify thresholds for capacity alarms (warning, critical) for both under and over-capacity </li></ul></ul></ul><ul><ul><ul><li>Identify “run book” for handling capacity constraints </li></ul></ul></ul><ul><ul><li>Automate alarms through proactive monitoring </li></ul></ul><ul><ul><li>Iteratively tune policy to use as close to 100% of provisioned infrastructure as possible </li></ul></ul>
  15. 15. Step 4: Rightsize <ul><ul><li>Start to standardize all infrastructure to reference architecture </li></ul></ul><ul><ul><li>Target infrastructure for rightsizing </li></ul></ul><ul><ul><ul><li>Non-standard – infrastructure that deviates from reference architecture </li></ul></ul></ul><ul><ul><ul><li>Underutilized – infrastructure that can be consolidated based on capacity management policy </li></ul></ul></ul><ul><ul><ul><li>Unused infrastructure – infrastructure that is used infrequently or not at all </li></ul></ul></ul><ul><ul><li>Implement iteratively over time to minimize disruption </li></ul></ul>
  16. 16. Step 5: Optimize <ul><ul><li>Optimize costs through use of reserved & spot instances </li></ul></ul><ul><ul><li>Reserved instances </li></ul></ul><ul><ul><ul><li>Purchase reserved instances based on reference architecture & growth projections </li></ul></ul></ul><ul><ul><ul><li>Reserved instances are limited by region - don’t paint yourself in corner </li></ul></ul></ul><ul><ul><ul><li>Can purchase for 1 or 3 years </li></ul></ul></ul><ul><ul><ul><li>Cost savings: 40% cost reduction on compute </li></ul></ul></ul><ul><ul><li>Spots </li></ul></ul><ul><ul><ul><li>Great for managing capacity bursts </li></ul></ul></ul><ul><ul><ul><li>Requires architecture that support idempotence </li></ul></ul></ul><ul><ul><ul><li>Cost savings: 20% reduction on compute </li></ul></ul></ul><ul><ul><li>True “gaming the cloud” requires right software architecture </li></ul></ul>
  17. 17. Identify Metrics To Manage <ul><ul><li>Cloud suffering from lack of standard metrics to measure </li></ul></ul><ul><ul><li>Most metrics over focus on costs </li></ul></ul><ul><ul><li>Identify & trend metrics that matter to your organization </li></ul></ul><ul><ul><li>Some to consider: </li></ul></ul><ul><ul><li>Infrastructure Cost of Goods Sold (ICOGS) = current infrastructure costs / revenue </li></ul></ul><ul><ul><li>Infrastructure Utilization (IU) = utilization of metric / maximum available quantity </li></ul></ul><ul><ul><li>Cloud Elasticity (CE) = (ICOGS at projected 12 month maximum revenue – current ICOGS) / current ICOGS </li></ul></ul><ul><ul><li>Infrastructure Hourly Rate (IHR) = cost for all infrastructure in hour </li></ul></ul>Cloud Moneyball
  18. 18. Comparing Costs: Cloud vs. Physical <ul><ul><li>Cloud is more expensive than physical infrastructure for always-on </li></ul></ul><ul><ul><li>Cloud wins year 1, physical wins year 2+ </li></ul></ul><ul><ul><li>Cloud optimized for </li></ul></ul><ul><ul><ul><li>Sometimes-on infrastructure </li></ul></ul></ul><ul><ul><ul><li>Always-on where physical alternative underutilized </li></ul></ul></ul><ul><ul><li>Typical reasons for underutilization </li></ul></ul><ul><ul><ul><li>Incorrect growth projections </li></ul></ul></ul><ul><ul><ul><li>Building for peak utilization </li></ul></ul></ul><ul><ul><ul><li>Building ahead of growth </li></ul></ul></ul><ul><ul><ul><li>Building for temporary utilization </li></ul></ul></ul>
  19. 19. Demo: Sonian CloudControl (internal app)