TDWI India Chapter 2011 Feb 05 Hosted at Intel, Presentation from Praveen Habbagodi, Director, Akamai Technologies

1. Data warehousing infrastructure on Cloud
Praveen Hebbagodi
Director of Engineering, Akamai Technologies
Akamai Confidential

2. Agenda
• Introduction to Akamai
• Akamai BI Solutions
• Data warehousing platform
• Features
• Architecture
• Operations
• Conclusions
Akamai Confidential Powering a Better Internet ©2011 Akamai

3. The Akamai Network…
… a large-scale on-demand distributed computing platform
Accelerating Daily Traffic of:
• 3+ Tbps
• 11+ million hits per second
• 10+ million concurrent streams
• 800+ billion deliveries/day
• 30+ petabytes/day
Connecting:
- 88,000+ Servers • 465 million unique IP addresses
- 1,100+ Networks • From 234 countries
- 1,600+ Locations
- 650+ Cities
Deflecting Attack Traffic:
- 71 Countries
• From 198 countries
• Targeting 10,000 unique ports
Akamai Confidential Powering a Better Internet ©2011 Akamai

4. Major services provided by Akamai
Content delivery:
• HTTP/S (15-30% of total HTTP traffic!)
• Live and On-Demand Streaming
Application delivery:
• Web Application Acceleration
• Dynamic Site Acceleration
• EdgeComputing
• IP Application Acceleration
Example applications:
• Online commerce, media delivery,
B2B/B2C applications, software downloads,
social networking sites, …
• You likely use many of our services each day
Akamai Confidential Powering a Better Internet ©2011 Akamai

5. Akamai BI Solutions
Helps our customers to get deeper insights into their audience and
content usage in the context of their business
Example Solutions:
Media Analytics
• A comprehensive solution for
content and audience intelligence
for broadband media
• Features
• Dashboards for Engagement
Overview, Ad Optimizations and
Content Usage
• Standard Reports with detailed
engagement and audinece
information
• Custom dimensions and reports to
suit business specific needs
Akamai Confidential Powering a Better Internet ©2011 Akamai

6. Akamai BI Solutions: Examples
QOS Monitor
• Real-time quality of
service monitoring
solution for online media
delivery
• Features
• Set thresholds for
breaches, find root causes
and resolve issues using
“Notifications”
• Live Real-time monitoring
console with data
aggregation as fine as 30
sec.
• 20 standard reports &
dashboards for historical
diagnostics & debugging.
Akamai Confidential Powering a Better Internet ©2011 Akamai

7. Features
• Intuitive dashboard & report builder UI with advanced
visualizations
• Over 50 Standard dimensions & metrics
• Support for any customer specific dimensions
• Regular expression extraction
• Plug-in API
• Ad-hoc query, drill-down
• Lookup tables
• Dashboards & reports can be provisioned on the fly
• Real-time notifications
• Data access via web interface, SOAP API, Email & download
(CSV, PDF, HTML)
Akamai Confidential Powering a Better Internet ©2011 Akamai

8. Data warehousing Platform
xml
xml
Reporting Data Data xml
Storage Processing Data
Collection
Data Sources
Analytics workflow programmable via portal
• Data sources, filters, metrics, dimensions, reports, dashboards configured via
xml metadata
Distributed data collection in the Cloud
• Data sources: end user machines (beacons), edge server logs, agents
• Filtering, and partial aggregation at the source and in collection layers
• Facilitates scalability and better utilization of resources
Akamai Confidential Powering a Better Internet ©2011 Akamai

9. Data warehousing Platform
xml
xml
Reporting Data Data xml
Storage Processing Data
Collection
Data Sources
Data Processing is a flexible map-reduce framework
• Dataflow graph of map-reduce operations
• Enhancements for better latencies, scheduling optimizations
• Faster message passing interfaces (network, in-memory)
Akamai Confidential Powering a Better Internet ©2011 Akamai

10. Data-warehousing Platform
xml
Processing
Reporting
Data
Artist Name
Storage
Time
Data abstraction is a set of data cubes
• Supports fast slice-and-dice, drill-down operations,…
Data cubes are physically realized in distributed columnar DB
• SQL interface, column compression, bitmap indexes
• In-situ updates, write-optimized store
• Sharding and cluster management
Akamai Confidential Powering a Better Internet ©2011 Akamai

11. Approach to Operations
• Treat failures as normal
• Build in layers of redundancy
• At all levels: geo/network, with in a cluster
• Multi-path communications
• Weaker data consistency models
• Zoning
• Dynamic Configuration
• Software Installs
• Design systems that run themselves
• Autonomic response where appropriate
Akamai Confidential Powering a Better Internet ©2011 Akamai

12. Conclusions
• Being on cloud facilitates in building highly scalable platform for “big
data” applications
• Design for failures
• Build redundant systems at all levels
• Multiple levels of fault-tolerance
• Automation, autonomics, more automation…
• Avoid “manual changes”
• They will happen, so have good process to minimize/track
• Deterministic software and config management system
• Converges to consistent state & built-in safe roll-back
• Good tools for understanding system behavior and data quality
• Sophisticated tools for capacity management and performance
monitoring
Akamai Confidential Powering a Better Internet ©2011 Akamai

13. To find out more about Akamai…
More info: www.akamai.com
Contacting me:
Praveen Hebbagodi
phebbagodi@akamai.com
Technical publications:
http://www.akamai.com/html/perspectives/techpubs.html
Jobs:
http://www.akamai.com/html/careers/index.html
http://twitter.com/akamaijobsindia
Questions?
Akamai Confidential Powering a Better Internet ©2011 Akamai

14. Thank you
Akamai Confidential