CloudLab: A File System Friendly Key Value Store

A File-System Friendly Key Value Store Cloud-Scale File Serving and Storage MaxiScale FLEX Software Platform @garyorenstein | go@maxiscale.com April 2010 Page MaxiScale

The Era of Billions 10 billion Tweets 10 billion downloads 3 billion photos/month 12 billion views January April 2010 Page MaxiScale

Beyond Capacity to Users and Objects Everyone is a content consumer AND creator April 2010 Page MaxiScale

Impact of The Era of Billions ,[object Object],April 2010 Page MaxiScale

Cloud Applications Requirements: Fast, Scalable, Affordable “Light” Structured Data Retrieval Page April 2010 MaxiScale

Basics Anatomy of a Key-Value Store April 2010 Page Source: http://dominicfallows.co.uk MaxiScale

MaxiScale Key Value Store Benefits Page April 2010 MaxiScale

Breakthrough FLEX Software Platform ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Web / App Servers File Serving and Storage April 2010 Page Internet MaxiScale

MaxiScale FLEX Platform: Core Technologies Distributed Platform Single Namespace Distributed Metadata and Data Peer Sets Optimized File Serving ,[object Object],[object Object],[object Object],[object Object],[object Object],Linux and Windows Clients Small File Performance (key-value store) High-Density Storage Scale-Out NAS April 2010 Page MaxiScale Storage I/O Bandwidth CPU

Simplicity of a Single Namespace April 2010 Page Small File Performance (key-value store) Scale-Out NAS Single mount point / /NFR/ /SFR/ ,[object Object],[object Object],[object Object],[object Object],[object Object],MaxiScale

FLEX Software Platform In Action Utmost system efficiency from client to network to cluster Peer Set April 2010 Page MaxiScale

Fast Lookups Directly from Disk ,[object Object],[object Object],[object Object],[object Object],April 2010 Page Index (RAM) Metadata and Data (Disk) MaxiScale

Performance Counts ,[object Object],[object Object],[object Object],[object Object],[object Object],Files/spindle/sec Read tests with 32KB files and small file optimizations April 2010 Page MaxiScale

Selecting a Key Value Store April 2010 Page MaxiScale

Accelerating Web content serving Page Brute-force speed-up Hardware and development cost Performance Efficient key-value lookup without database queries Slash cost Fast lookup Web-scale April 2010 MaxiScale Do-it-yourself key value store RAM Cache RAM Cache Relational DB (MySQL …) Relational DB (MySQL …) MaxiScale key value store

Page Which key value store and file system are right for your application? High Availability? Key Length? Lookup Latency? Number of Keys? Cost? Size of Values? Data Storage Density? April 2010 MaxiScale

Key value store landscape Page Circles represent relative usable capacity at load Memcache High RAM cost in large quantities Cost MaxiScale Single disk I/O Reads Petabyte scale Relational DB Long lookup latency File systems Long read latency Solid state disk High flash cost in large quantities ,[object Object],[object Object],[object Object],[object Object],[object Object],Latency April 2010 MaxiScale

Placing Key Value Stores in Context April 2010 Page MaxiScale Relational Databases Key-Value Stores File System Friendly Key-Value Stores File Systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Social Networking Case Study Page April 2010 MaxiScale

Social networking application example ,[object Object],[object Object],[object Object],[object Object],[object Object],April 2010 Page 175 TB content repository across multiple NAS systems Sharded database instances for profiles, messages… Application Servers Web Servers MaxiScale

Social networking application example ,[object Object],[object Object],[object Object],[object Object],April 2010 Page Application Servers Web Servers Financial savings Capital expense: 60% Operating expense: $105K First-year ROI: 187% News feed, message, and profile data Direct photo retrieval and upload MaxiScale Value

A Deeper Look at Photo Serving Case Study Page April 2010 MaxiScale

Examples of Photo Serving Requirements Page April 2010 MaxiScale

[object Object],[object Object],[object Object],[object Object],High-Resolution Original Images IMG_01234.JPG IMG_04567.JPG IMG_06789.JPG Page April 2010 MaxiScale

[object Object],[object Object],[object Object],[object Object],Pre-Rendered Thumbnails IMG_01234.JPG 96 pixels wide 480 pixels wide 640 pixels wide Application Process Page April 2010 MaxiScale

User Profiles ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Page April 2010 MaxiScale

MaxiScale Photo Serving Features and Benefits Page April 2010 MaxiScale MaxiScale Feature Benefit POSIX-compliant file system hierarchy with NFR Segment users, album sets, and albums into traditional directory structure Small File Repository (SFR) ,[object Object],[object Object],[object Object],Key Value Store ,[object Object],[object Object],[object Object],Photo serving feature set leads to reduced infrastructure costs to meet user demand.

Delivering Cloud-Scale File Serving and Storage ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Page MaxiScale helps companies handle relentless pressure to store and serve data cost-effectively April 2010 MaxiScale

Thank You @garyorenstein | go@maxiscale.com @maxiscale | www.MaxiScale.com Page April 2010 MaxiScale

CloudLab: A File System Friendly Key Value Store

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à CloudLab: A File System Friendly Key Value Store

Similaire à CloudLab: A File System Friendly Key Value Store (20)

Dernier

Dernier (20)

CloudLab: A File System Friendly Key Value Store