SQL or NoSQL How to
choose Venu Anuganti Jan 2011 http://venublog.com/

Who am I <ul><li>Data Architect,
Database Kernel / Internals Engineer </li></ul><ul><li>Implement and Scale SQL, NoSQL, Analytics and Data Warehouse solutions </li></ul><ul><li>Large scale data handling for Games, Social Networking, SaaS, Click Tracking, Recommendation, Advertisement, Mobile and SEM marketing </li></ul><ul><li>Blog: http://venublog.com/ </li></ul>

Agenda <ul><li>Buzz around SQL and
NoSQL </li></ul><ul><li>How to design and implement Data Flow Architecture </li></ul><ul><li>How to choose Data Store Solution </li></ul><ul><ul><li>Performance, Scalability and Availability </li></ul></ul><ul><ul><li>SQL vs NoSQL </li></ul></ul><ul><ul><li>Where SQL and NoSQL fits </li></ul></ul><ul><ul><li>Types of SQL and NoSQL data stores </li></ul></ul><ul><li>Evaluation & Decision Making </li></ul>

Buzzzzzzz <ul><li>Why everyone is talking
about NoSQL </li></ul><ul><li>What is happening to SQL </li></ul><ul><li>Does that mean end of SQL ? NoSQL era begins ? </li></ul><ul><li>Why nobody talks about large SQL implementations ? </li></ul>

<ul><li>Evolution of Data Architecture </li></ul>

Data Architecture <ul><li>No standard solution
that fits to all </li></ul><ul><li>Business and it’s data defines architecture </li></ul><ul><li>It’s all about solving problems </li></ul><ul><li>You need to find the right tool that does the job </li></ul>

Traditional Architecture <ul><li>Relational database is
everything </li></ul><ul><ul><li>SQL </li></ul></ul><ul><ul><li>Embedded </li></ul></ul><ul><ul><li>Client-Server based </li></ul></ul><ul><li>Data Stack </li></ul><ul><ul><li>Web, CDN, Load Balancers, Application, Database and Storage </li></ul></ul>

Traditional Scalability … <ul><li>Scale-up </li></ul><ul><ul><li>Memory
and hardware has limitations </li></ul></ul><ul><li>Scale-out </li></ul><ul><ul><li>Scaling reads </li></ul></ul><ul><ul><ul><li>Cache is the king </li></ul></ul></ul><ul><ul><ul><ul><li>Query cache </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Memcache </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Olap </li></ul></ul></ul></ul><ul><ul><ul><li>Pre-fetching </li></ul></ul></ul><ul><ul><ul><li>Replication </li></ul></ul></ul><ul><ul><li>Scaling writes </li></ul></ul><ul><ul><ul><li>Redundant disk arrays, RAID </li></ul></ul></ul><ul><ul><ul><li>Sharding </li></ul></ul></ul>

Common Problems… <ul><ul><li>Relational model is
heavy : Parsing, Locking, Logging, Buffer pool and threads </li></ul></ul><ul><ul><li>Not every case can work within single node SMP </li></ul></ul><ul><ul><li>Sharding does not solve all problems </li></ul></ul><ul><ul><ul><li>Cross shard or join between shards </li></ul></ul></ul><ul><ul><ul><li>Need to update across multiple shards within a transaction </li></ul></ul></ul><ul><ul><ul><li>Shard failure </li></ul></ul></ul><ul><ul><ul><li>Online schema changes without taking the shard offline </li></ul></ul></ul><ul><ul><ul><li>Add or replace shards in-line </li></ul></ul></ul>

Evolution <ul><li>Data is growing rapidly
on day by day </li></ul><ul><li>Motivated by the needs of large web applications </li></ul><ul><li>Hardware is not emerging as that of data growth </li></ul><ul><li>Things are moving to Cloud and API driven </li></ul><ul><li>Social networking and Cloud makes hard to scale using traditional way </li></ul>

Data is the Business <ul><li>Lot
of new business models are DATA centric </li></ul><ul><ul><li>Real-time and Interactive </li></ul></ul><ul><ul><li>Big Data </li></ul></ul><ul><ul><ul><li>Millions of user base, clients, customers, applications, … </li></ul></ul></ul><ul><ul><ul><li>Tera bytes to peta bytes of data on day to day </li></ul></ul></ul><ul><ul><li>Business can only grow if they can properly make use of data </li></ul></ul><ul><ul><ul><li>Statistics, Reporting </li></ul></ul></ul><ul><ul><ul><li>Real-time </li></ul></ul></ul><ul><ul><ul><li>Re-targeting </li></ul></ul></ul><ul><ul><ul><li>Recommendation </li></ul></ul></ul><ul><ul><li>Examples of data driven companies </li></ul></ul><ul><ul><ul><li>Facebook, Twitter, LinkedIn, Zynga, Groupon, Quora, Apple AppStore, FourSquare, any API Driven, all most all new emerging companies </li></ul></ul></ul>

Solution that works <ul><li>Data architecture
is not just choosing a right data store, but should be a solution, with: </li></ul><ul><ul><li>Low In Cost (preferably open source, no hidden cost..) </li></ul></ul><ul><ul><li>Simple To Implement </li></ul></ul><ul><ul><li>High Performance </li></ul></ul><ul><ul><li>Highly Available </li></ul></ul><ul><ul><li>Highly Scalable </li></ul></ul><ul><ul><li>Highly Reliable </li></ul></ul><ul><ul><li>Highly Recoverable </li></ul></ul><ul><ul><li>Rapid Development </li></ul></ul><ul><ul><li>Zero Learning Curve </li></ul></ul><ul><ul><li>Ability to do online changes (schema or node or automatic) </li></ul></ul><ul><ul><li>Less Operational Maintenance </li></ul></ul><ul><ul><li>No firefighting on day to day </li></ul></ul>

NoSQL Solution Emerges <ul><li>Lot of
companies emerged to solve data centric problems </li></ul><ul><li>Big Table: Google started to implement massively distributed scalable system, followed by many, first foot step to the world of massive data scale </li></ul><ul><li>Many companies followed building scale-out architecture using commodity hardware </li></ul><ul><li>ACID was termed as bad for scaling, so relaxed consistency model came into picture </li></ul><ul><li>Google Big Table and Amazon Dynamo are notable </li></ul>

Relaxed Consistency <ul><li>Consistency is a
major bottleneck for scalability </li></ul><ul><li>People started implementing eventual consistency </li></ul><ul><li>CAP Theorem ( C onsistency, A vailability and P artition-tolerance) </li></ul><ul><ul><li>Consistency: “ Is the data I’m looking at now the same if I look at it somewhere else?” </li></ul></ul><ul><ul><li>Availability: “ What happens if my database goes down?” </li></ul></ul><ul><ul><li>Partitioning: “ What if my data is on different node?” </li></ul></ul><ul><ul><li>SQL – CA </li></ul></ul><ul><ul><li>NoSQL – AP </li></ul></ul><ul><li>http://venublog.com/2010/04/07/cap-theorem-eventual-consistency-nosql/ </li></ul>

<ul><li>Data Stores </li></ul>

Data Stores <ul><ul><li>3 Major Data
Store Solutions </li></ul></ul><ul><ul><ul><li>SQL, OLTP </li></ul></ul></ul><ul><ul><ul><ul><li>Relational, transactional processing </li></ul></ul></ul></ul><ul><ul><ul><li>Analytics, OLAP </li></ul></ul></ul><ul><ul><ul><ul><li>Data Warehousing, Analytics and reporting </li></ul></ul></ul></ul><ul><ul><ul><li>NoSQL </li></ul></ul></ul><ul><ul><ul><ul><li>Non relational, distributed, high performance and highly scalable </li></ul></ul></ul></ul>

SQL Stores <ul><li>Disk based storage
</li></ul><ul><li>Data is stored as table (row by row and columns – row store) </li></ul><ul><li>Mainly B-tree as the indexing mechanism </li></ul>

SQL Stores … <ul><li>Dynamic locking/
Lock free for concurrency control </li></ul><ul><li>Write-ahead log (WAL) / transactional log for crash recovery </li></ul><ul><li>SQL as the access language </li></ul>

SQL Stores <ul><li>Proven and widely
adopted </li></ul><ul><ul><li>MySQL </li></ul></ul><ul><ul><li>PostGreSQL </li></ul></ul><ul><ul><li>VoltDB </li></ul></ul><ul><ul><li>Clustrix </li></ul></ul><ul><ul><li>MySQL Cluster </li></ul></ul><ul><ul><li>ScaleDB </li></ul></ul><ul><ul><li>ScaleBase </li></ul></ul><ul><ul><li>DbShards </li></ul></ul><ul><ul><li>Oracle </li></ul></ul><ul><ul><li>SQL Server </li></ul></ul><ul><ul><li>DB2 </li></ul></ul><ul><ul><li>Sybase & … </li></ul></ul><ul><li>Supports </li></ul><ul><ul><li>ACID </li></ul></ul><ul><ul><li>Crash recovery </li></ul></ul><ul><ul><li>DDL, DML, DCL </li></ul></ul>

Analytic Stores <ul><li>Data warehousing, mainly
for large sets of data </li></ul><ul><li>Data marts, Dimensional, Fact and Aggregate tables </li></ul><ul><li>ETL, BI, Reporting, Analytics </li></ul><ul><li>Columnar and Compression is the key </li></ul><ul><li>OLAP Cubes built-in or middle-tier </li></ul><ul><li>Mostly SQL and also MDX driven </li></ul>

Analytic Stores <ul><li>Columnar data warehouse
solutions </li></ul><ul><ul><li>GreenPlum (+ DCA appliance) </li></ul></ul><ul><ul><li>Vertica (Break through, I love it) </li></ul></ul><ul><ul><li>Aster </li></ul></ul><ul><ul><li>ParAccel </li></ul></ul><ul><ul><li>InfoBright (MySQL based) </li></ul></ul><ul><ul><li>InfiniDB (open source, Calpont appliance) </li></ul></ul><ul><ul><li>Netezza (appliance) </li></ul></ul><ul><ul><li>XtremeData dbX (appliance) </li></ul></ul><ul><ul><li>TeraData </li></ul></ul>

NoSQL Stores <ul><li>Does not mean
No to SQL </li></ul><ul><li>Actually No t only SQL </li></ul><ul><li>Data store that may not require fixed table schemas </li></ul><ul><li>Mainly derived from Google BigTable and Amazon Dynamo </li></ul>

NoSQL Stores … <ul><li>Non relational,
schema free </li></ul><ul><li>Distributed, ability to horizontally scale </li></ul><ul><li>Simple CLI or API protocol </li></ul><ul><li>Eventually consistent, depends … </li></ul><ul><li>Limitations of SQL to scale large data </li></ul><ul><li>Ability to dynamically define new attributes </li></ul>

NoSQL Stores … <ul><li>Multiple Types
based on storage architecture </li></ul><ul><ul><li>Key Value, KV </li></ul></ul><ul><ul><li>Document </li></ul></ul><ul><ul><li>Graph </li></ul></ul><ul><ul><li>Column Family </li></ul></ul>

NoSQL Stores <ul><ul><li>Key-Value Stores </li></ul></ul><ul><ul><ul><li>Dynamo
Clones </li></ul></ul></ul><ul><ul><ul><li>Membase </li></ul></ul></ul><ul><ul><ul><li>Riak </li></ul></ul></ul><ul><ul><ul><li>Redis </li></ul></ul></ul><ul><ul><ul><li>Tokyo Cabinet </li></ul></ul></ul><ul><ul><ul><li>Voldemort </li></ul></ul></ul><ul><ul><li>Document Stores </li></ul></ul><ul><ul><ul><li>MongoDB </li></ul></ul></ul><ul><ul><ul><li>CouchDB </li></ul></ul></ul><ul><ul><li>Column Family </li></ul></ul><ul><ul><ul><li>BigTable Clones </li></ul></ul></ul><ul><ul><ul><li>Cassandra </li></ul></ul></ul><ul><ul><ul><li>HBase </li></ul></ul></ul><ul><ul><ul><li>HyperTable </li></ul></ul></ul><ul><ul><li>Graph Databases </li></ul></ul><ul><ul><ul><li>Neo4J </li></ul></ul></ul><ul><ul><ul><li>InfoGrid </li></ul></ul></ul><ul><ul><ul><li>AllegroGraph </li></ul></ul></ul><ul><ul><ul><li>FlockDB </li></ul></ul></ul>

<ul><li>What they are good at
</li></ul><ul><li>& </li></ul><ul><li>How to choose </li></ul>

Basic Decision Principles <ul><li>Do not
over architect from day-1, it’s overkill </li></ul><ul><li>Startups can’t afford to spend time </li></ul><ul><li>Understand business and implement with simple well known solutions to begin with </li></ul><ul><li>Do not follow the models, just inspire from the problem solving </li></ul><ul><li>Engineering talent is crucial, make sure you have right resources </li></ul><ul><li>Evaluate and implement new solutions as the business grows </li></ul>

Basic Decision Principles … <ul><li>High
availability & disaster recovery is a must </li></ul><ul><li>Understand pros and cons of each and every design model, and weigh towards the best interest of the company </li></ul><ul><li>Remember some of the big outage stories </li></ul><ul><ul><li>Tumblr, FourSuaure & Twitter </li></ul></ul><ul><li>Lean towards community winner and widely adopted </li></ul><ul><li>Do not lean towards only performance, unless you can create the state of the data back </li></ul>

SQL – Good <ul><ul><li>High Performance
OLTP, Transactions, ACID </li></ul></ul><ul><ul><li>Structured, SQL Access , portability and tools </li></ul></ul><ul><ul><li>Small amounts of data, typically < 500G per server, </li></ul></ul><ul><ul><li>supports inline UPDATE, DELETE, multi-condition/rows </li></ul></ul><ul><ul><li>Relational model at data store, application independent </li></ul></ul><ul><ul><li>Many tables with different types of dtaa </li></ul></ul>

SQL – Good <ul><ul><li>Simple or
complex aggregation </li></ul></ul><ul><ul><li>Statistics, reports at data store level </li></ul></ul><ul><ul><li>Need access to more than one tuple of information </li></ul></ul><ul><ul><li>Results based on multiple search conditions </li></ul></ul><ul><ul><ul><li>SELECT foo FROM bar where X=1 and Y=2 </li></ul></ul></ul><ul><ul><li>Fetching of ordered or array of data </li></ul></ul><ul><ul><li>Compatible with many tools </li></ul></ul>

SQL – Bad <ul><ul><li>SQL complexity,
parsing cost </li></ul></ul><ul><ul><li>Learning and relational model design </li></ul></ul><ul><ul><li>Performance and Scalability </li></ul></ul><ul><ul><ul><li>Strictly single node </li></ul></ul></ul><ul><ul><ul><li>Sharding causes more trouble operationally </li></ul></ul></ul><ul><ul><ul><li>Operational maintenance, fire fighting </li></ul></ul></ul><ul><ul><li>Puts a break to rapid development cycles </li></ul></ul>

NoSQL - Good <ul><ul><li>Fits very
well for volatile data </li></ul></ul><ul><ul><li>High read or write throughput </li></ul></ul><ul><ul><li>Automatic horizontal scalability (Consistent hashing) </li></ul></ul><ul><ul><li>Simple to implement, no investment for developers to design and implement relational model </li></ul></ul><ul><ul><li>Application logic defines object model </li></ul></ul><ul><ul><li>Support of MVCC in some form </li></ul></ul><ul><ul><li>Compaction and un-compaction happens at top tier </li></ul></ul><ul><ul><li>In-memory or disk based or combination </li></ul></ul>

NoSQL - Good <ul><ul><li>Rapid development
cycles, programmer friendly </li></ul></ul><ul><ul><li>Reduces the footprint at data store level </li></ul></ul><ul><ul><li>NoSQL in general faster than SQL </li></ul></ul><ul><ul><li>Supports INSERT, DELETE, SELECT </li></ul></ul><ul><ul><li>Data is distributed by KEY over nodes </li></ul></ul><ul><ul><li>Lists, sets, queues, pub-sub are also supported by some NoSQL – Redis </li></ul></ul><ul><ul><li>S3 can handle large blobs; not all NoSQL can handle it </li></ul></ul>

NoSQL - Bad <ul><ul><li>Packing and
Un-packing of each key </li></ul></ul><ul><ul><li>Lack of relation from one key to another </li></ul></ul><ul><ul><li>Need whole value from the key; to read/write any partial information </li></ul></ul><ul><ul><li>No security or authentication </li></ul></ul><ul><ul><li>Data store is merely a storage layer, can’t be used for: </li></ul></ul><ul><ul><ul><li>Analytics </li></ul></ul></ul><ul><ul><ul><li>Reporting </li></ul></ul></ul><ul><ul><ul><li>Aggregation </li></ul></ul></ul><ul><ul><ul><li>Ordered values </li></ul></ul></ul>

SQL/NoSQL – Good and Bad
<ul><ul><li>Performance mainly depends on amount of memory </li></ul></ul><ul><ul><li>Disk bound both takes a hit </li></ul></ul><ul><ul><ul><li>SQL has advantage due to sequential and read-ahead </li></ul></ul></ul><ul><ul><li>Optimization towards frequently accessed data </li></ul></ul><ul><ul><ul><li>SQL engines maintain LRU </li></ul></ul></ul><ul><ul><li>SQL Engines are proven and widely in use </li></ul></ul><ul><ul><li>NoSQL is pretty much new; but marching … </li></ul></ul>

Cache <ul><ul><li>MySQL HandlerSocket </li></ul></ul><ul><ul><ul><li>InnoDB bufferpool
acts as cache </li></ul></ul></ul><ul><ul><ul><li>No explicit cache needed, no write invalidation </li></ul></ul></ul><ul><ul><li>Write Through Cache (WTC) is a good candidate for high reads or writes </li></ul></ul><ul><ul><ul><li>Gaming world really need this </li></ul></ul></ul><ul><ul><ul><li>Membase or periodic flush to persistent storage layer </li></ul></ul></ul><ul><ul><li>Flash cache can also help to scale IO bound workloads </li></ul></ul><ul><ul><ul><li>We might see them pitching in private cloud slowly </li></ul></ul></ul>

Document Store <ul><ul><li>Document Stores </li></ul></ul><ul><ul><ul><li>Supports
complex data model than KV </li></ul></ul></ul><ul><ul><ul><li>Good at handling content management, session, profile data </li></ul></ul></ul><ul><ul><ul><li>Multi index support </li></ul></ul></ul><ul><ul><ul><li>Dynamic schemas, Nested schemas </li></ul></ul></ul><ul><ul><ul><li>Auto distributed, eventual consistency </li></ul></ul></ul><ul><ul><ul><li>MVCC (CouchDB) or automic (MongoDB) </li></ul></ul></ul><ul><ul><li>MongoDB, SimpleDB: widely adopted in this space </li></ul></ul><ul><ul><li>Use Case: Search by complex patterns & CRUD apps </li></ul></ul>

Column Family Store <ul><ul><li>Hbase (Apache),
Cassanda (Facebook) and HyperTable (Bidu) </li></ul></ul><ul><ul><ul><li>Hbase – CA </li></ul></ul></ul><ul><ul><ul><li>Cassandra – AP </li></ul></ul></ul><ul><ul><li>Model consists of rows and columns </li></ul></ul><ul><ul><li>Scalability: Splitting of both rows and columns </li></ul></ul><ul><ul><ul><li>Rows are split across nodes using primary key, range </li></ul></ul></ul><ul><ul><ul><li>Columns are distributed using groups </li></ul></ul></ul><ul><ul><ul><li>Horizonal and vertical partitioning can be used simultaneous </li></ul></ul></ul><ul><ul><li>Extension of document store </li></ul></ul><ul><ul><li>HBase uses HDFS; Pig, Hive, Cascading can help </li></ul></ul><ul><ul><li>Use case: Grouping of frequently used and un-used over data centers / stream of writes </li></ul></ul>

Graph Store <ul><ul><li>Social Graph </li></ul></ul><ul><ul><li>Relationship
between entities </li></ul></ul><ul><ul><li>Data modeling on social networks </li></ul></ul><ul><ul><li>Common Use Cases </li></ul></ul><ul><ul><ul><li>List of friends </li></ul></ul></ul><ul><ul><ul><li>Recommendation system </li></ul></ul></ul><ul><ul><ul><li>Following </li></ul></ul></ul><ul><ul><ul><li>Followers </li></ul></ul></ul><ul><ul><ul><li>Common Connections </li></ul></ul></ul><ul><ul><ul><li>FAN IN/OUT </li></ul></ul></ul>

Cloud Data Stores <ul><li>Database Cloud
Services </li></ul><ul><ul><li>Xeround (MySQL) </li></ul></ul><ul><ul><li>Microsoft SQL Azure Database (SQL Server) </li></ul></ul><ul><ul><li>SimpleDB (NoSQL) </li></ul></ul><ul><ul><li>Google App Engine (NoSQL) </li></ul></ul><ul><ul><li>SalesForce Database.com (Oracle) </li></ul></ul><ul><ul><li>ClearDB (MySQL) </li></ul></ul>

SUMMARY <ul><ul><li>Pick right data model
for the right problem </li></ul></ul><ul><ul><li>Understand the data storage and use cases on read, write and growth patterns; and then come up with a plan to implement. Use case dictates everything. </li></ul></ul><ul><ul><li>Compare pros and cons, and weigh towards the one that helps business </li></ul></ul><ul><ul><li>Public cloud, private cloud or data center also dictates what model to choose </li></ul></ul><ul><ul><li>You need right people </li></ul></ul>

Finally … <ul><ul><li> SQL </li></ul></ul><ul><ul><li>Works
great, can’t scale for large data </li></ul></ul><ul><ul><li>NoSQL </li></ul></ul><ul><ul><li>Works great, can’t fit for all </li></ul></ul><ul><ul><li> SQL + NoSQL </li></ul></ul>

Questions ? <ul><li>http://venublog.com/ </li></ul><ul><li>[email_address] </li></ul><ul><li>Twitter:
@vanuganti </li></ul>