These slides explain briefly about the big data concept, NoSQL database systems and the ElasticSearch technology which are quite modern concepts

1. BIG DATA, NOSQL &
ELASTICSEARCH
BY SANURA HETTIARACHCHI
INTERN AT VOCANIC

2. GRAPH DATABASE
• Database that uses graph structures with nodes, edges, and properties to
represent and store data.
• Nodes represent entities.
• Properties are pertinent information that relate to nodes.
• Edges represent the relationship between the two. Most of the important
information is really stored in the edges.

3. ADVANTAGES & DISADVANTAGES
• Faster for associative data sets
• Map more directly to the structure of object-oriented applications.
• Do not typically require expensive join operations.
• Depend less on a rigid schema, they are more suitable to manage ad hoc and
changing data with evolving schemas.
o Relational databases are typically faster at performing the same operation on
large numbers of data elements.
o Relational databases are well known.

4. BIG DATA
• Any collection of data sets so large and complex that it becomes difficult to
process using traditional data processing applications.
• Require "massively parallel software running on tens, hundreds, or even
thousands of servers"

5. FACTORS OF GROWTH, CHALLENGES AND
OPPORTUNITIES OF BIG DATA
• Volume – the quantity of data that is generated.
• Variety – category to which Big Data belongs to.
• Velocity – how fast the data is generated and processed to meet the demands.
• Variability – the inconsistency which can be shown by the data at times.
• Complexity – data needs to be linked, connected and correlated in order to be able
to grasp the information.

6. HORIZONTAL & VERTICAL SCALING
• Horizontal scaling - scale by adding more machines to your pool of resources.
• Vertical scaling - scale by adding more power (CPU, RAM, etc.) to your existing
machine.
• Horizontal scaling is easier to scale dynamically by adding more machines into
the existing pool.
• Vertical scaling is often limited to the capacity of a single machine
• Horizontal scaling are the Cloud data stores, e.g. DynamoDB, Cassandra ,
MongoDB
• Vertical scaling is MySQL - Amazon RDS (The cloud version of MySQL)

7. ACID
• Atomicity - all of a transaction happens, or none of it does.
• Consistency - data will be consistent.
• Isolation - one transaction cannot read data from another transaction that is not yet
completed.
• Durability - once a transaction is complete, it is guaranteed that all of the changes have
been recorded to a durable medium.

8. NOSQL
• Basically a large serialized object store
• (mostly) retrieve objects by defined ID
• In general, doesn’t support complicated queries
• Doesn’t have a structured schema
• Recommends de-normalization
• Designed to be distributed (cloud-scale) out of the box
• Because of this, drops the ACID requirements
• Any database can answer any query
• Any write query can operate against any database and will “eventually” propagate to other
distributed servers

9. BASE-THE OPPOSITE OF ACID
• Basically Available – guaranteed availability
• Soft-state – the state of the system may change, even without a query
(because of node updates)
• Eventually Consistent – the system will become consistent over time

10. WHY NOSQL?
• Today, data is becoming easier to access and capture through third parties such as
Facebook, Google+ and others.
• Personal user information, social graphs, geo-location data, user-generated content
and machine logging data are just a few examples where the data has been
increasing exponentially.
• To use the above services properly requires the processing of huge amounts of data.
Which SQL databases are no good for, and were never designed for.
• NoSQL databases have evolved to handle this huge data properly.

11. CAP THEOREM
• Consistency - This means that the data in the database
remains consistent after the execution of an operation.
• Availability - This means that the system is always on,
no downtime.
• Partition Tolerance - This means that the system
continues to function even if the communication
among the servers is unreliable
Distributed systems must be partition tolerant , so we
have to choose between Consistency and Availability.

12. DIFFERENT TYPES OF NOSQL
 Column Store
• Column data is saved together, as opposed to
row data
• Super useful for data analytics
• Hadoop, Cassandra, Hypertable
 Key-Value Store
• A key that refers to a payload
• MemcacheDB, Azure Table Storage, Redis
 Document / XML / Object Store
• Key (and possibly other indexes) point at a
serialized object
• DB can operate against values in document
• MongoDB, CouchDB, RavenDB
 Graph Store
• Nodes are stored independently, and the
relationship between nodes (edges) are stored
with data

13. RDBMS VS NOSQL
RDBMS NoSQL
Structured and organized data Semi-structured or unorganized data
Structured Query Language (SQL) No declarative query language
Tight consistency Eventual consistency
ACID transactions BASE transactions
Data and Relationships stored in tables No pre defined schema

14. ELASTICSEARCH
• Elasticsearch is a flexible and powerful open source, distributed, real-time
search and analytics engine.
• It is based on Apache Lucene which is a free open source information retrieval
software library, originally written in Java.
• ElasticSearch is distributed, which means that indices can be divided
into shards and each shard can have zero or more replicas.

15. OVERVIEW
• Real time
Elasticsearch supports real-time GET requests, which makes it
suitable as a NoSQL solution.
• Distributed
It is built to scale horizontally out of the box. As you need more
capacity, just add more nodes, and let the cluster reorganize itself to
take advantage of the extra hardware.
• High Availability
They will detect and remove failed nodes, and reorganize themselves
to ensure that your data is safe and accessible.

16. OVERVIEW
• Multi Tenancy
A cluster can host multiple indices which can be queried
independently or as a group.
• Document Oriented
Store complex real world entities in Elasticsearch as structured JSON
documents. All fields are indexed by default, and all the indices can
be used in a single query, to return results at breath taking speed.
• Conflict Management
Optimistic version control can be used where needed to ensure that
data is never lost due to conflicting changes from multiple processes

17. THANK YOU