Con8862 no sql, json and time series data

NoSQL , JSON and TimeSeries
Data - CON8862
Anuj Sahni, Principal Product Manager
Ashok Holla, Senior Sales Consultant

Safe Harbor
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into
any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality
described for Oracle’s products remains at the sole discretion of Oracle.

Agenda
 Why JSON ?
 Its role in NoSQL and Big Data
 NoSQL Use Cases
 Oracle NoSQL DB overview
 Architecture
 Data Modeling using JSON schema
 Time Series case study and demo

Brief History of data Interchange
 10 years ago XML was primary data exchange format
– Vast improvement over SGML (Standard Generalized Markup Language)
– Enabled people to exchange documents across HTTP
Language of Internet

Brief History of data Interchange (cont…)
 Last few years, a bold transformation happening in world of data
interchange
– JSON (JavaScript Object Notation) emerged as an alternative to XML
 looks more data-structure like,
 Light weight/bandwidth-non-intensive,
 language independent,
Language of Internet

What made JSON Cool ?
 API
– No business value gain operating in silos
– REST replacing SOAP as data transfer
protocol
 The Internet of Things
– “JSON better adapted to devices with
limited capabilities”
 Full-stack JavaScript
– JavaScript is new hot
– Node.js gone mainstream
 Big Data

Thoughts Things Processes
Newer Challenges

Get Fast Answers
to New Questions
What is Big data ?

NoSQL and Big Data
Where did it come from?
SQL
JDBC,
ODBC
General
Purpose
Managed
Schemas
Security,
Backups
Analytics
…
Distributed
Processing
Distributed,
Replicated
File System
Driver
Application
NoSQL databases
Flexible
Schemas
Sharded,
Replicated
Database
High Speed,
Simple Ops
More Flexible Schema
Management
Globally Distributed,
“Always On” data
Competitive Advantages
of “Fast Data”
Lower TCO,
commodity HW scale-out

Kinds of NoSQL Database
Based on Storage Model
Key Value Columnar Document Graph
Oracle NoSQL DB
Riak
Dynamo
Voldemort
Cassandra
HBase
MongoDB
CouchBase
Neo4J
GraphDB

Few common characteristics
 They all store data in de-normalized fashion
– Don’t support PK-> FK relationship (no cascade deletes)
– Don’t support complex Joins
– Don’t manage constraints (application’s job)
 Maintains data relationships using JSON/BSON representation
– Simple lookup operations by primary keys are generally enough
In NoSQL Databases

Agenda
 Why JSON ?
 NoSQL Use Cases
 Architecture

PointFlexibility Point
Web-Scale
Transaction Processing
High velocity, volume,
variety, low information
density data capture
Web browsing, Shopping
Carts, CDR processing,
Sensor data, Stock data
Web-Scale
Personalization
Guaranteed low latency
lookups for end-customers
Advertising, Product
Recommendations, Online
Catalogs, Social Media, Profile
Management, Personalization
Real-Time
Event Processing
Real time events trigger rule
that perform low latency lookups
Medical Monitoring, Factory
Automation, Oil & Gas, Geo-
location
NoSQL Database Use Case Summary

Use Case – Online Social Gaming
 Problem
– Very low latency requirements – Player movement must feel like a real time operation, while
being tracked on the server
– Extreme data velocity – Popular games, large scale user base (Farmville boasts 80 million active
users)
– Highly available – 24/7 sites
– Write heavy workloads
 Solution – Where to use a NoSQL Database?
– Player interaction data store – Database to track player movement and game interaction
– Game play statistics – Per player usage statistics
– Persistent chat store – For games that allow player communication via chat, the NoSQL
database is used as a persistent message store (auditing and COPA compliance)
Web Scale Personalization

Use Case – Online Social Gaming
Web Scale Personalization
JSON maps well
with HTML5/JS
based Client
Player interaction
stored as simple
JSON events
Easy to store
entire player state
reliably
NoSQLDBDriver
Application
Shard 2
Shard N
Shard 1
Schema Evolution
for agile
development
Transparent load
balancing On-demand
cluster expansion
REST

Use Case – Smart Meter Analytics
 Problem
– Real Time Access to time-series data – identity patterns hidden within the terabytes of data
 Identify theft,
 predict system failures ,
 better manage operations
– Large volume of machine data – 1000 fold increase in data collection
 Write heavy workloads
– Highly available – 24/7 sites
 Solution – Where to use a NoSQL Database?
– Fast time based queries – Daily, weekly, yearly consumption trends.
– Flexible data model – can adapt to various kind of sensors or changes in sensor format
– Scalable and Reliable performance – Easy to scale as workload increases because of more
sensors
Web Scale Transactions

Smart Meter Analytics
Web Scale Transactions
Analytic
Engine
User
Interface
Export
back to
Smart
Grid
NoSQLDBDriver
Application
Shard 2
Shard N
Shard 1
Time Series Data
Data coming in
many different
format
Data
coming
fast
Horizontally
scalable
database
JSON
Easy to model
keys for time
bases queries

Agenda
 Why JSON ?
 Introduction to NoSQL
 Architecture

 Flexible Key-Value Data Model
 ACID transactions
 Horizontally Scalable
 Highly Available
 Elastic Configuration
 Simple administration
 Intelligent Driver
 Commercial grade software and
support
Features
Oracle NoSQL Database
Application
Storage Nodes
Datacenter B
Storage Nodes
Datacenter A
Application
NoSQL DB Driver
Application
NoSQL DB Driver
Application
Java SE 6 (JDK 1.6.0 u25)+; Solaris or Linux
Scalable, Highly Available, Key-Value Database

Simple Data Model
 Simple, flexible key-value paradigm
 Simple operations – read/insert/update/delete, RMW support
 Ordered scan of key ranges
 Unordered scan of all data (non-transactional)
 Streaming API for LOBs
 Java and C APIs
 Key-value pairs

 Compact, highly efficient serialization
 Synergy with Hadoop
 Supports serialization from and to JSON strings
 Bindings from serialized formats to language constructs
 Easy to use mechanism for schema evolution
 Schema definition tracked with schema ID of data writer
 Schema versions can be opaque to readers
Oracle NoSQL – Why AVRO?

JSON support in Oracle NoSQL DB
Schema Creation

JSON support in Oracle NoSQL DB (cont…)
Data Manipulation

Schema Evolution

Agile development

Time Series Analysis
using
Oracle NoSQL database

Big Data Challenge
 High velocity of stock tick data generated
at a massive volume each day
 Millions of customers trading stocks on-
line based on the current value and the
historical trends.
 How do you ensure these buy/sell
transactions happens in real time?
 How do you store large volume of tick
data in a quick and consistent manner?
– So you can run the trend analysis on the time
series data.
UC - Stock Tick Analysis

Introduction – Oracle Investments Inc.
 Goal
– Ensure fast and consistent storage of
large volume of tick data
– Deliver a platform where customers can:
 Analyze historical trends by plotting
time-series data
 Trade stocks in real time by placing
buy/sell orders
 Challenge
– 100k’s customers trading concurrently.
– 10k’s of ticks each second
– Real time response required for order
execution

Introduction – Oracle Investments Inc.
 Value
– Real time:
 Data capture and random access to
time sensitive data
 Order processing
– Automatic data partitioning for easy
scalability
– Lowest $/ops for a consistent storage
– Highly Available system for 24/7 operation

Big Data Appliance
Oracle NoSQL DB
Oracle NoSQL DB as Data Store
NoSQL Driver
Read, Update
 Stores all key interactions required to
drive application. For example:
 User Profile
 Stock Tick Data
 Transaction information
 Why Oracle NoSQL Database?
 Highly Scalable
 Extremely performant
 Super low-latency

Architecture
Business Layer
Big Data Appliance
Oracle NoSQL DB
NoSQL Driver
Stock Tick Generator
Tick data every 5
seconds
Trading
Read, Update
Analytics
Time Series
Analysis
Order requests
User Profile
Updates
Order processing
Historical Data

NoSQL Data Representation
{"lasttimestamp":"1367251181219“,"lastclose":"45.17","lasthigh":"32.25","lastlow":"31.94","lastopen":"32.21","lastvolume":"26"
,
"stockArray":
[
{"timestamp":"1367245601219“, "value":"32.21“, "open":"32.23“, "high":"32.27“, "low":"32.14“,"volume":"4700"}
{"timestamp":"1367245602220“, "value":"32.15“, "open":"32.14“, "high":"32.27“, "low":"32.09“,"volume":"4000"}
{"timestamp":"1367245603319“, "value":"32.11“, "open":"32.09“, "high":"32.27“, "low":"32.14“,"volume":"3500“}
]
}
Denormalized Data

NoSQL Data Representation
 Get Stock Information for a Company
– Key = /SYMBOL/YYYY-MM/-/DD
– (Ex: /ORCL/2013-05/-/02 , Stock data for 2nd of May 13)
 Write = put(key, value)
 Read = get(key)

Demo

Oracle NoSQL DB Resources
 Support via OTN forums and Oracle Support process
 OTN Forum:
– Forum Home » Big Data » NoSQL Database
– forums.oracle.com/forums/forum.jspa?forumID=1388
 Oracle.com:
– www.oracle.com/us/products/database/nosql/overview/index.html
 OTN (including documentation and download):
– www.oracle.com/technetwork/products/nosqldb/overview/index.html
Support

Oracle NoSQL DB Resources
 On OTN and in download
– docs.oracle.com/cd/NOSQL/html/index.html
 Getting Started Guides
 Programmatic API
 Installation & Release Notes
 FAQ
Documentation

References
 Why JSON will continue to push XML out of the picture
 Interactive Matter Lab
 Architecting the Internet of Things(p. 102)

Appendix

 Configurable Durability per operation
 Configurable Consistency per
operation
 ACID by default
 Transaction scope is single API call
 Records share same major key
 Multiple operations supported
Greater Flexibility
Features – Configurable ACID Transactions

Developing Applications
Data Modeling
Subject IDQualifier – Value Type Range
Binary
JSON/Avro
RDF Triples
Tables/Rows
Queries
sensors by section
Pressure Sensor Sensor IDLeft Front What a List The Value
Value Types
=
A specific sensor
All measures of a sensor
A range of sensor measures
Major Key
// / / / /
Minor Key
CT 123-PS3234LF – psi, irate Timestamp Array of Int=/ / / / / /

 Increase Data Capacity
– Add more storage nodes
– New shards automatically created
 Increase Data Throughput
– More shards = better write
throughput
– More replicas/shard = better read
throughput
On Demand
Elasticity
NoSQL DB Driver
Application
Master
Replica
Replica
StorageNode StorageNode StorageNode
Shard-1
Master
Replica
Replica
Shard-2
On-Demand Cluster Expansion

 Supports heterogeneous storage
topology
 Replicas move from over-utilized to
under-utilized storage nodes
 Number of shards and replication
factor remain unchanged
Improve Performance
Rebalance an Unbalanced Store
Storage Node 1 Storage Node 2 Storage Node 3
Represents a data fragment

What’s Really Important?
Technical Feature Importance Why
Storage Model Not really Will merge over time
Specific Features Somewhat Application requirements?
Performance Somewhat Rapid changes, YMWV
Integration Critical Long term, Repetitive cost
Reliability/Support Critical
Early products, Product
direction
Predictability Critical Production reqs & SLAs

 Query NoSQL data from
Oracle Database
 Access NoSQL data from
Hadoop for DW and analytics
 Share data with Coherence for
extensible in-memory cache grid
 Persist history & event streams
for processing with OEP
 Store & query RDF data using
Oracle RDF for NoSQL
Integration
Oracle NoSQL Database: Integrated out of the box

Reliability & Support
 Decades of widespread, reliable deployment experience
 15+ years of mission-critical non-relational database technology
 Oracle Support available for both Enterprise and Community Edition
Oracle NoSQL Database: Enterprise-Grade Software & Support

 Automatic election of new Master
 Rejoining nodes automatically
synchronize with the Master
 Isolated nodes can still service reads
 All nodes are symmetric
Automatic Failover
Highly Reliable
Replication factor = 5
Rep Node
Master
Rep Node
Replica
Rep Node
Replica
Rep Node
Replica
Rep Node
Replica
New Master

Con8862 no sql, json and time series data

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Con8862 no sql, json and time series data

Similar to Con8862 no sql, json and time series data (20)

Recently uploaded

Recently uploaded (20)

Con8862 no sql, json and time series data