2. Speaker Introduction
John F. Miller III
Lead Architect, STSM
Keshava Murthy
NoSQL & SQL Architect, STSM
Jef Treece
Product Manager
3. Overview
Business phenomenon of NoSQL
−
Value: Open Source vs Enterprise Class
What is NoSQL, JSON and BSON?
−
Quick overview
−
IBM/Mongo Partnership
New Informix NoSQL/JSON Capabilities
−
Solution Highlights
Mongo Compatibility
−
The Power of a Hybrid Solution
−
Analytics
Demo Deep Dive
−
−
3
Putting together an application
Seeing it live
NewSQL
13. Price Point Comparison Estimate, 3-year cost
Dual Core Intel
Nehalem
Innovator-C
Express
Workgroup
(4 core, 8 GB,
2 nodes)
(16 core, 16 GB,
unlimited nodes)
Product Cost
$0
$8,540
$19,740
$1,680
Included
Included
Support Renewal Year
2
$1,680
$1,708
$3,948
Support Renewal Year
3
$1,680
$1,708
$3,948
Total
$5,040
$11,956
$27,636
Support Subscription
Year 1
24 x 7 x 365
Production System Down
Development Call
Emergency Patches
Free Upgrades
MongoDB Enterprise, 3-year cost: $22,500
Subscription information obtained from 10Gen site, June 26, 2013.
Retail U.S. prices subject to change, valid as of June 26, 2013.
13
21. Some Typical NoSQL Use Cases
- Mostly Interactive Web/Mobile
Online/Mobile Gaming
−
−
−
−
−
Leaderboard (high score table)
management
Dynamic placement of visual elements
Game object management
Persisting game/user state information
Persisting user generated data (e.g.
drawings)
Display Advertising on Web Sites
−
−
Ad Serving: match content with profile and
present
Real-time bidding: match cookie profile with
ad inventory, obtain bids, and present ad
Communications
−
Device provisioning
Social Networking/Online
Communities
21
E-commerce/Social Commerce
– Storing frequently changing product
catalogs
Logging/message passing
– Drop Copy service in Financial
Services (streaming copies of trade
execution messages into (for
example) a risk or back office
system)
Dynamic Content Management
and Publishing (News & Media)
– Store content from distributed
authors, with fast retrieval and
placement
– Manage changing layouts and user
generated content
23. NOSQL, JSON AND BSON
OVERVIEW
Technical Opportunities/ Motivation
What are NoSQL Databases?
Quick overview of JSON
What is sharding?
24. New Era in Application Requirements
Store data from web/mobile application in their
native form
−
New web applications use JSON for storing and
exchanging information
−
Very lightweight – write more efficient applications
−
It is also the preferred data format for mobile
application back-ends
Move from development to production in no
time!
−
Ability to create and deploy flexible JSON schema
−
Gives power to application developers by reducing
dependency on IT
−
Ideal for agile, rapid development and continuous
integration
25. Why NoSQL?
Non-traditional data management requirements driven by Web
2.0 companies
−
Document stores, key-value stores, graph and columnar dbms
The Three Vs:
−
Velocity – high frequency of data arrivals
−
Volume – BigData
−
Variability – unstructured data, flexibility in
schema design
New data interchange formats – like JSON
(JavaScript Object Notation) and BSON
(Binary JSON)
Scale-out requirements across
heterogeneous environment – Cloud
computing
25
26. What is a NoSQL Database?
Not Only SQL or NOt allowing SQL
A non-relational database management systems
−
Does not require a fixed schema
−
Avoids join operations
−
Scales horizontally
−
No ACID (eventually consistent)
Good with distributing data and fast application development
Provides a mechanism for storage and retrieval of
data while providing horizontal scaling.
30. Partnership with IBM and MongoDB
MongoDB and IBM announced a partnership in June 2013
There are many common use cases of interest addressed by the
partnership
−
−
−
−
Accessing JSON Data in DB2, Informix MongoDB using JSON query
Schema-less JSON Data for variety of applications
Making JSON data available for variety of applications
Securing JSON Data
IBM and MongoDB are collaborating in 3 areas:
−
−
−
Open Governance: Standards and Open Source
Technology areas of mutual interest
Products and solutions
31. Basic Translation Terms/Concepts
Mongo/NoSQL Terms
Traditional SQL Terms
Database
Database
Collection
Table
Document
Row
Field
Column
Collection
Table
Name
John
31
Value
Document
28
Scott
Key
21
Tim
{"name":"John","age":21}
{"name":"Tim","age":28}
{"name":"Scott","age":30}
Age
30
Row
32. JSON Details
JSON Syntax Rules
–
–
–
–
–
JSON syntax is a subset of the JavaScript object notation syntax:
Data is in name/value pairs
Data is separated by commas
Curly braces hold objects
Square brackets hold arrays
JSON Name/Value Pairs
– JSON data is written as key/value pairs.
– A key/value pair consists of a field name (in double quotes), followed by a
colon, followed by a value:
"name":"John Miller"
The 6 types of JSON Values:
–
–
–
–
–
–
32
A number (integer or floating point)
A string (in double quotes)
A Boolean (true or false)
An array (in square brackets)
An object (in curly brackets)
Null
33. Example of Supported JSON Types
Example of each JSON
type
Mongo specific JSON
types in blue
– date
{
"string":"John",
"number":123.45,
"boolean":true,
"array":[ "a", "b", "c" ],
"object: { "str":"Miller", "num":711 },
"value": NULL,
"date": ISODate("2013-10-01T00:33:14.000Z")
}
35. The Power of JSON Drives Flexible Schema
JSON key value pair enables a flexible schema
36. Data Access Examples
Relational Representation
LName
FName
Address
Miller
John
123 Blazer St
JSON Representation
JSON_string = ‘{“LName”:”Miller”,”FName”,”John”,”Address”:”123 Blazer St”}’;
Javascript data access
var JSONelems = JSON.parse( JSON_string )
f_name = JSONelems.FNname;
l_name = JSONelems.FName;
l_addr = JSONelems.Address;
37. Simple Code Example
db.posts.insert( ‘{“author”:”John”, “date”,”2013-04-20”,”post”,”mypost
Creates the database “db” if it does not exists
Creates the collection “posts” if the it does not exists
Insert a record into a blog post by user John
db.posts.find ( ‘{ “author”:”John” }’ )
Retrieve all posts by user John
”}’ )
38. Dynamic Elasticity
Rapid horizontal scalability
−
Ability for the application to grow by adding low cost hardware
to the solution
−
Ability to add or delete nodes dynamically
−
Ability rebalance the data dynamically
Application transparent elasticity
39. Why Scale Out Instead of Up?
Scaling Out
−
Adding more servers with less processors and RAM
−
Advantages
Startup costs much less
Can grow instep with the application
Individual servers cost less
−
Several less expensive server rather than fewer high cost servers
−
Redundant servers cost more
Greater chance of isolating catastrophic failures
Scaling Up
−
Adding more processors and RAM to a single server
−
Advantages
Less power consumption than running multiple servers
Less infrastructure (network, licensing,..)
40. Difference between Sharding Data VS Replication
Shard Key
state= “CA”
Shard Key
state= “WA”
Sharding
Each node hold a
portion of the data
• Hash
• Expression
Data is copied to all
nodes
Actions are shipped
to applicable nodes
40
Same data on each
node
Inserted data is
placed on the correct
node
Shard Key
state= “OR”
Replication
Work on local copy
and modification are
propagated
42. Sharding is not for Data Availability
Sharding is for growth, not availability
Redundancy of a node provides high availability for the data
−
Both Mongo and Informix allow for multiple redundant nodes
−
Mongo refers to this as Replica Sets and the additional nodes
slaves
−
Informix refers to this as MACH, and additional nodes secondary
With Informix the secondary server can:
−
Provide high availability
−
Scale out
Execute select
Allow Insert/Update/Deletes on the secondary servers
Share Disks with the master/primary node
43. Basic Data Distribution/Replication Terms
Term
Description
Informix Term
Shard
A single node or a group of nodes holding the same data
(replica set)
Instance
Replica Set
A collection of nodes contain the same data
MACH Cluster
Shard Key
The field that dictates the distribution of the documents.
Must always exist in a document.
Shard Key
Sharded
Cluster
A group shards were each shard contains a portion of the
data.
Grid/Region
Slave
A server which contains a second copy of the data for read
only processing.
Secondary Server
Remote Secondary
43
44. Sharding is not for Data Availibility
Shard Key
state= “CA”
Shard Key
state= “WA”
Shard Key
state= “OR”
44
48. Flexible Schema, Native JSON and BSON
Provide native JSON & BSON support in the Informix Database Server
Two new built in “first-class” data types called JSON and BSON
Support for MongoDB client side APIs throughout the entire database
Enhance horizontal scaling by enabling sharding on all database objects
and models
Adaptive default system initialization
−
−
48
Up and running in seconds
Adapts to computer an environment
49. Two New Data Types JSON and BSON
Native JSON and BSON data types
Index support for NoSQL data types
Native operators and comparator functions
allow for direct manipulation of the BSON
data type
Database Server seamlessly converts to
and from
JSON
BSON
Character data
49
JSON
50. JSON and BSON Data Type Details
Row locks the individual BSON/JSON document
−
MongoDB must lock the database
Bigger documents – 2GB maximum size
−
MongoDB caps at 16MB
Ability to compress documents
−
MongoDB currently not available
Ability to intelligently cache commonly used documents
−
MongoDB currently not available
51. Client Applications
Applications
MongoDB
native Client
New Wire Protocol Listener supports
existing MongoDB drivers
Connect to MongoDB or Informix with same
application!
MongoDB
driver
MongoDB
Wire
Protocol
IBM
JDBC
NoSQL
Driver
Wire
Protocol
Listener
MongoDB
web browser
MongoDB shell
Mobile
Informix
DB
52. MongoDB Application Compatibly
Ability to use any of the MongodB client drivers and frameworks
against the Informix Database Server
−
Little to no change required when running MongoDB programs
−
Informix listens on the same default port as mongo, no need to change.
Leverage the different programming languages available
−
Language examples C, C#, Erlang, JAVA, node.js, Perl, Python, Ruby
Mongo Action
Description
db.customer.insert( { name: “John", age: 21 } )
Insert into database “db” the customer
collection the associated document.
db.customer.find( {age: { $gt:21 } } )
Retrieve from database “db” the customer
collection all documents with the age
greater than 21.
52
53. Fix up JSON Basic Mongo Operations
Conceptual Operations
Mongo Action
db.customer.insert( { name: “John", age:
21 } )
CREATE DATABASE if not exist db
CREATE TABLE if not exist customer
INSERT INTO customer
VALUES ( { “name”:”John”,”age:21”} )
db.customer.find()
SELECT bson_doc FROM customer
db.customer.find( {age: { $gt:21 } } )
SELECT * FROM customer WHERE age > 21
db.customer.drop()
DROP TABLE customer
db.customer.ensureIndex( { name : 1,
age : -1 } )
CREATE INDEX idx_1 on customer(name ,
age DESC)
db.customer.remove( {age: { $gt:21 } } )
DELETE FROM customer where age > 21
db.customer.update( { age: { $gt: 20 } }, {
$set: { status: “Drink" } }, { multi: true } )
53
Traditional SQL Action
UPDATE customer
SET bson_doc_field (status) = { “status”:“Drink"
}
WHERE age > 20
59. Sharding with Hash
Hash based sharding simplifies the
partitioning of data across the shards
Benefits
−
No data layout planning is required
−
Adding additional nodes is online and
dynamic
Cons
−
Adding additional node requires data to be
moved
Data automatically broken in pieces
64. Focus Areas.
Applications need easier way to persist objects.
−
Object-relational layers like hibernate has performance & functional issues.
−
Support Mongo API and its eco system
Flexible Schema: Changing schema in RDBMS is a significant operation,
especially in production.
−
Need exclusive access to table & application downtime
−
Fast changing schema & sparse data has issues with fixed schema.
-- Implement
JSON (BSON) type, data store & indexing
Scale Out: While RDBMS has solutions for cluster and MPP, features
enabling application development and deployment in cloud+MPP
architecture is limited.
−
Use Informix flexible grid & replication to shard tables;
−
Enhance query processing for distributed queries
72. NoSQL Feature
Informix Implementation
1. Flexible Schema
Use BSON and JSON data type. Complete row is stored in a single column; BSON,
JSON are multi-rep types and can store up to 2GB.
2. Accessing KV pairs
within JSON.
Translate the Mongo queries into SQL expressions to access key-value pairs.
Informix has added expressions to extract specific KV pairs.
Select bson_new(data, “{id:1, name:1, addr:1}”) from tab;
3. Indexing
Support standard B-tree index via functional indexing.
create index itid on t(bson_value_int(data, “{id:1}”);
Create index itnamestate on t(bson_value_lvarchar(data, “{name,1}”),
bson_value_lvarchar(data, “{city,1}”);
Informix also supports indexing bsons keys with different data types.
4. Sharding
Supports range & hash based sharding.Informix has built-in technology for
replication. Create identical tables in multiple nodes. Add meta data for partitioning
the data across based on range or hash expressions.
5. SELECT
Limited support now. Mongo API helps by disallowing joins. The translated query on
a single table is transformed into federated UNION ALL query; includes shard
elimination.
73. NoSQL Feature
Informix Implementation
6. Updates (single node)
INSERT: Simple direct insert.
DELETE: DELETE statement with WHERE bson_extract() > ?; or bson_value..() > ?
UPDATE: bson_update(bson, “update expr”) will return a new bson after applying the
bson expression. Simple updates to non_sharded tables will be direct UPDATE
statement in a single transaction. UPDATE tab bsoncol = bson_update(bsoncol,
“expr”) where
7. Updates (sharded env)
INSERT – All rows are inserted to LOCAL shard, replication threads will read logs
and replicate to the right shard and delete from local node (log only inserts
underway).
DELETE – Do the local delete and replicate the “delete statements” to target node in
the background
UPDATE – Slow update via select-delete-insert.
8. Transaction
Each statement is a distinct transaction in a single node environment.
The data and the operation is replicated via enterprise replication.
9. Isolation levels.
NoSQL session can use any of Informix isolation levels.
-- Examples from Mongo applications.
10. Locking
Application control only on the node they’re connected to. Standard
Informix locking semantics will apply when data is replicated and applied
on the target shard.
-- Verify.. What’s the default locking level?
-- Any option to change it? Just ONCONFIG??
74. Feature
Informix Implementation
11. Hybrid access
Mongo APIs can simply access Informix tables, views, virtual tables as if they’re
JSON collections. db.foo.find(); foo can be JSON collection, relational table, a
view or a virtual table on top of timeseries or Websphere MQ.
From MongoAPI to
relational tables.
12. Hybrid access
1.
Directly get binary BSON or cast to JSON to get in textual form.
From SQL to JSON data
2.
Use expressions to extract to extract specific key-value pairs.
3.
To be done: NoSQL collections will only have one BSON object in the table.
We can “imply” the expressions when the SQL refers to a column.
SELECT t.c1, t.c2 from t;
SELECT bson_extract(t.data, “{c1:1}”), bson_extract(t.data, “{c2:1}”) from t;
So, that’s twelve steps for NoSQL Anonymous!
75. 1. Flexible Schema
Clients exchange BSON document with the server both for queries and data.
Thus, BSON becomes a fundamental data type.
The explicit key-value pairs withing the JSON/BSON document will be roughly
equivalent to columns in relational tables.
However, there are differences!
−
The type of the KV data encoded within BSON is determined by the client
−
Server is unaware of data type of each KV pair at table definition time.
−
No guarantees that data type for each key will remain consistent in the
collection.
−
The keys in the BSON document can be arbitrary;
−
While customers exploit flexible schema, they’re unlikely to create a single
collection and dump everything under the sun into that collection.
−
Due to the limitations of Mongo/NoSQL API, customers typically denormalize
the tables (customer will have customer+customer addr + customer
demographics/etc) to avoid joins.
76. 1. Flexible Schema – Informix Implementation
• Informix has a new data type BSON to store the data.
• Informix also has a JSON data type to convert between binary and
text form.
• BSON and JSON are abstract data types (like spatial, etc).
• BSON and JSON multi-representational types.
•Objects up to 4K is stored in data pages.
•Larger objects (up to 2GB) are stored out of row, in BLOBs.
•MongoDB limits objects to 16MB.
•This is all seamless and transparent to applications.
77. 1. Flexible Schema – Informix Implementation
CREATE TABLE customer (data BSON);
• BSON is the binary represenation of JSON.
•It has length and types of the key-value pairs in JSON.
• MongoDB drivers send and receive in BSON form.
78. 2. Accessing KV pairs within JSON.
• We’ll have number of extract expressions/functions
•Expressions returning base type
bson_value_bigint(BSON, “key”);
bson_value_lvarchar(bsoncol, “key.key2”);
bson_value_date(bsoncol, “key.key2.key3”);
bson_value_timestamp(bsoncol, “key”)
bson_value_double(bsoncol, “key”);
bson_value_boolean(bsoncol, “key”);
bson_value_array(bsoncol, “key”);
bson_keys_exist(bsoncol, “key”);
Bson_value_document(bsoncol, “key”)
Bson_value_binary(bsonol, “key”)
Bson_value_objectid(bsoncol, “key”)
•Expression returning BSON subset. Used for bson indices.
bson_extract(bsoncol, “projection specification”)
•Expressions to project out of SELECT statement.
bson_new(bsoncol, “{key1:1, key2:1, key3:1}”);
bson_new(bsoncol, “{key5:0}”);
79. 2. Accessing KV pairs within JSON.
Mongo Query
SQL Query
db.customers.find();
SELECT SKIP ? Data::bson
db.customers.find({},{num:1,name:1})
;
SELECT SKIP ? bson_new( data, '{ "num" : 1.0 , "name"
: 1.0}')::bson FROM
customers
db.customers.find({},
{_id:0,num:1,name:1});
SELECT SKIP ? bson_new( data, '{_id:0.0, "num" : 1.0
, "name" : 1.0}')::bson FROM
customers
db.customers.find({status:”A”})
SELECT SKIP ? data FROM customers WHERE
bson_extract(data, ‘status') = “A”
db.customers.find({status:”A”},
{_id:0,num:1,name:1});
FROM
customers
SELECT SKIP ?
bson_new( data, '{ "_id" : 0.0 , "num" : 1.0 , "name"
: 1.0}')::bson
FROM customers WHERE bson_extract(data, 'name') = “A”
80. 3. Indexing
•
•
•
•
•
Supports B-Tree indexes on any key-value pairs.
Indices could be on simple basic type (int, decimal) or BSON
Indices could be created on BSON and use BSON type comparison
Listener translates ensureIndex() to CREATE INDEX
Listener translates dropIndex() to DROP INDEX
Mongo Query
SQL Query
db.customers.ensureIndex({orderD
ate:1})
CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x')
ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" :
"newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1"
, "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64
db.customers.ensureIndex({orderD
ate:1, zip:-1})
CREATE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC,
bson_extract(data,'c2') DESC) using bson (ns='{ "name" :
"newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0
, "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" : "c1_1_c2__1" ,
"index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64
db.customers.ensureIndex({orderD
ate:1}, {unique:true)
CREATE UNIQUE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC,
bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}',
idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 ,
"$extract"]} , "name" :"c1_1_c2__1" , "unique" : true , "index" : "v_c1_1_c2__1"}')
EXTENT SIZE
64 NEXT SIZE 64
81. 3. Indexing
db.w.find({x:1,z:44},{x:1,y:1,z:1})
Translate to:
SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1}
FROM w
WHERE ( bson_extract(data, 'x') = '{ "x" : 1.0 }'::json::bson ) AND
( bson_extract(data, 'z') = '{ "z" : 44.0 }'::json::bson)
Estimated Cost: 2
Estimated # of Rows Returned: 1
1) keshav.w: SEQUENTIAL SCAN
Filters: (informix.equal(informix.bson_extract(keshav.w.data ,'z' ),UDT )
AND informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT))
82. 3. Indexing
•Functional Index is built on bson expressions
CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC)
using bson (ns='{ "name" : "newdb.w.$x_1"}',
idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name"
: "x_1" , "index" : "w_x_1"}')
EXTENT SIZE 64 NEXT SIZE 64
•Listener is aware of the available index and therefore
generates right predicates.
db.w.find({x:1});
gets translated to
SELECT SKIP ? data FROM w WHERE bson_extract(data, 'x') = ?
83. 3. Indexing
db.w.find({x:5,z:5}, {x:1,y:1,z:1})
Translates to:
SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1 .0}')::bson
FROM w
WHERE ( bson_extract(data, 'x') = '{ "x" : 5.0 }'::json::bson )
AND ( bson_extract(data, 'z') = '{ "z" : 5.0 }'::json::bson )
Estimated Cost: 11
Estimated # of Rows Returned: 1
1) keshav.w: INDEX PATH
Filters: informix.equal(informix.bson_extract(keshav.w.data ,'z' ), UDT )
(1) Index Name: keshav.w_x_1
Index Keys: informix.bson_extract(data,'x') (Serial, fragments: ALL)
Lower Index Filter: informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT )
86. 4. Mongodb SHARDING (roughly)
Shard a single table by range or hashing.
Mongos will direct the INSERT to target shard.
Mongos tries to eliminate shards for update, delete, selects as well.
FIND (SELECT) can happen ONLY a SINGLE table.
It also works as coordinator for multi-node ops.
Once a row is inserted to a shard, it remains there despite any key
update.
No transactional support on multi-node updates.
−
Each document update is unto its own.
89. 4. Sharding – Informix Implementation
•Shard a single table by range or hashing.
cdr define shard myshard mydb:usr1.mytab
–type=delete –key=”bson_get(bsoncol, ‘STATE’)” –stragety=expression
versionCol=version
servA “in (‘TX’, ‘OK’)”
servB “in (‘NY’,’NJ’) “
servC “in (‘AL’,’KS’) “
servD remainder
cdr define shard myshard mydb:usr1.mytab
–type=delete –key=state –stragety=hash --versionCol=version
servA servB servC servD
•Shard a single table by range or hashing.
<<Mongo syntax>>
90. 4. Sharding – Informix Implementation
•Shard a single table by range or hashing.
•Sharding is transparent to application.
•Each CRUD statement will only touch a single table.
•Limitation of MongoDB…Makes it easier for Informix.
•Lack of joins is a big limitation for SQL applications.
•Lacks transactional support for distributed update.
91. 4. Informix Sharding
Identical table is created on each node and meta data is replicated
on each node.
Schema based replication is the foundation for our sharding.
CRUD operations can go to any node.
−
We’ll use replication and other techniques to reflect the data in the
target node, eventually.
−
Right now, replication is asynchronous.
−
Informix has synchronous replication
not used for sharding now.
92. 5. SELECT on sharded tables
•The query can be submitted to any of the nodes via the listener.
•That node acts as the “coordinator” for the distributed query.
•It also does the node elimination based on the query predicate.
•After that, the query is transformed to UNION ALL query
SELECT SKIP ? bson_new( data, '{"_id":0.0 ,"num":1.0 ,"name" : 1.0}')::bson
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X”
is transformed into:
SELECT SKIP ? bson_new( data, '{"_id":0.0
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or
UNION ALL
SELECT SKIP ? bson_new( data, '{"_id":0.0
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or
,"num":1.0 ,"name" : 1.0}')::bson
bson_extract(data, 'name') = “X”
,"num":1.0 ,"name" : 1.0}')::bson
bson_extract(data, 'name') = “X”
93. 6. INSERT: Single node
•If necessary, automatically create database &
table (collection) on the application INSERT
•Collections: CREATE TABLE t(a GUID, d
BSON);
•GUID column is needed to ensure unique row across
the SHARDs. Also used as PK for replication.
•Client application inserts JSON, client API converts
this into BSON, generates _id (object id) if necessary &
sends to server over JDBC/ODBC.
•Server saves the data into this table as BSON, with
an automatically generated GUID
94. 6. DELETE: Single node
•Mongo remove are translated to SQL DELETE
•Will always remove all the qualifying rows.
•WHERE clause translation is same as SELECT
95. 6. UPDATE: Single node
•Simple set, increment, decrement updates are
handled directly by the server.
Mongo: db.w.update({x: {$gt:55}}, {$set:{z:9595}});
SQL: UPDATE w SET data = bson_update(data, “{$set:{z:9595}}”)
WHERE bson_extract(data, 'x') > “{x:55}”::bson ?
•bson_update is a built-in expression updating a
BSON document with a given set of operations.
•Complex updates are handled via select batch,
delete, insert.
•Always updates all the rows or no rows, under a
transaction.
96. 7. INSERT: shard implementation
•If the table is sharded:
•Insert the data into local table as usual.
•Replication threads in the background will evaluate
the log record to see which rows will have moved.
•Replication thread will move the necessary row to
target and delete from the local table.
•Work underway to avoid the insert into local table.
•For each inserted row ending up in non-local shard,
simply generate the logical log and avoid insert into
local table & indices.
97. 7. DELETE : shard implementation
•Application delete could delete rows from any of the
shards in the table.
•DELETE will come in as shard_delete() procedure.
–Execute procedure shard_delete(tabname, delete_stmt);
–This procedure will issue delete locally.
–It will then INSERT the delete statement into a “special”
shard delete table (single for all tables).
–Enterprise replication will propagate the delete to
applicable target systems.
98. 7. UPDATE: shard implementation
•When the update have to be done on multiple
nodes:
•Client application does UPDATE and CLIENT API
converts that into three operations: SELECT,
DELETE & INSERT.
•GUID column is needed to ensure unique row across the
SHARD
•Client application inserts JSON, client API converts this
into BSON & sends to server.
99. 8. Transactions (single node)
•Mongo does not have the notion of transactions.
•Each document update is atomic, but not the app statement
•For the first release of Informix-NoSQL
•By default, JDBC listener simply uses AUTO COMMIT option
•Each server operation INSERT, UPDATE, DELETE, SELECT will be
automatically be committed after each operation.
•No locks are held across multiple operation.
•However, customers can create multi-statement transaction
via $sql and issuing begin work, commit work, rollback work
100. 8. Transactions (sharded environment)
•In sharded environment, mongo runs database via two different
instances: mongos and mongod.
•Mongos simply redirects operations to relevant mongod.
•No statement level transactoinal support.
•Informix
•Informix does not have the 2-layer architecture
•Informix server the application connected to becomes the
transaction coordinator
•Informix does have the 2-phase commit transaction support,
but is unused for NoSQl, for now.
•SELECT statement goes thru distributed query infrastructure
•INSERT,UPDATE DELETE goes thru enterprise replication.
101. 9. ISOLATION levels
•Default isolation level is DIRTY READ.
•Change this directly or sysdbopen()
•You can also use USELASTCOMMITTED variable in
ONCONFIG.
•If you’re using procedures for executing multistatement transaction, you can set it within your
procedure.
102. 10. LOCKING
•Page level locking is the default.
•You can change it to ROW level locks easily.
•ALTER TABLE jc MODIFY lock mode (row)
• DEF_TABLE_LOCKMODE onconfig variable.
• SET LOCK MODE can be set via sysdbopen()
•Each statement is executed with auto-commit and locking
semantics will apply there.
104. Hybrid Access between relational & JSON Collections
Relational Table
SQL API
Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.
MongoDB API
(NoSQL)
?
JSON Collections
?
Mongo APIs for Java,
Javascript, C++, C#,...
105. Benefits of Hybrid Power
Access consistent data from its source
Avoid ETL, continuous data sync and conflicts.
Exploit the power of SQL, MongoAPI seamlessly
Exploit the power of RDBMS technologies in MongoAPI:
−
Informix Warehouse accelerator,
−
Cost based Optimizer
−
R-tree indices for spatial, Lucene text indexes, and more.
Access all your data thru any interface: MongoAPI or SQL.
Store data in one place and efficiently transform and use them on
demand.
Existing SQL based tools and APIs can access new data in JSON
106. Why do you need hybrid access?
Data model
should not restrict
Data Access
107. Hybrid Access between relational & JSON Collections
Relational Table
SQL API
MongoDB API
(NoSQL)
Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.
Mongo APIs for Java,
Javascript, C++, C#,...
JSON Collections
Direct SQL Access.
Dynamic Views
Row types
Mongo APIs for Java,
Javascript, C++, C#,...
109. Hybrid access: From MongoAPI to relational tables.
You want to develop an application with MongoAPI, but
1. You already have relational tables with data.
2. You have views on relational data
3. You need to join tables
4. You need queries with complex expressions. E.g. OLAP window functions.
5. You need to get results from a stored procedure.
6. You need to exploit Informix stored procedure
7. You need federated access to other data
8. You have timeseries data.
110. How to treat relational data as JSON store.
Relational data (relations or resultset) can treated as structured JSON
documents; column name-value becomes key-value pair.
SELECT partner, pnum, country from partners;
partner
pnum Country
Pronto
1748 Australia
Kazer
1746 USA
Diester
1472 Spain
Consultix
1742 France
{parnter:
{parnter:
{parnter:
{parnter:
“Pronto”, pnum:”1748”, Country: “Australia”}
“Kazer”, pnum:”1746”, Country: “USA”}
“Diester”, pnum:”1472”, Country: “Spain”}
“Consultix”, pnum:”1742”, Country: “France”}
Listner translates the query and the data object between relational and
JSON/BSON form.
111. Hybrid access: From MongoAPI to relational tables.
Mongo Application
JSON
JSON
db.customer.find({state:”MO”})
db.partners.find({state:”CA”})
IBM Wire Listener
JDBC connections
Access JSON
SELECT bson_new(bson, ‘{}’) FROM customer
WHERE bson_value_lvarchar(bson,‘state’)= “MO”
Access Relational
SELECT * FROM partners WHERE
state = “CA”
Informix Dynamic Server
JSON Collections
Customer
Distributed
Queries
Tables
IDXs
Tables
Relational Tables
Logs
partners
Tables
IDXs
Enterprise replication + Flexible Grid
112. Accessing data in relational tables.
Create table partners(pnum int, name varchar(32), country
varchar(32));
db.partners.find();
SELECT * from partners;
db.partners.find({name:”Pronto”});
SELECT * FROM PARTNERS WHERE name = “Pronto”;
db.partners.find({name:”Pronto”}, {pnum:1, country:1});
SELECT a, b FROM t WHERE a = 2.0;
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).limit(10);
SELECT LIMIT 10 pnum, country FROM WHERE name = “Pronto”;
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:1})
SELECT pnum,country FROM partners WHERE name = “Pronto” ORDER BY b ASC
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:-1})
SELECT a, b FROM t WHERE a = 2.0 ORDER BY b DESC
db.t.find({a:{$gt:1}}, {a:1, b:1}).sort({b:-1})
SELECT SKIP ? a, b FROM t WHERE query > 1.0 ORDER BY b DESC
113. Accessing data in relational tables.
db.partners.save({pnum:1632, name:”EuroTop”, Country: “Belgium”});
INSERT into partners(pnum, name, country values(1632, ”EuroTop”,
“Belgium”);
db.partners.delete({name:”Artics”});
DELETE FROM PARTNERS WHERE name = “Artics”;
Db.partners.update({country:”Holland”},{$set:{country:”Netherland”}}
, {multi: true});
UPDATE partners SET country = “Netherland” WHERE country =
“Holland”;
db.partners.drop();
DROP TABLE partners;
114. Views and JOINS
A dynamically created relation created from one or more database
tables or procedures.
create table pcontact(pnum int, name varchar(32), phone varchar(32));
insert into pcontact values(1748, "Joe Smith", "61-123-4821");
insert into pcontact values(1746, "John Kelley", "1-729-284-2893");
insert into pcontact values(1472, "Ken Garcia", "34-829-2842");
insert into pcontact values(1742, "Adam Roy", "33-380-3892");
create view partnerphone(pname, pcontact, pphone) as select a.name, b.name, b.phone
FROM pcontact b left outer join partners a on (a.pnum = b.pnum);
db.partnerphone.find();
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" }
{ "pname" : "Kazer", "pcontact" : "John Kelley", "pphone" : "1-729-284-2893" }
{ "pname" : "Diester", "pcontact" : "Ken Garcia", "pphone" : "34-98-829-2842" }
{ "pname" : "Consultix", "pcontact" : "Adam Roy", "pphone" : "33-82-380-3892" }
db.partnerphone.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" }
115. complex expressions. E.g. OLAP window functions
create view contactreport(pname, pcontact, totalcontacts) as
select b.name, a.name,
count(a.name) over(partition by b.pnum)
from pcontact a left outer join partners b on (a.pnum = b.pnum);
db.contactreport.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 }
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 }
116. Seamless federated access
1.
2.
create database newdb2;
create synonym oldcontactreport for
newdb:contactreport;
> use newdb2
> db.oldcontactreport.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 }
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 }
SELECT data FROM oldcontactreport WHERE
bson_extract(data, 'pname') = “Pronto”;
•
create synonym oldcontactreport for
custdb@nydb:contactreport;
117. Get results from a stored procedure.
create function "keshav".p6() returns int, varchar(32);
define x int; define y varchar(32);
foreach cursor for select tabid, tabname into x,y from systables
return x,y with resume;
end foreach;
end procedure;
create view "keshav".v6 (c1,c2) as
select x0.c1 ,x0.c2 from table(function p6())x0(c1,c2);
db.v6.find().limit(5)
{
{
{
{
{
"c1"
"c1"
"c1"
"c1"
"c1"
:
:
:
:
:
1,
2,
3,
4,
5,
"c2"
"c2"
"c2"
"c2"
"c2"
:
:
:
:
:
"systables" }
"syscolumns" }
"sysindices" }
"systabauth" }
"syscolauth" }
118. Access Timeseries data
create table daily_stocks
( stock_id integer,
stock_name lvarchar,
stock_data timeseries(stock_bar)
);
-- Create virtual relational table on top (view)
EXECUTE PROCEDURE
TSCreateVirtualTab('daily_stocks_virt',
'daily_stocks', 'calendar(daycal),origin(2011-01-03
00:00:00.00000)' );
create table daily_stocks_virt
( stock_id integer,
stock_name lvarchar,
timestamp datetime year to fraction(5),
high smallfloat,
121. Hybrid access: From MongoAPI to relational tables.
You want to develop an application with MongoAPI, but
1. You already have relational tables with data.
2. You have views on relational data
3. You need to join tables
4. You need queries with complex expressions. E.g. OLAP window functions.
5. You need to get results from a stored procedure.
6. You need to exploit Informix stored procedure
7. You need federated access to other data
8. You have timeseries data.
Put all of these together!
125. Join JSON collections
-- Wrap aorund the built-in expressions
SELECT bson_value_int(jc1.data, 'x'),
bson_value_lvarchar(jc1.data, 'y'),
bson_value_int(jc1.data, 'z') ,
bson_value_int(jc2.data, 'c1'),
bson_value_lvarchar(jc2.data, 'c2')
FROM w jc1, v jc2
WHERE bson_value_int(jc1.data, 'x') =
bson_value_int(jc2.data, 'c1');
126. Join JSON collections
-- Create a view to make the access simple.
create view vwjc(jc1x, jc1y, jc1z, jc2c1, jc2c2) as
SELECT bson_value_int(jc1.data, 'x'),
bson_value_lvarchar(jc1.data, 'y'),
bson_value_int(jc1.data, 'z') ,
bson_value_int(jc2.data, 'c1'),
bson_value_lvarchar(jc2.data, 'c2')
FROM w jc1, v jc2
WHERE bson_value_int(jc1.data, 'x') =
bson_value_int(jc2.data, 'c1');
db.vwjc.find();
Db.vwjc.find({jc1x:100});
Db.vwjc.find({jc1x:{$gte:10}}).sort(jc1y:1)
127. You want to perform complex analytics on JSON data
BI Tools like Cognos, Tableau generate SQL on data sources.
Option 1: Do ETL
Need to expose JSON data as views so it’s seen as a database
object.
−
We use implicit casting to convert to compatible types
−
The references to non-existent key-value pair returns NULL
Create any combination of views
−
A view per JSON collection
−
Multiple views per JSON collection
−
Views joining JSON collections, relational tables and views.
Use these database objects to create reports, graphs, etc.
129. Hybrid Access between relational & JSON Collections
Relational Table
JSON Collections
SQL API
Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.
Direct SQL Access.
Dynamic Views
Row types
MongoDB API
Mongo APIs for Java,
(NoSQL)
Javascript, C++, C#,...
Mongo APIs for Java,
Javascript, C++, C#,...
130. Ability for MongoDB API to Access All Data Models
node.js Application
Traditional SQL
MongoDB Drivers
NoSQL - JSON
NoSQL - JSON
TimeSeries
TimeSeries
MQ Series
MQ Series
MQ Series
130
132. Benefits of Hybrid Power
Access consistent data from its source
Avoid ETL, continuous data sync and conflicts.
Exploit the power of SQL, MongoAPI seamlessly
Exploit the power of RDBMS technologies via MongoAPI:
− Informix Warehouse accelerator,
− Cost based Optimizer & power of SQL
− R-tree indices for spatial, Lucene text indexes, and more.
Access all your data thru any interface: MongoAPI & SQL
Store data in one place and efficiently transform and use
them on demand.
Existing SQL based tools and APIs can access new data
in JSON
141. Not just a Hybrid Database
Hybrid databases only solve half of the problem
Hybrid application are hear to stay
142. Scaling Out – Sharding Data
Shard Key
state= “CA”
Shard by either hash or
expression
When inserting data it is
automatically moved to the
correct node
•
142
Each node in the environment
hold a portion of the data
•
Shard Key
state= “OR”
•
•
Shard Key
state= “WA”
Queries automatically
aggregate data from the
required node(s)
143. Graphical Administration showing a Sharded System
Graphical administration of
the shards keep things
simple
Monitor the entire
environment from a single
graphical console
145. Simplify the “up and running” Experience
Install and setup a typical database instance with only 3 questions:
−
Where to place the Product?
−
Where to place the Data?
−
How many users do you anticipate?
Newly installed instance adapts to the resources on the
computer
Description
Auto tuning of CPU VPS
Auto Table Placement
Auto Buffer pool tuning
Auto Physical Log extension
Auto Logical Log Add
Auto Read Ahead
145
147. Relational and non-relational data in one system
NoSQL/MongoDB Apps can access Informix Relational Tables
Distributed Queries
Multi-statement Transactions
Enterprise Proven Reliability
Enterprise Ready Security
Enterprise Level Performance
Informix provides the capability to leverage
the abilities of both relational DBMS and document store systems.
MongoDB does not. It is a document store system lacking key
abilities like transaction durability.
147
148. Round Peg Square Hole
The DBA and/or programmer no longer have to decide upfront if it
is better to use a system entirely of JSON documents or a system
only of SQL tables, but rather have a single database in which the
programmer decides a if JSON document is optimal or a SQL table
is optimal for the data and usage.
148
149. Informix Specific Advantages with Mongo Drivers
Traditional SQL tables and JSON collections co-existing in the
same database
Using the MongoDB client drivers Query, insert, update, delete
−
JSON collections
−
Traditional SQL tables
−
Timeseries data
Join SQL tables to JSON collections utilizing indexes
Execute business logic in stored procedures
Provide a view of JSON collections as a SQL table
−
Allows existing SQL tools to access JSON data
Enterprises level functionality
149
151. Some NoSQL Use Cases - Mostly Interactive Web/Mobile
Online/Mobile Gaming
−
−
−
−
−
Leaderboard (high score table)
management
Dynamic placement of visual elements
Game object management
Persisting game/user state information
Persisting user generated data (e.g.
drawings)
Display Advertising on Web Sites
−
−
Ad Serving: match content with profile
and present
Real-time bidding: match cookie profile
with ad inventory, obtain bids, and
present ad
Dynamic Content Management
and Publishing (News & Media)
−
−
151
Store content from distributed authors,
with fast retrieval and placement
Manage changing layouts and user
generated content
E-commerce/Social Commerce
– Storing frequently changing product
catalogs
Social Networking/Online
Communities
Communications
– Device provisioning
Logging/message passing
– Drop Copy service in Financial
Services (streaming copies of trade
execution messages into (for
example) a risk or back office
system)
154. Ability for All Clients to Access All Data Models
Informix SQLI
Drivers
IBM DRDA
Drivers
MongoDB Drivers
Traditional SQL
NoSQL - JSON
TimeSeries
MQ Series
154
155. Ability for All Clients to Access All Data Models
node.js Application
Traditional SQL
MongoDB Drivers
NoSQL - JSON
TimeSeries
MQ Series
155
158. IOD Demo and Beyond
A fun exciting demo application that attendees
can interact with that makes people take notice
of IBM technologies
A social networking photo application utilizing
smart phones/tablets that allows the end user to
take photos, tag and add photos to an Informix
NoSQL hybrid database
Random photos can be displayed at sessions,
booths or on attendee smart devices
159. IOD Attendee Photo Application
Allow conference attendee to take and share photo!
160. Technology Highlights
•
Create a hybrid application using NoSQL, traditional SQL,
timeseries mobile web application
•
•
•
Utilizing both JSON collections, SQL tables and timeseries
Utilize IBM Dojo Mobile tools to build a mobile application
Leverage new mongo client side drivers for fast application
delvelopment
•
Demonstrate sharding with over 100 nodes
•
Cloud based solution on top of Amazon Cloud
•
Provide real-time analytics on all forms of data
•
•
Leverage existing popular analytic front-end IBM-Congos
Utilize an in-memory columnar database accelerator to
provide real-time trending analytics on data
161. Mobile Device Application Architecture
Apache Web Server
IOD Photo App - UPLOAD
Photo Application
IBM Dojo Mobile
Informix
tag
Van Gogh
Informix JSON
Listener
Photo
collection
User
Table
162. Application Architecture (Big Picture)
Apache web server
IOD application
Informix JSON Listener
Informix
Photo collection
Amazon Cloud
Sharding
commands
Node1
Aa-Am
Node2
An-Az
Node3
Ba-Bf
Node3
Bg-Bp
Node101
“Dalmation” shard farm – 101 sharded nodes on Amazon Cloud
Photo collection sharded across these nodes
165. Application Considerations
Photo meta-data varies from camera to camera
A Picture and all its meta data are stored in-document
Pictures are stored in a NoSQL collection
Pre-processing on the phone ensures only reasonable size
photos are sent over the network.
166. Example of Live JSON Photo Data
{"_id":ObjectId("526157c8112c2fe70cc06a75"), "Make":"NIKON CORPORA
TION","Model":"NIKON D60","Orientation":"1","XResolution":"300","
YResolution":"300","ResolutionUnit":"2","Software":"Ver.1.00 ","D
ateTime":"2013:05:15 19:46:36","YCbCrPositioning":"2","ExifIFDPoi
nter":"216","ExposureTime":"0.005","FNumber":"7.1","ExposureProgr
am":"Not defined","ISOSpeedRatings":"100",
"Contrast":"Normal","Saturation":"Normal","Sharpness":"Normal",
"SubjectDistanceRange":"Unknown","name":"DSC_0078.JPG","img_d
ata":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABcQ
167. Sharding
NoSQL “Dalmatian” cloud deployment
Sharding by hash
Heterogeneous Cloud deployment with sharding
−
Processor Type and Number
Intel and ARM processors
−
Operating System Types and Versions
Linux, Windows, AIX, Apple, Solaris, HP,
168. Basic PHP Programming Overview Information
List of NoSQL collection names and SQL tables names
Function to set the active database and return the Collection
private $conn;
private
private
private
private
private
private
private
$dbname = "photo_demo";
$photoCollectionName = "photos";
$contactsCollectionName = "contacts";
$sqlCollectionName = 'system.sql';
$userTableName = "users";
$tagsTableName = "tags";
$likesTableName = "likes";
private $photoQueryProjection = array("_id" => 1, "tags" => 1,
"user_id" => 1, "img_data" => 1);
/**
* Get collection by name
* @param MongoCollection $collectionName
*/
private function getCollection($collectionName) {
return $this->conn->selectDB($this->dbname)->selectCollection ($collectionName);
}
170. Insert Data into a Collection
Very simple to insert JSON data into a collection using the
MongoAPIs
/**
* Insert user's contact information into contacts table.
*/
public function insertContact( $json ) {
if (! is_array ( $json )) {
return "Contact info not in JSON format.";
}
try {
$result = $this->getCollection($this->contactsTableName)->insert($json);
if ($result ["ok"] != 1) {
return $result ["err"];
}
} catch ( MongoException $e ) {
return $e->getMessage ();
}
return "ok";
}
171. Retrieve Collection Information
Very simple to retrieve data from a collection using the
MongoAPIs
Data is returned as a JSON document
/**
* Get contact info
*/
public function adminContacts() {
$photoCollection = $this->getCollection($this->contactsCollectionName);
$cursor = $photoCollection->find();
$results = $this->getQueryResults($cursor);
return $results;
}
172. Retrieve Collection Information
/**
* Get contact info for admin panel
*/
public function adminContacts() {
$photoCollection = $this->getCollection($this->contactsCollectionName);
$cursor = $photoCollection->find();
$results = $this->getQueryResults($cursor);
return $results;
}
174. Delete a Photo and its Information
Deleting from SQL Tables and NoSQL Collection is exactly
the same
/**
* Delete photo
*/
public function deletePhoto($id) {
try {
// First delete from likes and tags tables
$query = array('photo_id' => $id['_id']);
$result = $this->getCollection($this->likesTableName)->remove($query);
if ($result ["ok"] != 1) {
return $result["err"];
}
$result = $this->getCollection($this->tagsTableName)->remove($query);
if ($result ["ok"] != 1) {
return $result["err"];
}
// Then delete the photo from the collection
$query = array('_id' => new MongoId($id['_id']));
$result = $this->getCollection ( $this->photoCollectionName )->remove ( $query );
if ($result ["ok"] != 1) {
return $result["err"];
}
} catch ( MongoException $e ) {
return $e->getMessage();
}
return "ok";
}
175. Executing a Stored Procedure in MongoAPI
/**
* Get the user_id for a particular user name (email address).
*
* Calls a stored procedure that will insert into the users table if the
* user does not exist yet and returns the user_id.
*
* @param string $username
* @return int $user_id
*/
public function getUserId($username) {
$username = trim($username);
try {
$sql = "EXECUTE FUNCTION getUserID('" . $username . "')";
$result = $this->getCollection($this->sqlCollectionName)->
findOne(array('$sql'=>$sql));
if (isset($result['errmsg'])) {
return "ERROR. " . $result['errmsg'];
}
return $result['user_id'];
} catch (MongoException $e) {
return "ERROR. " . $e->getMessage();
}
}
176. Real Time Analytics
Customer Issues
−
Several different models of data (SQL, NoSQL,
TimeSeries/Sensor)
−
NoSQL is not strong building relations between collections
−
Most valuable analytics combine the results of all data models
−
Most protonate Analytic system written using standard SQL
−
ETL & YAS (Yet Another System)
Solution
Provide a mapping of the required data in SQL form
−
Enables common tools like Congo's
177. Analytics on a Hybrid Database
Photo
collection
SQL
MongoAPI
Informix
User
Table
180. IWA Integration
What is trending at IOD?
Set up to work on frequently updated snapshots of data to
perform analytic queries on the photo meta-data
Since BSON/JSON data can not be directly compressed
and searched, snapshot process would selectively extract
data from the documents for copying to IWA
Use Genero to show the power of IWA
182. Configure Informix on Amazon Cloud Simple
•
•
•
•
•
Instantiate the Amazon image
Setup the storage
Install the product
Start the system
Configure sharding