SlideShare une entreprise Scribd logo
1  sur  185
Télécharger pour lire hors ligne
Informix NoSQL
-- Very Deep Dive
John F. Miller III, IBM miller3@us.ibm.com
Keshava Murthy, IBM rkeshav@us.ibm.com

© 2013 IBM Corporation
Speaker Introduction
John F. Miller III
Lead Architect, STSM

Keshava Murthy
NoSQL & SQL Architect, STSM

Jef Treece
Product Manager
Overview
Business phenomenon of NoSQL
−

Value: Open Source vs Enterprise Class

What is NoSQL, JSON and BSON?
−

Quick overview

−

IBM/Mongo Partnership

New Informix NoSQL/JSON Capabilities
−

Solution Highlights

Mongo Compatibility
−

The Power of a Hybrid Solution

−

Analytics

Demo Deep Dive
−
−
3

Putting together an application
Seeing it live

NewSQL
BUSINESS PHENOMENON OF NOSQL

© 2013 2013Corporation
© IBM IBM Corporation
Informix NoSQL Capabilities
Key-value Stores
– Hashed Key value model
– Capable of massive data storage

Column Family Stores
– Keys that point to multiple columns and column families

Document Data Base
– JSON / BSON Formatted documents
– Collections of Documents with Key-value stored versions

Graph Databases
–
–
–
–

Nodes / relationship between nodes
Properties of nodes
Data values
Supports SPARQL ?

© 2013 IBM Corporation
Business Value of Informix NoSQL
Increases business opportunity through capture and analysis of
“interactive / interaction data” for a full understanding of customers
and markets
Lowers business barriers and reduces IT costs through Rapid
Application Deployment
Increases business ability to respond to volume changes through
Scalable / extensible architecture
Lowers IT overhead through use of Low Cost Commodity Hardware
Changes business view to “what is happening” versus “what
happened”

© 2013 IBM Corporation
Use Cases for Informix NoSQL
Session Store
– High volume ingestion / low latency access requirements
– Linkage to a specific user provides immediate recognition of customer
preferences
– Session restore after “break” is immediate to last functions performed increase
customer confidence and connection to their desire(s)

User Profile Store
– Customer Profiles, orders and shipment status is immediate and searchable
– Fast access for Authentication and preferences
– Historical access and click streams increase personalization & targeting

Content and Metadata Store
– Document database for digital content, articles, instruction manuals
– Key-value linkages provide linkage to customer profiles and history
– Multiple image/data type support provide fast access to different data type

© 2013 IBM Corporation
Use Case for Informix NoSQL (cont)
Mobile Apps
– Ability to store content and user data in schema-less formats increase
development and deployment speed and changes to existing apps
– Supports multi-format data storage associated with differing device types
– Scalable storage provides for document, image, text oriented storage and
retrieval of apps data / information

Third Party Aggregation
– High speed ingestion of data feed from multiple sources:
• Retail store loyalty programs, social media, marketing information, purchase histories by groups,
person, industries

– Ease of format management and analysis

High Availability Cache
–
–
–
–

Storage for popular / frequent search results
Session information
High Frequency Ads tailored for User Profiles, Locations, searches
Dual function cache for data store and fast response time for retrieval

© 2013 IBM Corporation
Informix NoSQL Use Case (cont)
Globally Distributed Data Repository
– Scalable nodes and access across distributed systems
– Location affinity for workload optimization
– Capture of differing formats and languages to provide global and discrete views
of the business and analytics of searches, purchases, etc.

eCommerce
– Ability to handle purchase spikes / peak time through scalable system
architecture and low cost commodity hardware
– Fast / rapid application deployment through schema-less development model
– Ability to combine transaction data with user action reduces cost of upgrading
business apps for transaction management

Social Media & Gaming
– Rapid app development , deployment, and change implementation increase
ability to grow customer base and respond to trends – or create them
– Fast access to user profile for authentication and preferences / historical
information
– Real time Analytics and trend identification

© 2013 IBM Corporation
Informix NoSQL Use Cases (cont)
Ad Targeting
– Fast access to user profile and histories permit well managed Ad placement
increasing revenue and buy decision
– Click history and Pyschographic based suggestions
– Ability to ingest other feeds and quickly relate to a specific customer
• Loyalty programs
• Social Media
• Associations (LinkedIn, Dating sites, Industry Groups etc.)

© 2013 IBM Corporation
Informix and MongoDB Have Free Editions

Editions

MongoDB

Free

Developer
Innovator-C

Basic

For Purchase

11

Informix

Express, Workgroup,
Advanced Workgroup,
Enterprise, Advanced
Enterprise

Standard,
Enterprise
MongoDB Subscriptions
Basic

Standard

Enterprise

Edition

MongoDB

MongoDB

MongoDB
Enterprise

Support

9am-9pm local, M-F

24x7x365

24x7x365

License

AGPL

Commercial

Commercial

Emergency Patches

Not Included

Included

Included

Price

$2,500 / Server / Year

$5,000 / Server /
Year

$7,500 / Server /
Year

Additional monthly charges for backup services.

Subscription information obtained from 10Gen site, June 26, 2013.
12
Price Point Comparison Estimate, 3-year cost
Dual Core Intel
Nehalem

Innovator-C

Express

Workgroup

(4 core, 8 GB,
2 nodes)

(16 core, 16 GB,
unlimited nodes)

Product Cost

$0

$8,540

$19,740

$1,680

Included

Included

Support Renewal Year
2

$1,680

$1,708

$3,948

Support Renewal Year
3

$1,680

$1,708

$3,948

Total

$5,040

$11,956

$27,636

Support Subscription
Year 1
24 x 7 x 365
Production System Down
Development Call
Emergency Patches
Free Upgrades

MongoDB Enterprise, 3-year cost: $22,500
Subscription information obtained from 10Gen site, June 26, 2013.
Retail U.S. prices subject to change, valid as of June 26, 2013.
13
Back up

© 2013 IBM Corporation
Informix NoSQL Key Messages
Mobile computing represents one of the greatest opportunities for
organizations to expand their business, requiring mobile apps to have
access to all relevant data
DB2 and Informix have embraced the JSON standard and MongoDB API
to give mobile app developers the ability to combine critical data managed
by DB2 or Informix with the next generation mobile apps
DB2 JSON and Informix JSON let developers leverage the abilities of
relational DBMS together with document store systems

15

IBM Confidential

© 2013 IBM Corporation
DB2/Informix JSON for Install Base Clients
Leverage all relevant enterprise data for your mobile app development

The Offer

Client Value

Any for-purchase edition of

One solution for relational and non-relational data

• Latest version of DB2 LUW 10.5
• Latest version of Informix 12.1

• Relational tables and NoSQL collections co-exist in the
same database

Includes modern interface with JSON /BSON
support

• Allows joins between NoSQL and Relational tables

• Lets DB2/Informix be compatible with existing
MongoDB apps with minimal/no application changes
• Ability to handle a mix of structured and unstructured
data

• Joins utilize indexes on both relational and NoSQL
Brings Enterprise level performance, reliability, and
security to mobile apps
• Includes ACID compliance
Includes built-in online backup and restore capabilities

• Flexible schema allows rapid delivery of apps
• Compatible with MongoDB programming interfaces

Automatic Document Compression (only if they buy
Advanced Enterprise or add-on compression with
Enterprise Edition?)

16

IBM Confidential

© 2013 IBM Corporation
DB2 JSON Go-to-Market Timeline
Phase 1 – June through August 15

Phase 2 – eGA – Q4

Timing
Target

Phase 3 - 2014
Jan 1, 2014 – TBD

Targeted clients/partners
Opportunistic white space

Install Base – Top xxxx accounts
Key partners TBD
Opportunistic white space

DB2 Trial code with Tech Preview
Content
Tech preview on dW
A series of how-to articles on dW:
Part 1, Part 2, Part 3, Part 4.
Video: DB2 JSON Overview
Video: Getting started

Update NoSQL product page with
JSON content
Re-promote DB2 Trial code
IBM Champions for blogs
dW DM for JSON Community covreing
both DB2 and Informix

Who

Deliverables

Events

17

MongoDB World

Tech Bits and Tech Talks
Developer Days
DB2 Night Show via BP
Leverage existing 10Gen event
calendar to insert content
(these are potential – not confirmed)

IBM Confidential

On-going

Connect with AIM, Rational for joint
presence at 3rd conferences and
events.

© 2013 IBM Corporation
Informix JSON Go-to-Market Timeline
Phase 1 - EVP

Phase 2 – eGA – Q4

Phase 3 - 2014

Timing

Aug. 5, 2013 – Sep. 13, 2013

Sep. 13, 2013 – Dec. 31, 2013

Jan 1, 2014 – TBD

Target

•
•

•
•
•

Who

Confirmed EVP participants
Begooden
BP
Moukouri
BP
Consult-IX
BP
ADT
BP
Openbet
Cust
Technicolor
White space cust

Targeted clients/partners
Opportunistic white space

Install Base – Top xxxx accounts
Key partners TBD
Opportunistic white space
Top xxxx accounts
Key partners TBD
(criteria to be determined for each)

TBD

Potential EVP participants
Nuvento
White space cust
SRCE
BP
Cisco
OEM cust
Zebra Fish
BP
Deliverables

Events

18

NoSQL Client Business Value Deck
NoSQLTechnical White-paper
12.10.xC2 Release White Paper
Developer Works article on NoSQL Client
Updated NoSQL demo

•
•

Content on Informix marketing page
with re-direct to DevWorks page
Promote Informix trial code?

Chat with the Labs

•
•

IOD sessions
IOD Lab

IBM Confidential

Connect with AIM, Rational
for joint presence at 3rd
conferences and events.

© 2013 IBM Corporation
IOD 2013 Roadmap:
Extend Database Application Versatility with NoSQL
Session No.

Session Title

IPT-2059

JSON Document Support in IBM DB2 Schema Flexibility in a Trusted Database Environment

IPT-2909

Embracing NoSQL: IBM and the Community

IPT-3149

Big Data, Hadoop, and NoSQL: A Crash Course for a Fearless IBM DB2 Practitioner

IDB-1943

Firefighter Safety Enhancement through NoSQL Information Integration Architectures with IBM DB2
Graph Store

IDB-2075

Competitive Product Delivery with IBM DB2 JSON

IDB-1878

Agile Product Development Using IBM DB2 with JSON

IDX-3691

Get the Best of Both Worlds: Bring NoSQL to Your SQL Database

IDX-3722

Balancing Big Data Workloads with NoSQL Auto-Sharding

IDX-2551

"Developing ""Hybrid"" NoSQL/SQL Applications using JSON/BSON with Informix"

IPT-3070

Extending Information Management Solutions with NoSQL Capabilities--XML, RDF and JSON

IDZ-2599

NoSQL and DB2 for z/OS? Receiving Agility from a Trusted Enterprise Database

© 2013 IBM Corporation
DB2/Informix Joint Events 2014
Need to identify events where we can submit topics/speakers for
maximum exposure (not just booth participation)

20

IBM Confidential

© 2013 IBM Corporation
Some Typical NoSQL Use Cases
- Mostly Interactive Web/Mobile
Online/Mobile Gaming
−
−
−
−
−

Leaderboard (high score table)
management
Dynamic placement of visual elements
Game object management
Persisting game/user state information
Persisting user generated data (e.g.
drawings)

Display Advertising on Web Sites
−
−

Ad Serving: match content with profile and
present
Real-time bidding: match cookie profile with
ad inventory, obtain bids, and present ad

Communications
−

Device provisioning

Social Networking/Online
Communities

21

E-commerce/Social Commerce
– Storing frequently changing product
catalogs

Logging/message passing
– Drop Copy service in Financial
Services (streaming copies of trade
execution messages into (for
example) a risk or back office
system)

Dynamic Content Management
and Publishing (News & Media)
– Store content from distributed
authors, with fast retrieval and
placement
– Manage changing layouts and user
generated content
NoSQL = RDBMS + JSON + Sharding
NoSQL = JSON + Sharding
InformixNoSQL = JSON + Sharding + Transaction + RDBMS(Hybrid) +
HA(shared Disk) + HDR

22

IBM Confidential

© 2013 IBM Corporation
NOSQL, JSON AND BSON
OVERVIEW
Technical Opportunities/ Motivation
What are NoSQL Databases?
Quick overview of JSON
What is sharding?
New Era in Application Requirements
Store data from web/mobile application in their
native form
−

New web applications use JSON for storing and
exchanging information

−

Very lightweight – write more efficient applications

−

It is also the preferred data format for mobile
application back-ends

Move from development to production in no
time!
−

Ability to create and deploy flexible JSON schema

−

Gives power to application developers by reducing
dependency on IT

−

Ideal for agile, rapid development and continuous
integration
Why NoSQL?
Non-traditional data management requirements driven by Web
2.0 companies
−

Document stores, key-value stores, graph and columnar dbms

The Three Vs:
−

Velocity – high frequency of data arrivals

−

Volume – BigData

−

Variability – unstructured data, flexibility in
schema design

New data interchange formats – like JSON
(JavaScript Object Notation) and BSON
(Binary JSON)
Scale-out requirements across
heterogeneous environment – Cloud
computing

25
What is a NoSQL Database?
Not Only SQL or NOt allowing SQL
A non-relational database management systems
−

Does not require a fixed schema

−

Avoids join operations

−

Scales horizontally

−

No ACID (eventually consistent)

Good with distributing data and fast application development

Provides a mechanism for storage and retrieval of
data while providing horizontal scaling.
IBM Use Case Characteristics for JSON
Why Most Commercial Relational Databases cannot
meet these Requirements
NoSQL Database Philosophy Differences

No ACID
– No ACID (Atomicity, Consistency, Isolation, Durability)
– An eventual consistence model

No Joins
– Generally single row/document lookups

Flexible Schema
– Rigid format

29

© 2013 IBM Corporation
Partnership with IBM and MongoDB
MongoDB and IBM announced a partnership in June 2013

There are many common use cases of interest addressed by the
partnership
−
−
−
−

Accessing JSON Data in DB2, Informix MongoDB using JSON query
Schema-less JSON Data for variety of applications
Making JSON data available for variety of applications
Securing JSON Data

IBM and MongoDB are collaborating in 3 areas:
−
−
−

Open Governance: Standards and Open Source
Technology areas of mutual interest
Products and solutions
Basic Translation Terms/Concepts
Mongo/NoSQL Terms

Traditional SQL Terms

Database

Database

Collection

Table

Document

Row

Field

Column

Collection

Table
Name
John

31

Value

Document

28

Scott

Key

21

Tim

{"name":"John","age":21}
{"name":"Tim","age":28}
{"name":"Scott","age":30}

Age

30

Row
JSON Details
JSON Syntax Rules
–
–
–
–
–

JSON syntax is a subset of the JavaScript object notation syntax:
Data is in name/value pairs
Data is separated by commas
Curly braces hold objects
Square brackets hold arrays

JSON Name/Value Pairs
– JSON data is written as key/value pairs.
– A key/value pair consists of a field name (in double quotes), followed by a
colon, followed by a value:

"name":"John Miller"

The 6 types of JSON Values:
–
–
–
–
–
–
32

A number (integer or floating point)
A string (in double quotes)
A Boolean (true or false)
An array (in square brackets)
An object (in curly brackets)
Null
Example of Supported JSON Types

Example of each JSON
type
Mongo specific JSON
types in blue
– date

{
"string":"John",
"number":123.45,
"boolean":true,
"array":[ "a", "b", "c" ],
"object: { "str":"Miller", "num":711 },
"value": NULL,
"date": ISODate("2013-10-01T00:33:14.000Z")
}
JSON: JavaScript Object Notation
What is JSON?
−

JSON is lightweight text-data interchange format

−

JSON is language independent

−

JSON is "self-describing" and easy to understand

JSON is syntax for storing and exchanging text information much like
XML. However, JSON is smaller than XML, and faster and easier to
parse.
{
"name":"John Miller",
"age":21,
"count":27,
"employees": [
{ "f_name":"John" , "l_name":"Doe" },
{ "f_name":"Anna","m_name" : "Marie","l_name":"Smith" },
{ "f_name":"Peter" , "l_name":"Jones" }
]
}

BSON is a binary form of JSON.
34

© 2013 IBM Corporation
The Power of JSON Drives Flexible Schema
JSON key value pair enables a flexible schema
Data Access Examples
Relational Representation
LName

FName

Address

Miller

John

123 Blazer St

JSON Representation
JSON_string = ‘{“LName”:”Miller”,”FName”,”John”,”Address”:”123 Blazer St”}’;

Javascript data access
var JSONelems = JSON.parse( JSON_string )
f_name = JSONelems.FNname;
l_name = JSONelems.FName;
l_addr = JSONelems.Address;
Simple Code Example

db.posts.insert( ‘{“author”:”John”, “date”,”2013-04-20”,”post”,”mypost

Creates the database “db” if it does not exists
Creates the collection “posts” if the it does not exists
Insert a record into a blog post by user John

db.posts.find ( ‘{ “author”:”John” }’ )

Retrieve all posts by user John

”}’ )
Dynamic Elasticity

Rapid horizontal scalability
−

Ability for the application to grow by adding low cost hardware
to the solution

−

Ability to add or delete nodes dynamically

−

Ability rebalance the data dynamically

Application transparent elasticity
Why Scale Out Instead of Up?
Scaling Out
−

Adding more servers with less processors and RAM

−

Advantages
Startup costs much less
Can grow instep with the application
Individual servers cost less
−

Several less expensive server rather than fewer high cost servers

−

Redundant servers cost more

Greater chance of isolating catastrophic failures

Scaling Up
−

Adding more processors and RAM to a single server

−

Advantages
Less power consumption than running multiple servers
Less infrastructure (network, licensing,..)
Difference between Sharding Data VS Replication
Shard Key
state= “CA”

Shard Key
state= “WA”

Sharding
Each node hold a
portion of the data
• Hash
• Expression

Data is copied to all
nodes

Actions are shipped
to applicable nodes

40

Same data on each
node

Inserted data is
placed on the correct
node

Shard Key
state= “OR”

Replication

Work on local copy
and modification are
propagated
Motivation of Sharding

Synergistic with the application strategy
Start small and grow with commodity hardware

Sharding is not
Sharding is not for Data Availability
Sharding is for growth, not availability
Redundancy of a node provides high availability for the data
−

Both Mongo and Informix allow for multiple redundant nodes

−

Mongo refers to this as Replica Sets and the additional nodes
slaves

−

Informix refers to this as MACH, and additional nodes secondary

With Informix the secondary server can:
−

Provide high availability

−

Scale out
Execute select
Allow Insert/Update/Deletes on the secondary servers
Share Disks with the master/primary node
Basic Data Distribution/Replication Terms
Term

Description

Informix Term

Shard

A single node or a group of nodes holding the same data
(replica set)

Instance

Replica Set

A collection of nodes contain the same data

MACH Cluster

Shard Key

The field that dictates the distribution of the documents.
Must always exist in a document.

Shard Key

Sharded
Cluster

A group shards were each shard contains a portion of the
data.

Grid/Region

Slave

A server which contains a second copy of the data for read
only processing.

Secondary Server
Remote Secondary

43
Sharding is not for Data Availibility
Shard Key
state= “CA”

Shard Key
state= “WA”

Shard Key
state= “OR”
44
NEW INFORMIX NOSQL/JSON
CAPABILITIES
IBM Use Case Characteristics for JSON
High Level Solution

47
Flexible Schema, Native JSON and BSON
Provide native JSON & BSON support in the Informix Database Server
Two new built in “first-class” data types called JSON and BSON
Support for MongoDB client side APIs throughout the entire database
Enhance horizontal scaling by enabling sharding on all database objects
and models
Adaptive default system initialization
−
−

48

Up and running in seconds
Adapts to computer an environment
Two New Data Types JSON and BSON
Native JSON and BSON data types
Index support for NoSQL data types
Native operators and comparator functions
allow for direct manipulation of the BSON
data type
Database Server seamlessly converts to
and from

JSON
BSON
Character data

49

JSON
JSON and BSON Data Type Details
Row locks the individual BSON/JSON document
−

MongoDB must lock the database

Bigger documents – 2GB maximum size
−

MongoDB caps at 16MB

Ability to compress documents
−

MongoDB currently not available

Ability to intelligently cache commonly used documents
−

MongoDB currently not available
Client Applications
Applications
MongoDB
native Client

New Wire Protocol Listener supports
existing MongoDB drivers
Connect to MongoDB or Informix with same
application!

MongoDB
driver

MongoDB
Wire
Protocol
IBM
JDBC
NoSQL
Driver
Wire
Protocol
Listener

MongoDB
web browser

MongoDB shell

Mobile

Informix
DB
MongoDB Application Compatibly
Ability to use any of the MongodB client drivers and frameworks
against the Informix Database Server
−

Little to no change required when running MongoDB programs

−

Informix listens on the same default port as mongo, no need to change.

Leverage the different programming languages available
−

Language examples C, C#, Erlang, JAVA, node.js, Perl, Python, Ruby

Mongo Action

Description

db.customer.insert( { name: “John", age: 21 } )

Insert into database “db” the customer
collection the associated document.

db.customer.find( {age: { $gt:21 } } )

Retrieve from database “db” the customer
collection all documents with the age
greater than 21.

52
Fix up JSON Basic Mongo Operations
Conceptual Operations
Mongo Action
db.customer.insert( { name: “John", age:
21 } )

CREATE DATABASE if not exist db
CREATE TABLE if not exist customer
INSERT INTO customer
VALUES ( { “name”:”John”,”age:21”} )

db.customer.find()

SELECT bson_doc FROM customer

db.customer.find( {age: { $gt:21 } } )

SELECT * FROM customer WHERE age > 21

db.customer.drop()

DROP TABLE customer

db.customer.ensureIndex( { name : 1,
age : -1 } )

CREATE INDEX idx_1 on customer(name ,
age DESC)

db.customer.remove( {age: { $gt:21 } } )

DELETE FROM customer where age > 21

db.customer.update( { age: { $gt: 20 } }, {
$set: { status: “Drink" } }, { multi: true } )

53

Traditional SQL Action

UPDATE customer
SET bson_doc_field (status) = { “status”:“Drink"
}
WHERE age > 20
Scaling Out Using Sharded Queries

Shard Key
state= “CA”

Shard Key
state= “WA”

Find sold cars for
all states

1. Request data from local shard
2. Automatically sends request to
other shards requesting data
3. Returns results to client
Shard Key
state= “OR”
54

© 2013 IBM Corporation
Scaling Out Using Sharded Inserts

Shard Key
state= “CA”

Shard Key
state= “WA”

Row
state = “OR”

1. Insert row sent to your local
shard
2. Automatically forward the data to
the proper shard
Shard Key
state= “OR”
55

© 2013 IBM Corporation
Scaling Out Using Sharded Delete

Shard Key
state= “CA”

Shard Key
state= “WA”
Delete
state = “OR” AND
Dnum = “123123”

1. Delete condition sent to local shard
2. Local shard determine which
shard(s) to forward the operation
Shard Key
state= “OR”
56

3. Execute the operation

© 2013 IBM Corporation
Scaling Out Using Sharded Update

Shard Key
state= “CA”

Shard Key
state= “WA”

Row
state = “OR”

Row
state = “OR”

1. Insert a row on your local shard
2. Automatically forward the data to
the proper shard
Shard Key
state= “OR”
57

© 2013 IBM Corporation
Scaling Out Adding a Shard
Shard Key
state= “CA”

Shard Key
state= “WA”

Command
Add Shard “NV”

1. Send command to local node
2. New shard dynamically added,
data re-distributed (if required)

Shard Key
state= “OR”
58

Shard Key
state= “NV”
© 2013 IBM Corporation
Sharding with Hash

Hash based sharding simplifies the
partitioning of data across the shards
Benefits
−

No data layout planning is required

−

Adding additional nodes is online and
dynamic

Cons
−

Adding additional node requires data to be
moved

Data automatically broken in pieces
Scaling Out with Hash Sharding - Insert

Shard Key
HASH( gambler_ID )

Shard Key
HASH( gambler_ID )

Row
Gambler_ID =“11”

1. Insert row sent to your local shard
2. Local shard determine which
shard(s) to forward the insert
3. Execute the operation
Shard Key
HASH( gambler_ID )
60

© 2013 IBM Corporation
Move to END Informix NoSQL Cluster Architecture
Overview

Shard Key
state= “CA”

Shard Key
state= “OR”

61

two independent copies of the
data, but three servers to
share the workload (two
servers share the same
disks). Read/Write activity
supported on all servers independent copies of
two
the data and two servers
to share the workload.
Read/Write activity
supported on all servers
three independent copies of
the data, but four servers
to share the workload (two
servers share the same
disk). Read/Write activity
supported on all servers

Shard Key
state= “WA”

© 2013 IBM Corporation
SHARDING NEEDED HERE
INFORMIX NOSQL
DEEP DIVE INTO
IMPLEMENTATION
Focus Areas.
Applications need easier way to persist objects.
−

Object-relational layers like hibernate has performance & functional issues.

−

Support Mongo API and its eco system

Flexible Schema: Changing schema in RDBMS is a significant operation,
especially in production.
−

Need exclusive access to table & application downtime

−

Fast changing schema & sparse data has issues with fixed schema.
-- Implement

JSON (BSON) type, data store & indexing

Scale Out: While RDBMS has solutions for cluster and MPP, features
enabling application development and deployment in cloud+MPP
architecture is limited.
−

Use Informix flexible grid & replication to shard tables;

−

Enhance query processing for distributed queries
Client Applications
New Wire Protocol Listener supports existing MongoDB drivers
Connect to MongoDB or Informix with same application!
MongoDB
native Client
application

MongoDB

MongoDB
driver
IBM
JDBC
NoSQL
Driver
Wire
Protocol
Listener

MongoDB web
browser
application

Informix
NoSQL
Cluster

MongoDB shell

IBM Wire Protocol Listener logic shared with DB2 and Extreme Scale

65

© 2013 IBM Corporation
Informix for SQL and NoSQL Applications
NoSQL APP
IBM Wire Listener
IBM Wire Listener

JDBC connections
JDBC connections

SQL Apps + Tools

SQL Apps + Tools

SQL Drivers
ODBC/JDBC

NoSQL APP

SQL Drivers
ODBC/JDBC

IBM Wire Listener
JDBC connections

Informix Dynamic Server (shard 1)
Informix Dynamic

Query processing
Distributed
Queries
JSON
Tables

Collections

Server (shard…n)

IDXs
IDXs

JSON
Tables

Collections

Tables
Tables
Tables

IDXs

Logs

Enterprise replication + Flexible Grid

Tables

IDXs

ER + Grid

Logs
Informix for SQL and NoSQL Applications
Mongo Application

SQL Apps + Tools

NoSQL/BSON

SQL

IBM Wire Listener

SQL Drivers
ODBC/JDBC

JDBC connections
SQL + BSON

SQL

Informix Dynamic Server (shard 1)
Query processing
JSON
Tables
Collections

Distributed
Queries

SELECT

Informix Dynamic
Server (shard…n)

IDXs

IDXs

JSON
Tables

Collections

Tables
Tables
Tables

IDXs

Logs

Enterprise replication + Flexible Grid

INSERT
UPDATE
DELETE

Tables

IDXs

ER + Grid

Logs
NoSQL Feature Checklist.
Informix NoSQL & Hybrid SQL detailed deep dive
Informix NoSQL & Hybrid SQL detailed deep dive
Informix NoSQL & Hybrid SQL detailed deep dive
NoSQL Feature

Informix Implementation

1. Flexible Schema

Use BSON and JSON data type. Complete row is stored in a single column; BSON,
JSON are multi-rep types and can store up to 2GB.

2. Accessing KV pairs
within JSON.

Translate the Mongo queries into SQL expressions to access key-value pairs.
Informix has added expressions to extract specific KV pairs.
Select bson_new(data, “{id:1, name:1, addr:1}”) from tab;

3. Indexing

Support standard B-tree index via functional indexing.
create index itid on t(bson_value_int(data, “{id:1}”);
Create index itnamestate on t(bson_value_lvarchar(data, “{name,1}”),
bson_value_lvarchar(data, “{city,1}”);
Informix also supports indexing bsons keys with different data types.

4. Sharding

Supports range & hash based sharding.Informix has built-in technology for
replication. Create identical tables in multiple nodes. Add meta data for partitioning
the data across based on range or hash expressions.

5. SELECT

Limited support now. Mongo API helps by disallowing joins. The translated query on
a single table is transformed into federated UNION ALL query; includes shard
elimination.
NoSQL Feature

Informix Implementation

6. Updates (single node)

INSERT: Simple direct insert.
DELETE: DELETE statement with WHERE bson_extract() > ?; or bson_value..() > ?
UPDATE: bson_update(bson, “update expr”) will return a new bson after applying the
bson expression. Simple updates to non_sharded tables will be direct UPDATE
statement in a single transaction. UPDATE tab bsoncol = bson_update(bsoncol,
“expr”) where

7. Updates (sharded env)

INSERT – All rows are inserted to LOCAL shard, replication threads will read logs
and replicate to the right shard and delete from local node (log only inserts
underway).
DELETE – Do the local delete and replicate the “delete statements” to target node in
the background
UPDATE – Slow update via select-delete-insert.

8. Transaction

Each statement is a distinct transaction in a single node environment.
The data and the operation is replicated via enterprise replication.

9. Isolation levels.

NoSQL session can use any of Informix isolation levels.
-- Examples from Mongo applications.

10. Locking

Application control only on the node they’re connected to. Standard
Informix locking semantics will apply when data is replicated and applied
on the target shard.
-- Verify.. What’s the default locking level?
-- Any option to change it? Just ONCONFIG??
Feature

Informix Implementation

11. Hybrid access

Mongo APIs can simply access Informix tables, views, virtual tables as if they’re
JSON collections. db.foo.find(); foo can be JSON collection, relational table, a
view or a virtual table on top of timeseries or Websphere MQ.

From MongoAPI to
relational tables.
12. Hybrid access

1.

Directly get binary BSON or cast to JSON to get in textual form.

From SQL to JSON data

2.

Use expressions to extract to extract specific key-value pairs.

3.

To be done: NoSQL collections will only have one BSON object in the table.
We can “imply” the expressions when the SQL refers to a column.
SELECT t.c1, t.c2 from t;
SELECT bson_extract(t.data, “{c1:1}”), bson_extract(t.data, “{c2:1}”) from t;

So, that’s twelve steps for NoSQL Anonymous!
1. Flexible Schema
Clients exchange BSON document with the server both for queries and data.
Thus, BSON becomes a fundamental data type.
The explicit key-value pairs withing the JSON/BSON document will be roughly
equivalent to columns in relational tables.
However, there are differences!
−

The type of the KV data encoded within BSON is determined by the client

−

Server is unaware of data type of each KV pair at table definition time.

−

No guarantees that data type for each key will remain consistent in the
collection.

−

The keys in the BSON document can be arbitrary;

−

While customers exploit flexible schema, they’re unlikely to create a single
collection and dump everything under the sun into that collection.

−

Due to the limitations of Mongo/NoSQL API, customers typically denormalize
the tables (customer will have customer+customer addr + customer
demographics/etc) to avoid joins.
1. Flexible Schema – Informix Implementation
• Informix has a new data type BSON to store the data.
• Informix also has a JSON data type to convert between binary and
text form.
• BSON and JSON are abstract data types (like spatial, etc).
• BSON and JSON multi-representational types.
•Objects up to 4K is stored in data pages.
•Larger objects (up to 2GB) are stored out of row, in BLOBs.
•MongoDB limits objects to 16MB.
•This is all seamless and transparent to applications.
1. Flexible Schema – Informix Implementation
CREATE TABLE customer (data BSON);
• BSON is the binary represenation of JSON.
•It has length and types of the key-value pairs in JSON.
• MongoDB drivers send and receive in BSON form.
2. Accessing KV pairs within JSON.
• We’ll have number of extract expressions/functions
•Expressions returning base type
bson_value_bigint(BSON, “key”);
bson_value_lvarchar(bsoncol, “key.key2”);
bson_value_date(bsoncol, “key.key2.key3”);
bson_value_timestamp(bsoncol, “key”)
bson_value_double(bsoncol, “key”);
bson_value_boolean(bsoncol, “key”);
bson_value_array(bsoncol, “key”);
bson_keys_exist(bsoncol, “key”);
Bson_value_document(bsoncol, “key”)
Bson_value_binary(bsonol, “key”)
Bson_value_objectid(bsoncol, “key”)
•Expression returning BSON subset. Used for bson indices.
bson_extract(bsoncol, “projection specification”)
•Expressions to project out of SELECT statement.
bson_new(bsoncol, “{key1:1, key2:1, key3:1}”);
bson_new(bsoncol, “{key5:0}”);
2. Accessing KV pairs within JSON.
Mongo Query

SQL Query

db.customers.find();

SELECT SKIP ? Data::bson

db.customers.find({},{num:1,name:1})
;

SELECT SKIP ? bson_new( data, '{ "num" : 1.0 , "name"
: 1.0}')::bson FROM
customers

db.customers.find({},
{_id:0,num:1,name:1});

SELECT SKIP ? bson_new( data, '{_id:0.0, "num" : 1.0
, "name" : 1.0}')::bson FROM
customers

db.customers.find({status:”A”})

SELECT SKIP ? data FROM customers WHERE
bson_extract(data, ‘status') = “A”

db.customers.find({status:”A”},
{_id:0,num:1,name:1});

FROM

customers

SELECT SKIP ?
bson_new( data, '{ "_id" : 0.0 , "num" : 1.0 , "name"
: 1.0}')::bson
FROM customers WHERE bson_extract(data, 'name') = “A”
3. Indexing
•
•
•
•
•

Supports B-Tree indexes on any key-value pairs.
Indices could be on simple basic type (int, decimal) or BSON
Indices could be created on BSON and use BSON type comparison
Listener translates ensureIndex() to CREATE INDEX
Listener translates dropIndex() to DROP INDEX

Mongo Query

SQL Query

db.customers.ensureIndex({orderD
ate:1})

CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x')
ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" :
"newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1"
, "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64

db.customers.ensureIndex({orderD
ate:1, zip:-1})

CREATE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC,
bson_extract(data,'c2') DESC) using bson (ns='{ "name" :
"newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0
, "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" : "c1_1_c2__1" ,
"index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64

db.customers.ensureIndex({orderD
ate:1}, {unique:true)

CREATE UNIQUE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC,
bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}',
idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 ,
"$extract"]} , "name" :"c1_1_c2__1" , "unique" : true , "index" : "v_c1_1_c2__1"}')
EXTENT SIZE
64 NEXT SIZE 64
3. Indexing
db.w.find({x:1,z:44},{x:1,y:1,z:1})
Translate to:
SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1}
FROM w
WHERE ( bson_extract(data, 'x') = '{ "x" : 1.0 }'::json::bson ) AND
( bson_extract(data, 'z') = '{ "z" : 44.0 }'::json::bson)
Estimated Cost: 2
Estimated # of Rows Returned: 1
1) keshav.w: SEQUENTIAL SCAN
Filters: (informix.equal(informix.bson_extract(keshav.w.data ,'z' ),UDT )
AND informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT))
3. Indexing
•Functional Index is built on bson expressions
CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC)
using bson (ns='{ "name" : "newdb.w.$x_1"}',
idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name"
: "x_1" , "index" : "w_x_1"}')
EXTENT SIZE 64 NEXT SIZE 64

•Listener is aware of the available index and therefore
generates right predicates.
db.w.find({x:1});
gets translated to
SELECT SKIP ? data FROM w WHERE bson_extract(data, 'x') = ?
3. Indexing
db.w.find({x:5,z:5}, {x:1,y:1,z:1})
Translates to:
SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1 .0}')::bson
FROM w
WHERE ( bson_extract(data, 'x') = '{ "x" : 5.0 }'::json::bson )
AND ( bson_extract(data, 'z') = '{ "z" : 5.0 }'::json::bson )
Estimated Cost: 11
Estimated # of Rows Returned: 1
1) keshav.w: INDEX PATH
Filters: informix.equal(informix.bson_extract(keshav.w.data ,'z' ), UDT )
(1) Index Name: keshav.w_x_1
Index Keys: informix.bson_extract(data,'x') (Serial, fragments: ALL)
Lower Index Filter: informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT )
84

© 2013 IBM Corporation
4. Mongo Sharding
4. Mongodb SHARDING (roughly)
Shard a single table by range or hashing.
Mongos will direct the INSERT to target shard.
Mongos tries to eliminate shards for update, delete, selects as well.
FIND (SELECT) can happen ONLY a SINGLE table.
It also works as coordinator for multi-node ops.
Once a row is inserted to a shard, it remains there despite any key
update.
No transactional support on multi-node updates.
−

Each document update is unto its own.
4. Informix Sharding
App Server

App Server

App Server

App Server

Mongo Driver

Mongo Driver

Mongo Driver

Mongo Driver

Listener

Listener

Listener

Listener

JDBC

JDBC

JDBC

JDBC

Customer/1

Customer/2

Customer/3

Customer/4

Customer/5

Customer/6

Sales/1

Sales/2

Sales/3

Sales/4

Sales/5

Sales/6

Location/1

Location/2

Location/3

Location/4

Location/5

Location/6

Informix/2

Informix/3

Informix/1

Informix/4

Informix/5

Enterprise repliation + Flexible Grid

Informix/6
4. Informix sharding + High Availability
App Server

App Server

App Server

App Server

Mongo Driver

Mongo Driver

Mongo Driver

Mongo Driver

Listener

Listener

Listener

Listener

JDBC

JDBC

JDBC

JDBC

Informix/1
Primary

Informix/2
Primary

Informix/3
Primary

Informix/4
Primary

Informix/5
Primary

Informix/6
Primary

Informix/1
SDS/HDR

Informix/2
SDS/HDR

Informix/3
SDS/HDR

Informix/4
SDS/HDR

Informix/5
SDS/HDR

Informix/6
SDS/HDR

Informix/1
RSS

Informix/2
RSS

Informix/3
RSS

Informix/4
RSS

Informix/5
RSS

Informix/6
RSS

Enterprise repliation + Flexible Grid
4. Sharding – Informix Implementation
•Shard a single table by range or hashing.
cdr define shard myshard mydb:usr1.mytab 
–type=delete –key=”bson_get(bsoncol, ‘STATE’)” –stragety=expression 
versionCol=version 
servA “in (‘TX’, ‘OK’)” 
servB “in (‘NY’,’NJ’) “ 
servC “in (‘AL’,’KS’) “ 
servD remainder
cdr define shard myshard mydb:usr1.mytab 
–type=delete –key=state –stragety=hash --versionCol=version 
servA servB servC servD

•Shard a single table by range or hashing.
<<Mongo syntax>>
4. Sharding – Informix Implementation

•Shard a single table by range or hashing.
•Sharding is transparent to application.
•Each CRUD statement will only touch a single table.
•Limitation of MongoDB…Makes it easier for Informix.
•Lack of joins is a big limitation for SQL applications.
•Lacks transactional support for distributed update.
4. Informix Sharding
Identical table is created on each node and meta data is replicated
on each node.
Schema based replication is the foundation for our sharding.
CRUD operations can go to any node.
−

We’ll use replication and other techniques to reflect the data in the
target node, eventually.

−

Right now, replication is asynchronous.

−

Informix has synchronous replication

not used for sharding now.
5. SELECT on sharded tables
•The query can be submitted to any of the nodes via the listener.
•That node acts as the “coordinator” for the distributed query.
•It also does the node elimination based on the query predicate.
•After that, the query is transformed to UNION ALL query
SELECT SKIP ? bson_new( data, '{"_id":0.0 ,"num":1.0 ,"name" : 1.0}')::bson
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X”

is transformed into:
SELECT SKIP ? bson_new( data, '{"_id":0.0
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or
UNION ALL
SELECT SKIP ? bson_new( data, '{"_id":0.0
FROM customers@rsys1:db
WHERE bson_extract(data, 'name') = “A” or

,"num":1.0 ,"name" : 1.0}')::bson
bson_extract(data, 'name') = “X”
,"num":1.0 ,"name" : 1.0}')::bson
bson_extract(data, 'name') = “X”
6. INSERT: Single node

•If necessary, automatically create database &
table (collection) on the application INSERT
•Collections: CREATE TABLE t(a GUID, d
BSON);
•GUID column is needed to ensure unique row across
the SHARDs. Also used as PK for replication.
•Client application inserts JSON, client API converts
this into BSON, generates _id (object id) if necessary &
sends to server over JDBC/ODBC.
•Server saves the data into this table as BSON, with
an automatically generated GUID
6. DELETE: Single node
•Mongo remove are translated to SQL DELETE
•Will always remove all the qualifying rows.
•WHERE clause translation is same as SELECT
6. UPDATE: Single node

•Simple set, increment, decrement updates are
handled directly by the server.
Mongo: db.w.update({x: {$gt:55}}, {$set:{z:9595}});
SQL: UPDATE w SET data = bson_update(data, “{$set:{z:9595}}”)
WHERE bson_extract(data, 'x') > “{x:55}”::bson ?

•bson_update is a built-in expression updating a
BSON document with a given set of operations.
•Complex updates are handled via select batch,
delete, insert.
•Always updates all the rows or no rows, under a
transaction.
7. INSERT: shard implementation

•If the table is sharded:
•Insert the data into local table as usual.
•Replication threads in the background will evaluate
the log record to see which rows will have moved.
•Replication thread will move the necessary row to
target and delete from the local table.
•Work underway to avoid the insert into local table.
•For each inserted row ending up in non-local shard,
simply generate the logical log and avoid insert into
local table & indices.
7. DELETE : shard implementation
•Application delete could delete rows from any of the
shards in the table.
•DELETE will come in as shard_delete() procedure.
–Execute procedure shard_delete(tabname, delete_stmt);
–This procedure will issue delete locally.
–It will then INSERT the delete statement into a “special”
shard delete table (single for all tables).
–Enterprise replication will propagate the delete to
applicable target systems.
7. UPDATE: shard implementation

•When the update have to be done on multiple
nodes:
•Client application does UPDATE and CLIENT API
converts that into three operations: SELECT,
DELETE & INSERT.
•GUID column is needed to ensure unique row across the
SHARD
•Client application inserts JSON, client API converts this
into BSON & sends to server.
8. Transactions (single node)
•Mongo does not have the notion of transactions.
•Each document update is atomic, but not the app statement
•For the first release of Informix-NoSQL
•By default, JDBC listener simply uses AUTO COMMIT option
•Each server operation INSERT, UPDATE, DELETE, SELECT will be
automatically be committed after each operation.
•No locks are held across multiple operation.
•However, customers can create multi-statement transaction
via $sql and issuing begin work, commit work, rollback work
8. Transactions (sharded environment)
•In sharded environment, mongo runs database via two different
instances: mongos and mongod.
•Mongos simply redirects operations to relevant mongod.
•No statement level transactoinal support.
•Informix
•Informix does not have the 2-layer architecture
•Informix server the application connected to becomes the
transaction coordinator
•Informix does have the 2-phase commit transaction support,
but is unused for NoSQl, for now.
•SELECT statement goes thru distributed query infrastructure
•INSERT,UPDATE DELETE goes thru enterprise replication.
9. ISOLATION levels

•Default isolation level is DIRTY READ.
•Change this directly or sysdbopen()
•You can also use USELASTCOMMITTED variable in
ONCONFIG.
•If you’re using procedures for executing multistatement transaction, you can set it within your
procedure.
10. LOCKING

•Page level locking is the default.
•You can change it to ROW level locks easily.
•ALTER TABLE jc MODIFY lock mode (row)
• DEF_TABLE_LOCKMODE onconfig variable.
• SET LOCK MODE can be set via sysdbopen()
•Each statement is executed with auto-commit and locking
semantics will apply there.
Hybrid Power
Hybrid Access between relational & JSON Collections

Relational Table

SQL API

Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.

MongoDB API
(NoSQL)

?

JSON Collections

?
Mongo APIs for Java,
Javascript, C++, C#,...
Benefits of Hybrid Power
Access consistent data from its source
Avoid ETL, continuous data sync and conflicts.
Exploit the power of SQL, MongoAPI seamlessly
Exploit the power of RDBMS technologies in MongoAPI:
−

Informix Warehouse accelerator,

−

Cost based Optimizer

−

R-tree indices for spatial, Lucene text indexes, and more.

Access all your data thru any interface: MongoAPI or SQL.
Store data in one place and efficiently transform and use them on
demand.
Existing SQL based tools and APIs can access new data in JSON
Why do you need hybrid access?

Data model
should not restrict
Data Access
Hybrid Access between relational & JSON Collections

Relational Table

SQL API

MongoDB API
(NoSQL)

Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.

Mongo APIs for Java,
Javascript, C++, C#,...

JSON Collections

Direct SQL Access.
Dynamic Views
Row types

Mongo APIs for Java,
Javascript, C++, C#,...
Hybrid Power
MongoAPI to relational data
Hybrid access: From MongoAPI to relational tables.
You want to develop an application with MongoAPI, but
1. You already have relational tables with data.
2. You have views on relational data
3. You need to join tables
4. You need queries with complex expressions. E.g. OLAP window functions.
5. You need to get results from a stored procedure.
6. You need to exploit Informix stored procedure
7. You need federated access to other data
8. You have timeseries data.
How to treat relational data as JSON store.
Relational data (relations or resultset) can treated as structured JSON
documents; column name-value becomes key-value pair.
SELECT partner, pnum, country from partners;
partner
pnum Country
Pronto
1748 Australia
Kazer
1746 USA
Diester
1472 Spain
Consultix
1742 France
{parnter:
{parnter:
{parnter:
{parnter:

“Pronto”, pnum:”1748”, Country: “Australia”}
“Kazer”, pnum:”1746”, Country: “USA”}
“Diester”, pnum:”1472”, Country: “Spain”}
“Consultix”, pnum:”1742”, Country: “France”}

Listner translates the query and the data object between relational and
JSON/BSON form.
Hybrid access: From MongoAPI to relational tables.
Mongo Application
JSON

JSON

db.customer.find({state:”MO”})

db.partners.find({state:”CA”})

IBM Wire Listener
JDBC connections

Access JSON

SELECT bson_new(bson, ‘{}’) FROM customer
WHERE bson_value_lvarchar(bson,‘state’)= “MO”

Access Relational

SELECT * FROM partners WHERE

state = “CA”

Informix Dynamic Server
JSON Collections

Customer

Distributed
Queries

Tables
IDXs
Tables

Relational Tables

Logs

partners

Tables

IDXs

Enterprise replication + Flexible Grid
Accessing data in relational tables.
Create table partners(pnum int, name varchar(32), country
varchar(32));
db.partners.find();
SELECT * from partners;
db.partners.find({name:”Pronto”});
SELECT * FROM PARTNERS WHERE name = “Pronto”;
db.partners.find({name:”Pronto”}, {pnum:1, country:1});
SELECT a, b FROM t WHERE a = 2.0;
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).limit(10);
SELECT LIMIT 10 pnum, country FROM WHERE name = “Pronto”;
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:1})
SELECT pnum,country FROM partners WHERE name = “Pronto” ORDER BY b ASC
db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:-1})
SELECT a, b FROM t WHERE a = 2.0 ORDER BY b DESC
db.t.find({a:{$gt:1}}, {a:1, b:1}).sort({b:-1})
SELECT SKIP ? a, b FROM t WHERE query > 1.0 ORDER BY b DESC
Accessing data in relational tables.
db.partners.save({pnum:1632, name:”EuroTop”, Country: “Belgium”});
INSERT into partners(pnum, name, country values(1632, ”EuroTop”,
“Belgium”);
db.partners.delete({name:”Artics”});
DELETE FROM PARTNERS WHERE name = “Artics”;
Db.partners.update({country:”Holland”},{$set:{country:”Netherland”}}
, {multi: true});
UPDATE partners SET country = “Netherland” WHERE country =
“Holland”;
db.partners.drop();
DROP TABLE partners;
Views and JOINS
A dynamically created relation created from one or more database
tables or procedures.
create table pcontact(pnum int, name varchar(32), phone varchar(32));
insert into pcontact values(1748, "Joe Smith", "61-123-4821");
insert into pcontact values(1746, "John Kelley", "1-729-284-2893");
insert into pcontact values(1472, "Ken Garcia", "34-829-2842");
insert into pcontact values(1742, "Adam Roy", "33-380-3892");
create view partnerphone(pname, pcontact, pphone) as select a.name, b.name, b.phone
FROM pcontact b left outer join partners a on (a.pnum = b.pnum);

db.partnerphone.find();
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" }
{ "pname" : "Kazer", "pcontact" : "John Kelley", "pphone" : "1-729-284-2893" }
{ "pname" : "Diester", "pcontact" : "Ken Garcia", "pphone" : "34-98-829-2842" }
{ "pname" : "Consultix", "pcontact" : "Adam Roy", "pphone" : "33-82-380-3892" }

db.partnerphone.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" }
complex expressions. E.g. OLAP window functions
create view contactreport(pname, pcontact, totalcontacts) as
select b.name, a.name,
count(a.name) over(partition by b.pnum)
from pcontact a left outer join partners b on (a.pnum = b.pnum);

db.contactreport.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 }
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 }
Seamless federated access
1.
2.

create database newdb2;
create synonym oldcontactreport for
newdb:contactreport;

> use newdb2
> db.oldcontactreport.find({pname:"Pronto"})
{ "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 }
{ "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 }
SELECT data FROM oldcontactreport WHERE
bson_extract(data, 'pname') = “Pronto”;
•

create synonym oldcontactreport for
custdb@nydb:contactreport;
Get results from a stored procedure.
create function "keshav".p6() returns int, varchar(32);
define x int; define y varchar(32);
foreach cursor for select tabid, tabname into x,y from systables
return x,y with resume;
end foreach;
end procedure;
create view "keshav".v6 (c1,c2) as
select x0.c1 ,x0.c2 from table(function p6())x0(c1,c2);

db.v6.find().limit(5)
{
{
{
{
{

"c1"
"c1"
"c1"
"c1"
"c1"

:
:
:
:
:

1,
2,
3,
4,
5,

"c2"
"c2"
"c2"
"c2"
"c2"

:
:
:
:
:

"systables" }
"syscolumns" }
"sysindices" }
"systabauth" }
"syscolauth" }
Access Timeseries data
create table daily_stocks
( stock_id integer,
stock_name lvarchar,
stock_data timeseries(stock_bar)
);
-- Create virtual relational table on top (view)
EXECUTE PROCEDURE
TSCreateVirtualTab('daily_stocks_virt',
'daily_stocks', 'calendar(daycal),origin(2011-01-03
00:00:00.00000)' );
create table daily_stocks_virt
( stock_id integer,
stock_name lvarchar,
timestamp datetime year to fraction(5),
high smallfloat,
Access Timeseries data
db.daily_stocks_virt.find()
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0
0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0
0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0
0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 }
{ "stock_id" : 902, "stock_name" : "HPQ", "timestamp" : ISODate("2011-01-03T06:0
0:00Z"), "high" : 403, "low" : 401, "final" : 402, "vol" : 253 }
{ "stock_id" : 902, "stock_name" : "HPQ", "timestamp" : ISODate("2011-01-04T06:0
0:00Z"), "high" : 503, "low" : 501, "final" : 502, "vol" : 302 }
Access Timeseries data
db.daily_stocks_virt.find({stock_name:"IBM"})
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0
0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0
0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0
0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 }

db.daily_stocks_virt.find({stock_name:"IBM"}).sort({final:-1})
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0
0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0
0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 }
{ "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0
0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 }
mongos>
Hybrid access: From MongoAPI to relational tables.
You want to develop an application with MongoAPI, but
1. You already have relational tables with data.
2. You have views on relational data
3. You need to join tables
4. You need queries with complex expressions. E.g. OLAP window functions.
5. You need to get results from a stored procedure.
6. You need to exploit Informix stored procedure
7. You need federated access to other data
8. You have timeseries data.

Put all of these together!
Hybrid Power
Schemaless SQL
Hybrid access: From SQL API to JSON COllections.
SQL Applications
JDBC connections
Access JSON

Access Relational

SELECT bson_new(bson, “{customer:1}’)
FROM customer WHERE
bson_value_lvarchar(bson,‘state’)= “MO”

Select * from patners where state = “CA”;

Informix Dynamic Server
JSON Collections

Customer

Distributed
Queries

Tables
IDXs
Tables

Relational Tables

Logs

partners

Tables

IDXs

Enterprise replication + Flexible Grid
Join JSON collections
JSON Collection V:
{ "_id" : ObjectId("526a1bb1e1819d4d98a5fa4b"), "c1" : 1, "c2" : 2 }
JSON Collection w:
{ "_id" : ObjectId("526b005cfb9d36501fe31605"), "x" : 255, "y" : 265, "z" : 395}
{ "_id" : ObjectId("52716cdcf278706cb7bf9955"), "x" : 255, "y" : 265, "z" : 395}
{ "_id" : ObjectId("5271c749fa29acaeba572302"), "x" : 1, "y" : 2, "z" : 3 }

SELECT c1, c2, x,y,z
FROM V, W
WHERE V.c1 = W.x;
Join JSON collections
-- Wrap aorund the built-in expressions
SELECT bson_value_int(jc1.data, 'x'),
bson_value_lvarchar(jc1.data, 'y'),
bson_value_int(jc1.data, 'z') ,
bson_value_int(jc2.data, 'c1'),
bson_value_lvarchar(jc2.data, 'c2')
FROM w jc1, v jc2
WHERE bson_value_int(jc1.data, 'x') =
bson_value_int(jc2.data, 'c1');
Join JSON collections
-- Create a view to make the access simple.
create view vwjc(jc1x, jc1y, jc1z, jc2c1, jc2c2) as
SELECT bson_value_int(jc1.data, 'x'),
bson_value_lvarchar(jc1.data, 'y'),
bson_value_int(jc1.data, 'z') ,
bson_value_int(jc2.data, 'c1'),
bson_value_lvarchar(jc2.data, 'c2')
FROM w jc1, v jc2
WHERE bson_value_int(jc1.data, 'x') =
bson_value_int(jc2.data, 'c1');

db.vwjc.find();
Db.vwjc.find({jc1x:100});
Db.vwjc.find({jc1x:{$gte:10}}).sort(jc1y:1)
You want to perform complex analytics on JSON data
BI Tools like Cognos, Tableau generate SQL on data sources.
Option 1: Do ETL
Need to expose JSON data as views so it’s seen as a database
object.
−

We use implicit casting to convert to compatible types

−

The references to non-existent key-value pair returns NULL

Create any combination of views
−

A view per JSON collection

−

Multiple views per JSON collection

−

Views joining JSON collections, relational tables and views.

Use these database objects to create reports, graphs, etc.
Analytics
SQL & BI Applications
ODBC, JDBC connections

Informix Dynamic Server
JSON Collections
Customer

Tables & views

Tables

Tables

Orders

Relational Tables

partners
Inventory
Tables
CRM

Informix Warehouse Accelerator
Hybrid Access between relational & JSON Collections

Relational Table

JSON Collections

SQL API

Standard ODBC, JDBC,
.NET, OData, etc.
Language SQL.

Direct SQL Access.
Dynamic Views
Row types

MongoDB API

Mongo APIs for Java,

(NoSQL)

Javascript, C++, C#,...

Mongo APIs for Java,
Javascript, C++, C#,...
Ability for MongoDB API to Access All Data Models

node.js Application

Traditional SQL

MongoDB Drivers

NoSQL - JSON
NoSQL - JSON
TimeSeries
TimeSeries
MQ Series
MQ Series
MQ Series

130
All three drivers support all data models
Applications
native Client

Informix SQLI Drivers

Traditional SQL
IBM DRDA Drivers

MongoDB Drivers

web
browser

NoSQL - JSON
TimeSeries
MQ Series

Mobile

© 2013 IBM Corporation
Benefits of Hybrid Power
Access consistent data from its source
Avoid ETL, continuous data sync and conflicts.
Exploit the power of SQL, MongoAPI seamlessly
Exploit the power of RDBMS technologies via MongoAPI:

− Informix Warehouse accelerator,
− Cost based Optimizer & power of SQL
− R-tree indices for spatial, Lucene text indexes, and more.
Access all your data thru any interface: MongoAPI & SQL
Store data in one place and efficiently transform and use
them on demand.
Existing SQL based tools and APIs can access new data
in JSON
---------John----------
Simple SQL statements which Operate on the BSON

CREATE TABLE mycollection (data BSON);
INSERT INTO mycollection (data) VALUES ( '{"name":“John"}‘ ::JSON );
CREATE INDEX myindx ON mycollection(bson_extract(data,“name"))
USING BSON
SELECT data::JSON FROM mycollection;

134

© 2013 IBM Corporation
The SQL to Create a Collection
Formal definition of a collection used by the Wire Listener

CREATE COLLECTION TABLE mycollection
(
id
char(128),
modcnt integer,
data
bson,
flags
integer
);

135

© 2013 IBM Corporation
New Built-in BSON Expressions

bson_value_array(lvarchar doc, lvarchar key) returns lvarchar as BSON array
bson_value_bigint(lvarchar doc, lvarchar key) returns bigint
bson_value_binary(lvarchar doc, lvarchar key) returns lvarchar as BSON binary
bson_value_boolean(lvarchar doc, lvarchar key) returns boolean
bson_value_code(lvarchar doc, lvarchar key) returns lvarchar as string
bson_value_date(lvarchar doc, lvarchar key) returns datetime
bson_value_document(lvarchar doc, lvarchar key) returns lvarchar as BSON object
bson_value_double(lvarchar doc, lvarchar key) returns float
bson_value_int(lvarchar doc, lvarchar key) returns bigint
bson_value_lvarchar(lvarchar doc, lvarchar key) returns lvarchar as string
bson_value_objectid(lvarchar doc, lvarchar key) returns lvarchar as string
bson_value_timestamp(lvarchar doc, lvarchar key) returns datetime
bson_keys_exist(lvarchar doc, lvarchar key(s)) returns Boolean
bson_new(lvarchar doc, lvarchar projection_list) return lvarchar as BSON binary
bson_type(lvarchar doc, lvarchar key) return integer
bson_size(lvarchar doc, lvarchar key) return integer
bson_extract(lvarchar doc, lvarchar) return lvarchar as BSON binary

136
Combining Tables and Collections
SELECT customer.*,
bson_value_lvarchar(data,"customer_code") AS customer_code
FROM mycollection, customer
WHERE bson_value_lvarchar(data,"fname") = customer.fname
AND bson_value_lvarchar(data,"lname") = customer.lname
AND customer_num < 102;

Query Optimizer
QUERY: (OPTIMIZATION TIMESTAMP: 07-21-2013 23:15:31)
Estimated Cost: 6
Estimated # of Rows Returned: 10
1) miller3.customer: INDEX PATH
(1) Index Name: miller3. 100_1
Index Keys: customer_num
(Serial, fragments: ALL)
Upper Index Filter: miller3.customer.customer_num < 102
2) miller3.mycollection: INDEX PATH
Filters: informix.equal(BSON_VALUE_LVARCHAR (miller3.mycollection.data , 'fname' ) ,miller3.customer.fname )
(1) Index Name: miller3.mycollection_ix1
Index Keys: :informix.bson_value_lvarchar(data,'lname')
(Serial, fragments: ALL)
Lower Index Filter: equal(BSON_VALUE_LVARCHAR (miller3.mycollection.data , 'lname' ) ,miller3.customer.lname )
NESTED LOOP JOIN

137

© 2013 IBM Corporation
Understanding Informix BSON Indexes
Indexes are created on BSON data and support
–
–
–
–

Primary Key (enforced across all nodes)
Unique Indexes (enforced at a single node level)
Composite Indexes ( 8 parts )
Arrays
{
"fname":"Sadler",
"lname":"Sadler",
"company":"Friends LLC",
"age":21,
"count":27,
“phone": [ “408-789-1234”, “408-111-4779” ],
}

create index fnameix1 on customer(bson_value_lvarchar(bson,"fname")) using bson;
create index lnameix2 on customer(bson_value_lvarchar(bson,"lname")) using bson;
create index phoneix3 on customer(bson_value_lvarchar(bson,"phone")) using bson;

138

© 2013 IBM Corporation
Understanding Informix BSON Indexes
create index fnameix1 on customer(bson_value(bson,"fname")) using bson;
create index lnameix2 on customer(bson_value(bson,"lname")) using bson;
create index phoneix3 on customer(bson_value(bson,"phone")) using bson;

select * from customer where bson_value_lvarchar(bson,"fname") = "Ludwig";
-- use fnameix1

select * from customer where bson_value_lvarchar (bson,"lname") = "Sadler";
-- use lnameix2

select * from customer where bson_value_lvarchar(bson,"phone") = "408-789-8091";
-- use phoneix3

select * from customer where bson_value_lvarchar(bson,"phone") = "415-822-1289“
OR bson_value_lvarchar(bson,"phone") = "408-789-8091";
-- use phoneix3

select * from customer where bson_value_lvarchar(bson,"company") = "Los Altos
Sports";
-- no index use sequential scan
139

© 2013 IBM Corporation
Informix 12.1

Convert Traditional SQL Tables to JSON

Mongo API
–Leverage the numerous open
source drivers for Mongo
– Extend the power of the mongo API to
execute on
• Traditional SQL tables
• TimeSeries/Sensor Data

140

© 2013 IBM Corporation

© 2013 IBM Corporation
Not just a Hybrid Database

Hybrid databases only solve half of the problem
Hybrid application are hear to stay
Scaling Out – Sharding Data
Shard Key
state= “CA”

Shard by either hash or
expression
When inserting data it is
automatically moved to the
correct node

•

142

Each node in the environment
hold a portion of the data

•

Shard Key
state= “OR”

•

•

Shard Key
state= “WA”

Queries automatically
aggregate data from the
required node(s)
Graphical Administration showing a Sharded System
Graphical administration of
the shards keep things
simple
Monitor the entire
environment from a single
graphical console
Simply Administration - OAT Manages JSON Objects

144
Simplify the “up and running” Experience
Install and setup a typical database instance with only 3 questions:
−

Where to place the Product?

−

Where to place the Data?

−

How many users do you anticipate?

Newly installed instance adapts to the resources on the
computer
Description
Auto tuning of CPU VPS
Auto Table Placement
Auto Buffer pool tuning
Auto Physical Log extension
Auto Logical Log Add
Auto Read Ahead
145
SIMPLE POWER –
INFORMIX HYBRID
DATABASE CAPABILITIES
Relational and non-relational data in one system
NoSQL/MongoDB Apps can access Informix Relational Tables
Distributed Queries
Multi-statement Transactions
Enterprise Proven Reliability
Enterprise Ready Security
Enterprise Level Performance

Informix provides the capability to leverage
the abilities of both relational DBMS and document store systems.
MongoDB does not. It is a document store system lacking key
abilities like transaction durability.
147
Round Peg Square Hole

The DBA and/or programmer no longer have to decide upfront if it
is better to use a system entirely of JSON documents or a system
only of SQL tables, but rather have a single database in which the
programmer decides a if JSON document is optimal or a SQL table
is optimal for the data and usage.

148
Informix Specific Advantages with Mongo Drivers
Traditional SQL tables and JSON collections co-existing in the
same database
Using the MongoDB client drivers Query, insert, update, delete
−

JSON collections

−

Traditional SQL tables

−

Timeseries data

Join SQL tables to JSON collections utilizing indexes
Execute business logic in stored procedures
Provide a view of JSON collections as a SQL table
−

Allows existing SQL tools to access JSON data

Enterprises level functionality
149
Scalability
Better performance on multi-core, multi-session scenarios
−

Architecture has finer grain locking – not entire database

−

Better concurrency because less resources locked

Document Compression
−

60% to 90% observed

Bigger documents – 2GB maximum size
−

MongoDB caps at 16MB

Informix has decades of optimization on single node solution

150

© 2013 IBM Corporation
Some NoSQL Use Cases - Mostly Interactive Web/Mobile
Online/Mobile Gaming
−
−
−
−
−

Leaderboard (high score table)
management
Dynamic placement of visual elements
Game object management
Persisting game/user state information
Persisting user generated data (e.g.
drawings)

Display Advertising on Web Sites
−
−

Ad Serving: match content with profile
and present
Real-time bidding: match cookie profile
with ad inventory, obtain bids, and
present ad

Dynamic Content Management
and Publishing (News & Media)
−
−

151

Store content from distributed authors,
with fast retrieval and placement
Manage changing layouts and user
generated content

E-commerce/Social Commerce
– Storing frequently changing product
catalogs

Social Networking/Online
Communities
Communications
– Device provisioning

Logging/message passing
– Drop Copy service in Financial
Services (streaming copies of trade
execution messages into (for
example) a risk or back office
system)
Dual Capability

Hybrid databases are a way of the past
Hybrid application are hear to stay

152
Client Applications
Applications
native Client

All three drivers support all data models

Informix SQLI
Drivers
IBM DRDA
Drivers
MongoDB Drivers

web browser

Informix
NoSQL

Mobile

© 2013 IBM Corporation
Ability for All Clients to Access All Data Models

Informix SQLI
Drivers
IBM DRDA
Drivers
MongoDB Drivers

Traditional SQL

NoSQL - JSON
TimeSeries
MQ Series

154
Ability for All Clients to Access All Data Models

node.js Application

Traditional SQL

MongoDB Drivers

NoSQL - JSON
TimeSeries
MQ Series

155
Lance Code Snippet
BUILDING A REAL LIFE
APPLICATION
IOD Demo and Beyond
A fun exciting demo application that attendees
can interact with that makes people take notice
of IBM technologies
A social networking photo application utilizing
smart phones/tablets that allows the end user to
take photos, tag and add photos to an Informix
NoSQL hybrid database
Random photos can be displayed at sessions,
booths or on attendee smart devices
IOD Attendee Photo Application
Allow conference attendee to take and share photo!
Technology Highlights
•

Create a hybrid application using NoSQL, traditional SQL,
timeseries mobile web application
•
•
•

Utilizing both JSON collections, SQL tables and timeseries
Utilize IBM Dojo Mobile tools to build a mobile application
Leverage new mongo client side drivers for fast application
delvelopment

•

Demonstrate sharding with over 100 nodes

•

Cloud based solution on top of Amazon Cloud

•

Provide real-time analytics on all forms of data
•
•

Leverage existing popular analytic front-end IBM-Congos
Utilize an in-memory columnar database accelerator to
provide real-time trending analytics on data
Mobile Device Application Architecture

Apache Web Server
IOD Photo App - UPLOAD

Photo Application
IBM Dojo Mobile
Informix
tag

Van Gogh

Informix JSON
Listener

Photo
collection

User
Table
Application Architecture (Big Picture)
Apache web server
IOD application

Informix JSON Listener

Informix

Photo collection

Amazon Cloud

Sharding
commands

Node1
Aa-Am

Node2
An-Az

Node3
Ba-Bf

Node3
Bg-Bp

Node101

“Dalmation” shard farm – 101 sharded nodes on Amazon Cloud
Photo collection sharded across these nodes
Photo Application Schema
TimeSeries
NoSQL Collections
activity_photos

activity_data

timeseries(photo_like)
Photo Application NoSQL Collections

activity_photos

activity_data

timeseries(photo_like)
Application Considerations

Photo meta-data varies from camera to camera
A Picture and all its meta data are stored in-document
Pictures are stored in a NoSQL collection
Pre-processing on the phone ensures only reasonable size
photos are sent over the network.
Example of Live JSON Photo Data

{"_id":ObjectId("526157c8112c2fe70cc06a75"), "Make":"NIKON CORPORA
TION","Model":"NIKON D60","Orientation":"1","XResolution":"300","
YResolution":"300","ResolutionUnit":"2","Software":"Ver.1.00 ","D
ateTime":"2013:05:15 19:46:36","YCbCrPositioning":"2","ExifIFDPoi
nter":"216","ExposureTime":"0.005","FNumber":"7.1","ExposureProgr
am":"Not defined","ISOSpeedRatings":"100",
"Contrast":"Normal","Saturation":"Normal","Sharpness":"Normal",
"SubjectDistanceRange":"Unknown","name":"DSC_0078.JPG","img_d
ata":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABcQ
Sharding

NoSQL “Dalmatian” cloud deployment
Sharding by hash
Heterogeneous Cloud deployment with sharding
−

Processor Type and Number

Intel and ARM processors
−

Operating System Types and Versions

Linux, Windows, AIX, Apple, Solaris, HP,
Basic PHP Programming Overview Information
List of NoSQL collection names and SQL tables names
Function to set the active database and return the Collection

private $conn;
private
private
private
private
private
private
private

$dbname = "photo_demo";
$photoCollectionName = "photos";
$contactsCollectionName = "contacts";
$sqlCollectionName = 'system.sql';
$userTableName = "users";
$tagsTableName = "tags";
$likesTableName = "likes";

private $photoQueryProjection = array("_id" => 1, "tags" => 1,
"user_id" => 1, "img_data" => 1);
/**
* Get collection by name
* @param MongoCollection $collectionName
*/
private function getCollection($collectionName) {
return $this->conn->selectDB($this->dbname)->selectCollection ($collectionName);
}
Insert Example

Information is placed
in the contacts
collection
Insert Data into a Collection
Very simple to insert JSON data into a collection using the
MongoAPIs

/**
* Insert user's contact information into contacts table.
*/
public function insertContact( $json ) {
if (! is_array ( $json )) {
return "Contact info not in JSON format.";
}
try {
$result = $this->getCollection($this->contactsTableName)->insert($json);
if ($result ["ok"] != 1) {
return $result ["err"];
}
} catch ( MongoException $e ) {
return $e->getMessage ();
}
return "ok";
}
Retrieve Collection Information
Very simple to retrieve data from a collection using the
MongoAPIs
Data is returned as a JSON document

/**
* Get contact info
*/
public function adminContacts() {
$photoCollection = $this->getCollection($this->contactsCollectionName);
$cursor = $photoCollection->find();
$results = $this->getQueryResults($cursor);
return $results;
}
Retrieve Collection Information

/**
* Get contact info for admin panel
*/
public function adminContacts() {
$photoCollection = $this->getCollection($this->contactsCollectionName);
$cursor = $photoCollection->find();
$results = $this->getQueryResults($cursor);
return $results;
}
Naughty Pictures
Delete a Photo and its Information
Deleting from SQL Tables and NoSQL Collection is exactly
the same
/**
* Delete photo
*/
public function deletePhoto($id) {
try {
// First delete from likes and tags tables
$query = array('photo_id' => $id['_id']);
$result = $this->getCollection($this->likesTableName)->remove($query);
if ($result ["ok"] != 1) {
return $result["err"];
}
$result = $this->getCollection($this->tagsTableName)->remove($query);
if ($result ["ok"] != 1) {
return $result["err"];
}
// Then delete the photo from the collection
$query = array('_id' => new MongoId($id['_id']));
$result = $this->getCollection ( $this->photoCollectionName )->remove ( $query );
if ($result ["ok"] != 1) {
return $result["err"];
}
} catch ( MongoException $e ) {
return $e->getMessage();
}
return "ok";
}
Executing a Stored Procedure in MongoAPI

/**
* Get the user_id for a particular user name (email address).
*
* Calls a stored procedure that will insert into the users table if the
* user does not exist yet and returns the user_id.
*
* @param string $username
* @return int $user_id
*/
public function getUserId($username) {
$username = trim($username);
try {
$sql = "EXECUTE FUNCTION getUserID('" . $username . "')";
$result = $this->getCollection($this->sqlCollectionName)->
findOne(array('$sql'=>$sql));
if (isset($result['errmsg'])) {
return "ERROR. " . $result['errmsg'];
}
return $result['user_id'];
} catch (MongoException $e) {
return "ERROR. " . $e->getMessage();
}
}
Real Time Analytics
Customer Issues
−

Several different models of data (SQL, NoSQL,
TimeSeries/Sensor)

−

NoSQL is not strong building relations between collections

−

Most valuable analytics combine the results of all data models

−

Most protonate Analytic system written using standard SQL

−

ETL & YAS (Yet Another System)

Solution
Provide a mapping of the required data in SQL form
−

Enables common tools like Congo's
Analytics on a Hybrid Database

Photo
collection

SQL

MongoAPI

Informix

User
Table
Photo Application
SQL Mapping of NoSQL PHOTO Collection

activity_photos

activity_data

timeseries(photo_like)
Mapping A Collection To A SQL Table

CREATE VIEW photo_metadata (gpslatitude, gpslongitude,
make, model, orientation, datetimeoriginal,
exposuretime, fnumber, isospeedratings,
pixelxdimension, pixelydimension)
AS SELECT BSON_VALUE_LVARCHAR ( x0.data , 'GPSLatitude' ),
BSON_VALUE_LVARCHAR ( x0.data , 'GPSLongitude' ),
BSON_VALUE_LVARCHAR ( x0.data , 'Make' ),
BSON_VALUE_LVARCHAR ( x0.data , 'Model' ),
BSON_VALUE_LVARCHAR ( x0.data , 'Orientation' ),
BSON_VALUE_LVARCHAR ( x0.data , 'DateTimeOriginal' ) ,
BSON_VALUE_LVARCHAR ( x0.data , 'ExposureTime'),
BSON_VALUE_LVARCHAR ( x0.data , 'FNumber' ),
BSON_VALUE_LVARCHAR ( x0.data , 'ISOSpeedRatings' ),
BSON_VALUE_LVARCHAR ( x0.data , 'PixelXDimension' ) ,
BSON_VALUE_LVARCHAR ( x0.data , 'PixelYDimension')
FROM photos x0;
IWA Integration

What is trending at IOD?
Set up to work on frequently updated snapshots of data to
perform analytic queries on the photo meta-data
Since BSON/JSON data can not be directly compressed
and searched, snapshot process would selectively extract
data from the documents for copying to IWA
Use Genero to show the power of IWA
Mobile Device Application Architecture

Photo
collection

SQL

MongoAPI

Informix

User
Table
Configure Informix on Amazon Cloud Simple

•
•
•
•
•

Instantiate the Amazon image
Setup the storage
Install the product
Start the system
Configure sharding
Informix NoSQL & Hybrid SQL detailed deep dive
Informix NoSQL & Hybrid SQL detailed deep dive
Questions

Contenu connexe

Tendances

Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Databricks
 
Sql server performance tuning
Sql server performance tuningSql server performance tuning
Sql server performance tuningngupt28
 
The Data Warehouse Lifecycle
The Data Warehouse LifecycleThe Data Warehouse Lifecycle
The Data Warehouse Lifecyclebartlowe
 
Big data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseBig data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseMartin Toshev
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesCarole Gunst
 
Building an open data platform with apache iceberg
Building an open data platform with apache icebergBuilding an open data platform with apache iceberg
Building an open data platform with apache icebergAlluxio, Inc.
 
[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google Cloud[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google CloudPgDay.Seoul
 
Data Warehouse Project
Data Warehouse ProjectData Warehouse Project
Data Warehouse ProjectSunny U Okoro
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMark Kromer
 
MySQL Cluster 8.0 tutorial
MySQL Cluster 8.0 tutorialMySQL Cluster 8.0 tutorial
MySQL Cluster 8.0 tutorialFrazer Clement
 
A Survey on Big Data Analytics
A Survey on Big Data AnalyticsA Survey on Big Data Analytics
A Survey on Big Data AnalyticsBHARATH KUMAR
 
Welcome To The 2016 Query Store!
Welcome To The 2016 Query Store!Welcome To The 2016 Query Store!
Welcome To The 2016 Query Store!SolarWinds
 
FOSDEM MySQL & Friends Devroom, February 2018 MySQL Point-in-Time Recovery l...
FOSDEM MySQL & Friends Devroom, February 2018  MySQL Point-in-Time Recovery l...FOSDEM MySQL & Friends Devroom, February 2018  MySQL Point-in-Time Recovery l...
FOSDEM MySQL & Friends Devroom, February 2018 MySQL Point-in-Time Recovery l...Frederic Descamps
 
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
High Performance Data Lake with Apache Hudi and Alluxio at T3GoHigh Performance Data Lake with Apache Hudi and Alluxio at T3Go
High Performance Data Lake with Apache Hudi and Alluxio at T3GoAlluxio, Inc.
 
NoSQL Databases
NoSQL DatabasesNoSQL Databases
NoSQL DatabasesBADR
 
Mastering PostgreSQL Administration
Mastering PostgreSQL AdministrationMastering PostgreSQL Administration
Mastering PostgreSQL AdministrationEDB
 
All of the Performance Tuning Features in Oracle SQL Developer
All of the Performance Tuning Features in Oracle SQL DeveloperAll of the Performance Tuning Features in Oracle SQL Developer
All of the Performance Tuning Features in Oracle SQL DeveloperJeff Smith
 
اسلايد آموزشي هوش تجاري
اسلايد آموزشي هوش تجارياسلايد آموزشي هوش تجاري
اسلايد آموزشي هوش تجاريAli Masoombeigi
 

Tendances (20)

Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Sql server performance tuning
Sql server performance tuningSql server performance tuning
Sql server performance tuning
 
The Data Warehouse Lifecycle
The Data Warehouse LifecycleThe Data Warehouse Lifecycle
The Data Warehouse Lifecycle
 
Dimensional Modelling
Dimensional ModellingDimensional Modelling
Dimensional Modelling
 
Big data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseBig data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle Database
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data Pipelines
 
Building an open data platform with apache iceberg
Building an open data platform with apache icebergBuilding an open data platform with apache iceberg
Building an open data platform with apache iceberg
 
[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google Cloud[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google Cloud
 
Data Warehouse Project
Data Warehouse ProjectData Warehouse Project
Data Warehouse Project
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data Analytics
 
MySQL Cluster 8.0 tutorial
MySQL Cluster 8.0 tutorialMySQL Cluster 8.0 tutorial
MySQL Cluster 8.0 tutorial
 
A Survey on Big Data Analytics
A Survey on Big Data AnalyticsA Survey on Big Data Analytics
A Survey on Big Data Analytics
 
Welcome To The 2016 Query Store!
Welcome To The 2016 Query Store!Welcome To The 2016 Query Store!
Welcome To The 2016 Query Store!
 
FOSDEM MySQL & Friends Devroom, February 2018 MySQL Point-in-Time Recovery l...
FOSDEM MySQL & Friends Devroom, February 2018  MySQL Point-in-Time Recovery l...FOSDEM MySQL & Friends Devroom, February 2018  MySQL Point-in-Time Recovery l...
FOSDEM MySQL & Friends Devroom, February 2018 MySQL Point-in-Time Recovery l...
 
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
High Performance Data Lake with Apache Hudi and Alluxio at T3GoHigh Performance Data Lake with Apache Hudi and Alluxio at T3Go
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
 
NoSQL Databases
NoSQL DatabasesNoSQL Databases
NoSQL Databases
 
Mastering PostgreSQL Administration
Mastering PostgreSQL AdministrationMastering PostgreSQL Administration
Mastering PostgreSQL Administration
 
Data mesh
Data meshData mesh
Data mesh
 
All of the Performance Tuning Features in Oracle SQL Developer
All of the Performance Tuning Features in Oracle SQL DeveloperAll of the Performance Tuning Features in Oracle SQL Developer
All of the Performance Tuning Features in Oracle SQL Developer
 
اسلايد آموزشي هوش تجاري
اسلايد آموزشي هوش تجارياسلايد آموزشي هوش تجاري
اسلايد آموزشي هوش تجاري
 

Similaire à Informix NoSQL & Hybrid SQL detailed deep dive

IMS capabilities today
IMS capabilities todayIMS capabilities today
IMS capabilities todayKyle Charlet
 
Schnellere Digitalisierung mit einer cloudbasierten Datenstrategie
Schnellere Digitalisierung mit einer cloudbasierten DatenstrategieSchnellere Digitalisierung mit einer cloudbasierten Datenstrategie
Schnellere Digitalisierung mit einer cloudbasierten DatenstrategieMongoDB
 
NoSQL support in Informix (JSON storage, Mongo DB API)
NoSQL support in Informix (JSON storage, Mongo DB API)NoSQL support in Informix (JSON storage, Mongo DB API)
NoSQL support in Informix (JSON storage, Mongo DB API)Keshav Murthy
 
IMS integration 2017
IMS integration 2017IMS integration 2017
IMS integration 2017Helene Lyon
 
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...darwinodb
 
Analyst Webinar: Enabling a Customer Data Platform Using Data Virtualization
Analyst Webinar: Enabling a Customer Data Platform Using Data VirtualizationAnalyst Webinar: Enabling a Customer Data Platform Using Data Virtualization
Analyst Webinar: Enabling a Customer Data Platform Using Data VirtualizationDenodo
 
La creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBLa creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBMongoDB
 
Machine Learning for z/OS
Machine Learning for z/OSMachine Learning for z/OS
Machine Learning for z/OSCuneyt Goksu
 
Introduction to WebSphere Message Broker
Introduction to WebSphere Message BrokerIntroduction to WebSphere Message Broker
Introduction to WebSphere Message BrokerAnt Phillips
 
AnDevCon: Introduction to Darwino
AnDevCon: Introduction to DarwinoAnDevCon: Introduction to Darwino
AnDevCon: Introduction to DarwinoPhilippe Riand
 
The New Basics of Business Intelligence Lesson 5: Embedded Analytics
The New Basics of Business Intelligence Lesson 5: Embedded AnalyticsThe New Basics of Business Intelligence Lesson 5: Embedded Analytics
The New Basics of Business Intelligence Lesson 5: Embedded AnalyticsZoomdata
 
OC Big Data Monthly Meetup #6 - Session 1 - IBM
OC Big Data Monthly Meetup #6 - Session 1 - IBMOC Big Data Monthly Meetup #6 - Session 1 - IBM
OC Big Data Monthly Meetup #6 - Session 1 - IBMBig Data Joe™ Rossi
 
SD Big Data Monthly Meetup #4 - Session 1 - IBM
SD Big Data Monthly Meetup #4 - Session 1 - IBMSD Big Data Monthly Meetup #4 - Session 1 - IBM
SD Big Data Monthly Meetup #4 - Session 1 - IBMBig Data Joe™ Rossi
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Denodo
 
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...SoftServe
 
Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyMongoDB
 
Security Policies For Schema Less Or Dynamic Schema...
Security Policies For Schema Less Or Dynamic Schema...Security Policies For Schema Less Or Dynamic Schema...
Security Policies For Schema Less Or Dynamic Schema...Christina Boetel
 

Similaire à Informix NoSQL & Hybrid SQL detailed deep dive (20)

IMS capabilities today
IMS capabilities todayIMS capabilities today
IMS capabilities today
 
Schnellere Digitalisierung mit einer cloudbasierten Datenstrategie
Schnellere Digitalisierung mit einer cloudbasierten DatenstrategieSchnellere Digitalisierung mit einer cloudbasierten Datenstrategie
Schnellere Digitalisierung mit einer cloudbasierten Datenstrategie
 
NoSQL support in Informix (JSON storage, Mongo DB API)
NoSQL support in Informix (JSON storage, Mongo DB API)NoSQL support in Informix (JSON storage, Mongo DB API)
NoSQL support in Informix (JSON storage, Mongo DB API)
 
IMS integration 2017
IMS integration 2017IMS integration 2017
IMS integration 2017
 
VendorReview_IBMDB2
VendorReview_IBMDB2VendorReview_IBMDB2
VendorReview_IBMDB2
 
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...
IBM ConnectED SPOT104: Lightning-Fast Development of Native Mobile Apps for I...
 
Analyst Webinar: Enabling a Customer Data Platform Using Data Virtualization
Analyst Webinar: Enabling a Customer Data Platform Using Data VirtualizationAnalyst Webinar: Enabling a Customer Data Platform Using Data Virtualization
Analyst Webinar: Enabling a Customer Data Platform Using Data Virtualization
 
La creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDBLa creación de una capa operacional con MongoDB
La creación de una capa operacional con MongoDB
 
Machine Learning for z/OS
Machine Learning for z/OSMachine Learning for z/OS
Machine Learning for z/OS
 
Introduction to WebSphere Message Broker
Introduction to WebSphere Message BrokerIntroduction to WebSphere Message Broker
Introduction to WebSphere Message Broker
 
AnDevCon: Introduction to Darwino
AnDevCon: Introduction to DarwinoAnDevCon: Introduction to Darwino
AnDevCon: Introduction to Darwino
 
Sivagama_sundari_Sakthivel_Resume_2016
Sivagama_sundari_Sakthivel_Resume_2016Sivagama_sundari_Sakthivel_Resume_2016
Sivagama_sundari_Sakthivel_Resume_2016
 
The New Basics of Business Intelligence Lesson 5: Embedded Analytics
The New Basics of Business Intelligence Lesson 5: Embedded AnalyticsThe New Basics of Business Intelligence Lesson 5: Embedded Analytics
The New Basics of Business Intelligence Lesson 5: Embedded Analytics
 
OC Big Data Monthly Meetup #6 - Session 1 - IBM
OC Big Data Monthly Meetup #6 - Session 1 - IBMOC Big Data Monthly Meetup #6 - Session 1 - IBM
OC Big Data Monthly Meetup #6 - Session 1 - IBM
 
SD Big Data Monthly Meetup #4 - Session 1 - IBM
SD Big Data Monthly Meetup #4 - Session 1 - IBMSD Big Data Monthly Meetup #4 - Session 1 - IBM
SD Big Data Monthly Meetup #4 - Session 1 - IBM
 
Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)Data Virtualization: Introduction and Business Value (UK)
Data Virtualization: Introduction and Business Value (UK)
 
Symphony Driver Essay
Symphony Driver EssaySymphony Driver Essay
Symphony Driver Essay
 
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...
From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...
 
Accelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data StrategyAccelerating a Path to Digital with a Cloud Data Strategy
Accelerating a Path to Digital with a Cloud Data Strategy
 
Security Policies For Schema Less Or Dynamic Schema...
Security Policies For Schema Less Or Dynamic Schema...Security Policies For Schema Less Or Dynamic Schema...
Security Policies For Schema Less Or Dynamic Schema...
 

Plus de Keshav Murthy

N1QL New Features in couchbase 7.0
N1QL New Features in couchbase 7.0N1QL New Features in couchbase 7.0
N1QL New Features in couchbase 7.0Keshav Murthy
 
Couchbase Tutorial: Big data Open Source Systems: VLDB2018
Couchbase Tutorial: Big data Open Source Systems: VLDB2018Couchbase Tutorial: Big data Open Source Systems: VLDB2018
Couchbase Tutorial: Big data Open Source Systems: VLDB2018Keshav Murthy
 
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5Keshav Murthy
 
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...Keshav Murthy
 
Couchbase 5.5: N1QL and Indexing features
Couchbase 5.5: N1QL and Indexing featuresCouchbase 5.5: N1QL and Indexing features
Couchbase 5.5: N1QL and Indexing featuresKeshav Murthy
 
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram Vemulapalli
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram VemulapalliN1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram Vemulapalli
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram VemulapalliKeshav Murthy
 
Couchbase N1QL: Language & Architecture Overview.
Couchbase N1QL: Language & Architecture Overview.Couchbase N1QL: Language & Architecture Overview.
Couchbase N1QL: Language & Architecture Overview.Keshav Murthy
 
Couchbase Query Workbench Enhancements By Eben Haber
Couchbase Query Workbench Enhancements  By Eben Haber Couchbase Query Workbench Enhancements  By Eben Haber
Couchbase Query Workbench Enhancements By Eben Haber Keshav Murthy
 
Mindmap: Oracle to Couchbase for developers
Mindmap: Oracle to Couchbase for developersMindmap: Oracle to Couchbase for developers
Mindmap: Oracle to Couchbase for developersKeshav Murthy
 
Couchbase N1QL: Index Advisor
Couchbase N1QL: Index AdvisorCouchbase N1QL: Index Advisor
Couchbase N1QL: Index AdvisorKeshav Murthy
 
N1QL: What's new in Couchbase 5.0
N1QL: What's new in Couchbase 5.0N1QL: What's new in Couchbase 5.0
N1QL: What's new in Couchbase 5.0Keshav Murthy
 
From SQL to NoSQL: Structured Querying for JSON
From SQL to NoSQL: Structured Querying for JSONFrom SQL to NoSQL: Structured Querying for JSON
From SQL to NoSQL: Structured Querying for JSONKeshav Murthy
 
Tuning for Performance: indexes & Queries
Tuning for Performance: indexes & QueriesTuning for Performance: indexes & Queries
Tuning for Performance: indexes & QueriesKeshav Murthy
 
Understanding N1QL Optimizer to Tune Queries
Understanding N1QL Optimizer to Tune QueriesUnderstanding N1QL Optimizer to Tune Queries
Understanding N1QL Optimizer to Tune QueriesKeshav Murthy
 
Utilizing Arrays: Modeling, Querying and Indexing
Utilizing Arrays: Modeling, Querying and IndexingUtilizing Arrays: Modeling, Querying and Indexing
Utilizing Arrays: Modeling, Querying and IndexingKeshav Murthy
 
Extended JOIN in Couchbase Server 4.5
Extended JOIN in Couchbase Server 4.5Extended JOIN in Couchbase Server 4.5
Extended JOIN in Couchbase Server 4.5Keshav Murthy
 
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQL
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQLBringing SQL to NoSQL: Rich, Declarative Query for NoSQL
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQLKeshav Murthy
 
Query in Couchbase. N1QL: SQL for JSON
Query in Couchbase.  N1QL: SQL for JSONQuery in Couchbase.  N1QL: SQL for JSON
Query in Couchbase. N1QL: SQL for JSONKeshav Murthy
 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications Keshav Murthy
 
Introducing N1QL: New SQL Based Query Language for JSON
Introducing N1QL: New SQL Based Query Language for JSONIntroducing N1QL: New SQL Based Query Language for JSON
Introducing N1QL: New SQL Based Query Language for JSONKeshav Murthy
 

Plus de Keshav Murthy (20)

N1QL New Features in couchbase 7.0
N1QL New Features in couchbase 7.0N1QL New Features in couchbase 7.0
N1QL New Features in couchbase 7.0
 
Couchbase Tutorial: Big data Open Source Systems: VLDB2018
Couchbase Tutorial: Big data Open Source Systems: VLDB2018Couchbase Tutorial: Big data Open Source Systems: VLDB2018
Couchbase Tutorial: Big data Open Source Systems: VLDB2018
 
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5
 
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...
 
Couchbase 5.5: N1QL and Indexing features
Couchbase 5.5: N1QL and Indexing featuresCouchbase 5.5: N1QL and Indexing features
Couchbase 5.5: N1QL and Indexing features
 
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram Vemulapalli
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram VemulapalliN1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram Vemulapalli
N1QL: Query Optimizer Improvements in Couchbase 5.0. By, Sitaram Vemulapalli
 
Couchbase N1QL: Language & Architecture Overview.
Couchbase N1QL: Language & Architecture Overview.Couchbase N1QL: Language & Architecture Overview.
Couchbase N1QL: Language & Architecture Overview.
 
Couchbase Query Workbench Enhancements By Eben Haber
Couchbase Query Workbench Enhancements  By Eben Haber Couchbase Query Workbench Enhancements  By Eben Haber
Couchbase Query Workbench Enhancements By Eben Haber
 
Mindmap: Oracle to Couchbase for developers
Mindmap: Oracle to Couchbase for developersMindmap: Oracle to Couchbase for developers
Mindmap: Oracle to Couchbase for developers
 
Couchbase N1QL: Index Advisor
Couchbase N1QL: Index AdvisorCouchbase N1QL: Index Advisor
Couchbase N1QL: Index Advisor
 
N1QL: What's new in Couchbase 5.0
N1QL: What's new in Couchbase 5.0N1QL: What's new in Couchbase 5.0
N1QL: What's new in Couchbase 5.0
 
From SQL to NoSQL: Structured Querying for JSON
From SQL to NoSQL: Structured Querying for JSONFrom SQL to NoSQL: Structured Querying for JSON
From SQL to NoSQL: Structured Querying for JSON
 
Tuning for Performance: indexes & Queries
Tuning for Performance: indexes & QueriesTuning for Performance: indexes & Queries
Tuning for Performance: indexes & Queries
 
Understanding N1QL Optimizer to Tune Queries
Understanding N1QL Optimizer to Tune QueriesUnderstanding N1QL Optimizer to Tune Queries
Understanding N1QL Optimizer to Tune Queries
 
Utilizing Arrays: Modeling, Querying and Indexing
Utilizing Arrays: Modeling, Querying and IndexingUtilizing Arrays: Modeling, Querying and Indexing
Utilizing Arrays: Modeling, Querying and Indexing
 
Extended JOIN in Couchbase Server 4.5
Extended JOIN in Couchbase Server 4.5Extended JOIN in Couchbase Server 4.5
Extended JOIN in Couchbase Server 4.5
 
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQL
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQLBringing SQL to NoSQL: Rich, Declarative Query for NoSQL
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQL
 
Query in Couchbase. N1QL: SQL for JSON
Query in Couchbase.  N1QL: SQL for JSONQuery in Couchbase.  N1QL: SQL for JSON
Query in Couchbase. N1QL: SQL for JSON
 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications 
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications 
 
Introducing N1QL: New SQL Based Query Language for JSON
Introducing N1QL: New SQL Based Query Language for JSONIntroducing N1QL: New SQL Based Query Language for JSON
Introducing N1QL: New SQL Based Query Language for JSON
 

Dernier

IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 

Dernier (20)

IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 

Informix NoSQL & Hybrid SQL detailed deep dive

  • 1. Informix NoSQL -- Very Deep Dive John F. Miller III, IBM miller3@us.ibm.com Keshava Murthy, IBM rkeshav@us.ibm.com © 2013 IBM Corporation
  • 2. Speaker Introduction John F. Miller III Lead Architect, STSM Keshava Murthy NoSQL & SQL Architect, STSM Jef Treece Product Manager
  • 3. Overview Business phenomenon of NoSQL − Value: Open Source vs Enterprise Class What is NoSQL, JSON and BSON? − Quick overview − IBM/Mongo Partnership New Informix NoSQL/JSON Capabilities − Solution Highlights Mongo Compatibility − The Power of a Hybrid Solution − Analytics Demo Deep Dive − − 3 Putting together an application Seeing it live NewSQL
  • 4. BUSINESS PHENOMENON OF NOSQL © 2013 2013Corporation © IBM IBM Corporation
  • 5. Informix NoSQL Capabilities Key-value Stores – Hashed Key value model – Capable of massive data storage Column Family Stores – Keys that point to multiple columns and column families Document Data Base – JSON / BSON Formatted documents – Collections of Documents with Key-value stored versions Graph Databases – – – – Nodes / relationship between nodes Properties of nodes Data values Supports SPARQL ? © 2013 IBM Corporation
  • 6. Business Value of Informix NoSQL Increases business opportunity through capture and analysis of “interactive / interaction data” for a full understanding of customers and markets Lowers business barriers and reduces IT costs through Rapid Application Deployment Increases business ability to respond to volume changes through Scalable / extensible architecture Lowers IT overhead through use of Low Cost Commodity Hardware Changes business view to “what is happening” versus “what happened” © 2013 IBM Corporation
  • 7. Use Cases for Informix NoSQL Session Store – High volume ingestion / low latency access requirements – Linkage to a specific user provides immediate recognition of customer preferences – Session restore after “break” is immediate to last functions performed increase customer confidence and connection to their desire(s) User Profile Store – Customer Profiles, orders and shipment status is immediate and searchable – Fast access for Authentication and preferences – Historical access and click streams increase personalization & targeting Content and Metadata Store – Document database for digital content, articles, instruction manuals – Key-value linkages provide linkage to customer profiles and history – Multiple image/data type support provide fast access to different data type © 2013 IBM Corporation
  • 8. Use Case for Informix NoSQL (cont) Mobile Apps – Ability to store content and user data in schema-less formats increase development and deployment speed and changes to existing apps – Supports multi-format data storage associated with differing device types – Scalable storage provides for document, image, text oriented storage and retrieval of apps data / information Third Party Aggregation – High speed ingestion of data feed from multiple sources: • Retail store loyalty programs, social media, marketing information, purchase histories by groups, person, industries – Ease of format management and analysis High Availability Cache – – – – Storage for popular / frequent search results Session information High Frequency Ads tailored for User Profiles, Locations, searches Dual function cache for data store and fast response time for retrieval © 2013 IBM Corporation
  • 9. Informix NoSQL Use Case (cont) Globally Distributed Data Repository – Scalable nodes and access across distributed systems – Location affinity for workload optimization – Capture of differing formats and languages to provide global and discrete views of the business and analytics of searches, purchases, etc. eCommerce – Ability to handle purchase spikes / peak time through scalable system architecture and low cost commodity hardware – Fast / rapid application deployment through schema-less development model – Ability to combine transaction data with user action reduces cost of upgrading business apps for transaction management Social Media & Gaming – Rapid app development , deployment, and change implementation increase ability to grow customer base and respond to trends – or create them – Fast access to user profile for authentication and preferences / historical information – Real time Analytics and trend identification © 2013 IBM Corporation
  • 10. Informix NoSQL Use Cases (cont) Ad Targeting – Fast access to user profile and histories permit well managed Ad placement increasing revenue and buy decision – Click history and Pyschographic based suggestions – Ability to ingest other feeds and quickly relate to a specific customer • Loyalty programs • Social Media • Associations (LinkedIn, Dating sites, Industry Groups etc.) © 2013 IBM Corporation
  • 11. Informix and MongoDB Have Free Editions Editions MongoDB Free Developer Innovator-C Basic For Purchase 11 Informix Express, Workgroup, Advanced Workgroup, Enterprise, Advanced Enterprise Standard, Enterprise
  • 12. MongoDB Subscriptions Basic Standard Enterprise Edition MongoDB MongoDB MongoDB Enterprise Support 9am-9pm local, M-F 24x7x365 24x7x365 License AGPL Commercial Commercial Emergency Patches Not Included Included Included Price $2,500 / Server / Year $5,000 / Server / Year $7,500 / Server / Year Additional monthly charges for backup services. Subscription information obtained from 10Gen site, June 26, 2013. 12
  • 13. Price Point Comparison Estimate, 3-year cost Dual Core Intel Nehalem Innovator-C Express Workgroup (4 core, 8 GB, 2 nodes) (16 core, 16 GB, unlimited nodes) Product Cost $0 $8,540 $19,740 $1,680 Included Included Support Renewal Year 2 $1,680 $1,708 $3,948 Support Renewal Year 3 $1,680 $1,708 $3,948 Total $5,040 $11,956 $27,636 Support Subscription Year 1 24 x 7 x 365 Production System Down Development Call Emergency Patches Free Upgrades MongoDB Enterprise, 3-year cost: $22,500 Subscription information obtained from 10Gen site, June 26, 2013. Retail U.S. prices subject to change, valid as of June 26, 2013. 13
  • 14. Back up © 2013 IBM Corporation
  • 15. Informix NoSQL Key Messages Mobile computing represents one of the greatest opportunities for organizations to expand their business, requiring mobile apps to have access to all relevant data DB2 and Informix have embraced the JSON standard and MongoDB API to give mobile app developers the ability to combine critical data managed by DB2 or Informix with the next generation mobile apps DB2 JSON and Informix JSON let developers leverage the abilities of relational DBMS together with document store systems 15 IBM Confidential © 2013 IBM Corporation
  • 16. DB2/Informix JSON for Install Base Clients Leverage all relevant enterprise data for your mobile app development The Offer Client Value Any for-purchase edition of One solution for relational and non-relational data • Latest version of DB2 LUW 10.5 • Latest version of Informix 12.1 • Relational tables and NoSQL collections co-exist in the same database Includes modern interface with JSON /BSON support • Allows joins between NoSQL and Relational tables • Lets DB2/Informix be compatible with existing MongoDB apps with minimal/no application changes • Ability to handle a mix of structured and unstructured data • Joins utilize indexes on both relational and NoSQL Brings Enterprise level performance, reliability, and security to mobile apps • Includes ACID compliance Includes built-in online backup and restore capabilities • Flexible schema allows rapid delivery of apps • Compatible with MongoDB programming interfaces Automatic Document Compression (only if they buy Advanced Enterprise or add-on compression with Enterprise Edition?) 16 IBM Confidential © 2013 IBM Corporation
  • 17. DB2 JSON Go-to-Market Timeline Phase 1 – June through August 15 Phase 2 – eGA – Q4 Timing Target Phase 3 - 2014 Jan 1, 2014 – TBD Targeted clients/partners Opportunistic white space Install Base – Top xxxx accounts Key partners TBD Opportunistic white space DB2 Trial code with Tech Preview Content Tech preview on dW A series of how-to articles on dW: Part 1, Part 2, Part 3, Part 4. Video: DB2 JSON Overview Video: Getting started Update NoSQL product page with JSON content Re-promote DB2 Trial code IBM Champions for blogs dW DM for JSON Community covreing both DB2 and Informix Who Deliverables Events 17 MongoDB World Tech Bits and Tech Talks Developer Days DB2 Night Show via BP Leverage existing 10Gen event calendar to insert content (these are potential – not confirmed) IBM Confidential On-going Connect with AIM, Rational for joint presence at 3rd conferences and events. © 2013 IBM Corporation
  • 18. Informix JSON Go-to-Market Timeline Phase 1 - EVP Phase 2 – eGA – Q4 Phase 3 - 2014 Timing Aug. 5, 2013 – Sep. 13, 2013 Sep. 13, 2013 – Dec. 31, 2013 Jan 1, 2014 – TBD Target • • • • • Who Confirmed EVP participants Begooden BP Moukouri BP Consult-IX BP ADT BP Openbet Cust Technicolor White space cust Targeted clients/partners Opportunistic white space Install Base – Top xxxx accounts Key partners TBD Opportunistic white space Top xxxx accounts Key partners TBD (criteria to be determined for each) TBD Potential EVP participants Nuvento White space cust SRCE BP Cisco OEM cust Zebra Fish BP Deliverables Events 18 NoSQL Client Business Value Deck NoSQLTechnical White-paper 12.10.xC2 Release White Paper Developer Works article on NoSQL Client Updated NoSQL demo • • Content on Informix marketing page with re-direct to DevWorks page Promote Informix trial code? Chat with the Labs • • IOD sessions IOD Lab IBM Confidential Connect with AIM, Rational for joint presence at 3rd conferences and events. © 2013 IBM Corporation
  • 19. IOD 2013 Roadmap: Extend Database Application Versatility with NoSQL Session No. Session Title IPT-2059 JSON Document Support in IBM DB2 Schema Flexibility in a Trusted Database Environment IPT-2909 Embracing NoSQL: IBM and the Community IPT-3149 Big Data, Hadoop, and NoSQL: A Crash Course for a Fearless IBM DB2 Practitioner IDB-1943 Firefighter Safety Enhancement through NoSQL Information Integration Architectures with IBM DB2 Graph Store IDB-2075 Competitive Product Delivery with IBM DB2 JSON IDB-1878 Agile Product Development Using IBM DB2 with JSON IDX-3691 Get the Best of Both Worlds: Bring NoSQL to Your SQL Database IDX-3722 Balancing Big Data Workloads with NoSQL Auto-Sharding IDX-2551 "Developing ""Hybrid"" NoSQL/SQL Applications using JSON/BSON with Informix" IPT-3070 Extending Information Management Solutions with NoSQL Capabilities--XML, RDF and JSON IDZ-2599 NoSQL and DB2 for z/OS? Receiving Agility from a Trusted Enterprise Database © 2013 IBM Corporation
  • 20. DB2/Informix Joint Events 2014 Need to identify events where we can submit topics/speakers for maximum exposure (not just booth participation) 20 IBM Confidential © 2013 IBM Corporation
  • 21. Some Typical NoSQL Use Cases - Mostly Interactive Web/Mobile Online/Mobile Gaming − − − − − Leaderboard (high score table) management Dynamic placement of visual elements Game object management Persisting game/user state information Persisting user generated data (e.g. drawings) Display Advertising on Web Sites − − Ad Serving: match content with profile and present Real-time bidding: match cookie profile with ad inventory, obtain bids, and present ad Communications − Device provisioning Social Networking/Online Communities 21 E-commerce/Social Commerce – Storing frequently changing product catalogs Logging/message passing – Drop Copy service in Financial Services (streaming copies of trade execution messages into (for example) a risk or back office system) Dynamic Content Management and Publishing (News & Media) – Store content from distributed authors, with fast retrieval and placement – Manage changing layouts and user generated content
  • 22. NoSQL = RDBMS + JSON + Sharding NoSQL = JSON + Sharding InformixNoSQL = JSON + Sharding + Transaction + RDBMS(Hybrid) + HA(shared Disk) + HDR 22 IBM Confidential © 2013 IBM Corporation
  • 23. NOSQL, JSON AND BSON OVERVIEW Technical Opportunities/ Motivation What are NoSQL Databases? Quick overview of JSON What is sharding?
  • 24. New Era in Application Requirements Store data from web/mobile application in their native form − New web applications use JSON for storing and exchanging information − Very lightweight – write more efficient applications − It is also the preferred data format for mobile application back-ends Move from development to production in no time! − Ability to create and deploy flexible JSON schema − Gives power to application developers by reducing dependency on IT − Ideal for agile, rapid development and continuous integration
  • 25. Why NoSQL? Non-traditional data management requirements driven by Web 2.0 companies − Document stores, key-value stores, graph and columnar dbms The Three Vs: − Velocity – high frequency of data arrivals − Volume – BigData − Variability – unstructured data, flexibility in schema design New data interchange formats – like JSON (JavaScript Object Notation) and BSON (Binary JSON) Scale-out requirements across heterogeneous environment – Cloud computing 25
  • 26. What is a NoSQL Database? Not Only SQL or NOt allowing SQL A non-relational database management systems − Does not require a fixed schema − Avoids join operations − Scales horizontally − No ACID (eventually consistent) Good with distributing data and fast application development Provides a mechanism for storage and retrieval of data while providing horizontal scaling.
  • 27. IBM Use Case Characteristics for JSON
  • 28. Why Most Commercial Relational Databases cannot meet these Requirements
  • 29. NoSQL Database Philosophy Differences No ACID – No ACID (Atomicity, Consistency, Isolation, Durability) – An eventual consistence model No Joins – Generally single row/document lookups Flexible Schema – Rigid format 29 © 2013 IBM Corporation
  • 30. Partnership with IBM and MongoDB MongoDB and IBM announced a partnership in June 2013 There are many common use cases of interest addressed by the partnership − − − − Accessing JSON Data in DB2, Informix MongoDB using JSON query Schema-less JSON Data for variety of applications Making JSON data available for variety of applications Securing JSON Data IBM and MongoDB are collaborating in 3 areas: − − − Open Governance: Standards and Open Source Technology areas of mutual interest Products and solutions
  • 31. Basic Translation Terms/Concepts Mongo/NoSQL Terms Traditional SQL Terms Database Database Collection Table Document Row Field Column Collection Table Name John 31 Value Document 28 Scott Key 21 Tim {"name":"John","age":21} {"name":"Tim","age":28} {"name":"Scott","age":30} Age 30 Row
  • 32. JSON Details JSON Syntax Rules – – – – – JSON syntax is a subset of the JavaScript object notation syntax: Data is in name/value pairs Data is separated by commas Curly braces hold objects Square brackets hold arrays JSON Name/Value Pairs – JSON data is written as key/value pairs. – A key/value pair consists of a field name (in double quotes), followed by a colon, followed by a value: "name":"John Miller" The 6 types of JSON Values: – – – – – – 32 A number (integer or floating point) A string (in double quotes) A Boolean (true or false) An array (in square brackets) An object (in curly brackets) Null
  • 33. Example of Supported JSON Types Example of each JSON type Mongo specific JSON types in blue – date { "string":"John", "number":123.45, "boolean":true, "array":[ "a", "b", "c" ], "object: { "str":"Miller", "num":711 }, "value": NULL, "date": ISODate("2013-10-01T00:33:14.000Z") }
  • 34. JSON: JavaScript Object Notation What is JSON? − JSON is lightweight text-data interchange format − JSON is language independent − JSON is "self-describing" and easy to understand JSON is syntax for storing and exchanging text information much like XML. However, JSON is smaller than XML, and faster and easier to parse. { "name":"John Miller", "age":21, "count":27, "employees": [ { "f_name":"John" , "l_name":"Doe" }, { "f_name":"Anna","m_name" : "Marie","l_name":"Smith" }, { "f_name":"Peter" , "l_name":"Jones" } ] } BSON is a binary form of JSON. 34 © 2013 IBM Corporation
  • 35. The Power of JSON Drives Flexible Schema JSON key value pair enables a flexible schema
  • 36. Data Access Examples Relational Representation LName FName Address Miller John 123 Blazer St JSON Representation JSON_string = ‘{“LName”:”Miller”,”FName”,”John”,”Address”:”123 Blazer St”}’; Javascript data access var JSONelems = JSON.parse( JSON_string ) f_name = JSONelems.FNname; l_name = JSONelems.FName; l_addr = JSONelems.Address;
  • 37. Simple Code Example db.posts.insert( ‘{“author”:”John”, “date”,”2013-04-20”,”post”,”mypost Creates the database “db” if it does not exists Creates the collection “posts” if the it does not exists Insert a record into a blog post by user John db.posts.find ( ‘{ “author”:”John” }’ ) Retrieve all posts by user John ”}’ )
  • 38. Dynamic Elasticity Rapid horizontal scalability − Ability for the application to grow by adding low cost hardware to the solution − Ability to add or delete nodes dynamically − Ability rebalance the data dynamically Application transparent elasticity
  • 39. Why Scale Out Instead of Up? Scaling Out − Adding more servers with less processors and RAM − Advantages Startup costs much less Can grow instep with the application Individual servers cost less − Several less expensive server rather than fewer high cost servers − Redundant servers cost more Greater chance of isolating catastrophic failures Scaling Up − Adding more processors and RAM to a single server − Advantages Less power consumption than running multiple servers Less infrastructure (network, licensing,..)
  • 40. Difference between Sharding Data VS Replication Shard Key state= “CA” Shard Key state= “WA” Sharding Each node hold a portion of the data • Hash • Expression Data is copied to all nodes Actions are shipped to applicable nodes 40 Same data on each node Inserted data is placed on the correct node Shard Key state= “OR” Replication Work on local copy and modification are propagated
  • 41. Motivation of Sharding Synergistic with the application strategy Start small and grow with commodity hardware Sharding is not
  • 42. Sharding is not for Data Availability Sharding is for growth, not availability Redundancy of a node provides high availability for the data − Both Mongo and Informix allow for multiple redundant nodes − Mongo refers to this as Replica Sets and the additional nodes slaves − Informix refers to this as MACH, and additional nodes secondary With Informix the secondary server can: − Provide high availability − Scale out Execute select Allow Insert/Update/Deletes on the secondary servers Share Disks with the master/primary node
  • 43. Basic Data Distribution/Replication Terms Term Description Informix Term Shard A single node or a group of nodes holding the same data (replica set) Instance Replica Set A collection of nodes contain the same data MACH Cluster Shard Key The field that dictates the distribution of the documents. Must always exist in a document. Shard Key Sharded Cluster A group shards were each shard contains a portion of the data. Grid/Region Slave A server which contains a second copy of the data for read only processing. Secondary Server Remote Secondary 43
  • 44. Sharding is not for Data Availibility Shard Key state= “CA” Shard Key state= “WA” Shard Key state= “OR” 44
  • 46. IBM Use Case Characteristics for JSON
  • 48. Flexible Schema, Native JSON and BSON Provide native JSON & BSON support in the Informix Database Server Two new built in “first-class” data types called JSON and BSON Support for MongoDB client side APIs throughout the entire database Enhance horizontal scaling by enabling sharding on all database objects and models Adaptive default system initialization − − 48 Up and running in seconds Adapts to computer an environment
  • 49. Two New Data Types JSON and BSON Native JSON and BSON data types Index support for NoSQL data types Native operators and comparator functions allow for direct manipulation of the BSON data type Database Server seamlessly converts to and from JSON BSON Character data 49 JSON
  • 50. JSON and BSON Data Type Details Row locks the individual BSON/JSON document − MongoDB must lock the database Bigger documents – 2GB maximum size − MongoDB caps at 16MB Ability to compress documents − MongoDB currently not available Ability to intelligently cache commonly used documents − MongoDB currently not available
  • 51. Client Applications Applications MongoDB native Client New Wire Protocol Listener supports existing MongoDB drivers Connect to MongoDB or Informix with same application! MongoDB driver MongoDB Wire Protocol IBM JDBC NoSQL Driver Wire Protocol Listener MongoDB web browser MongoDB shell Mobile Informix DB
  • 52. MongoDB Application Compatibly Ability to use any of the MongodB client drivers and frameworks against the Informix Database Server − Little to no change required when running MongoDB programs − Informix listens on the same default port as mongo, no need to change. Leverage the different programming languages available − Language examples C, C#, Erlang, JAVA, node.js, Perl, Python, Ruby Mongo Action Description db.customer.insert( { name: “John", age: 21 } ) Insert into database “db” the customer collection the associated document. db.customer.find( {age: { $gt:21 } } ) Retrieve from database “db” the customer collection all documents with the age greater than 21. 52
  • 53. Fix up JSON Basic Mongo Operations Conceptual Operations Mongo Action db.customer.insert( { name: “John", age: 21 } ) CREATE DATABASE if not exist db CREATE TABLE if not exist customer INSERT INTO customer VALUES ( { “name”:”John”,”age:21”} ) db.customer.find() SELECT bson_doc FROM customer db.customer.find( {age: { $gt:21 } } ) SELECT * FROM customer WHERE age > 21 db.customer.drop() DROP TABLE customer db.customer.ensureIndex( { name : 1, age : -1 } ) CREATE INDEX idx_1 on customer(name , age DESC) db.customer.remove( {age: { $gt:21 } } ) DELETE FROM customer where age > 21 db.customer.update( { age: { $gt: 20 } }, { $set: { status: “Drink" } }, { multi: true } ) 53 Traditional SQL Action UPDATE customer SET bson_doc_field (status) = { “status”:“Drink" } WHERE age > 20
  • 54. Scaling Out Using Sharded Queries Shard Key state= “CA” Shard Key state= “WA” Find sold cars for all states 1. Request data from local shard 2. Automatically sends request to other shards requesting data 3. Returns results to client Shard Key state= “OR” 54 © 2013 IBM Corporation
  • 55. Scaling Out Using Sharded Inserts Shard Key state= “CA” Shard Key state= “WA” Row state = “OR” 1. Insert row sent to your local shard 2. Automatically forward the data to the proper shard Shard Key state= “OR” 55 © 2013 IBM Corporation
  • 56. Scaling Out Using Sharded Delete Shard Key state= “CA” Shard Key state= “WA” Delete state = “OR” AND Dnum = “123123” 1. Delete condition sent to local shard 2. Local shard determine which shard(s) to forward the operation Shard Key state= “OR” 56 3. Execute the operation © 2013 IBM Corporation
  • 57. Scaling Out Using Sharded Update Shard Key state= “CA” Shard Key state= “WA” Row state = “OR” Row state = “OR” 1. Insert a row on your local shard 2. Automatically forward the data to the proper shard Shard Key state= “OR” 57 © 2013 IBM Corporation
  • 58. Scaling Out Adding a Shard Shard Key state= “CA” Shard Key state= “WA” Command Add Shard “NV” 1. Send command to local node 2. New shard dynamically added, data re-distributed (if required) Shard Key state= “OR” 58 Shard Key state= “NV” © 2013 IBM Corporation
  • 59. Sharding with Hash Hash based sharding simplifies the partitioning of data across the shards Benefits − No data layout planning is required − Adding additional nodes is online and dynamic Cons − Adding additional node requires data to be moved Data automatically broken in pieces
  • 60. Scaling Out with Hash Sharding - Insert Shard Key HASH( gambler_ID ) Shard Key HASH( gambler_ID ) Row Gambler_ID =“11” 1. Insert row sent to your local shard 2. Local shard determine which shard(s) to forward the insert 3. Execute the operation Shard Key HASH( gambler_ID ) 60 © 2013 IBM Corporation
  • 61. Move to END Informix NoSQL Cluster Architecture Overview Shard Key state= “CA” Shard Key state= “OR” 61 two independent copies of the data, but three servers to share the workload (two servers share the same disks). Read/Write activity supported on all servers independent copies of two the data and two servers to share the workload. Read/Write activity supported on all servers three independent copies of the data, but four servers to share the workload (two servers share the same disk). Read/Write activity supported on all servers Shard Key state= “WA” © 2013 IBM Corporation
  • 63. INFORMIX NOSQL DEEP DIVE INTO IMPLEMENTATION
  • 64. Focus Areas. Applications need easier way to persist objects. − Object-relational layers like hibernate has performance & functional issues. − Support Mongo API and its eco system Flexible Schema: Changing schema in RDBMS is a significant operation, especially in production. − Need exclusive access to table & application downtime − Fast changing schema & sparse data has issues with fixed schema. -- Implement JSON (BSON) type, data store & indexing Scale Out: While RDBMS has solutions for cluster and MPP, features enabling application development and deployment in cloud+MPP architecture is limited. − Use Informix flexible grid & replication to shard tables; − Enhance query processing for distributed queries
  • 65. Client Applications New Wire Protocol Listener supports existing MongoDB drivers Connect to MongoDB or Informix with same application! MongoDB native Client application MongoDB MongoDB driver IBM JDBC NoSQL Driver Wire Protocol Listener MongoDB web browser application Informix NoSQL Cluster MongoDB shell IBM Wire Protocol Listener logic shared with DB2 and Extreme Scale 65 © 2013 IBM Corporation
  • 66. Informix for SQL and NoSQL Applications NoSQL APP IBM Wire Listener IBM Wire Listener JDBC connections JDBC connections SQL Apps + Tools SQL Apps + Tools SQL Drivers ODBC/JDBC NoSQL APP SQL Drivers ODBC/JDBC IBM Wire Listener JDBC connections Informix Dynamic Server (shard 1) Informix Dynamic Query processing Distributed Queries JSON Tables Collections Server (shard…n) IDXs IDXs JSON Tables Collections Tables Tables Tables IDXs Logs Enterprise replication + Flexible Grid Tables IDXs ER + Grid Logs
  • 67. Informix for SQL and NoSQL Applications Mongo Application SQL Apps + Tools NoSQL/BSON SQL IBM Wire Listener SQL Drivers ODBC/JDBC JDBC connections SQL + BSON SQL Informix Dynamic Server (shard 1) Query processing JSON Tables Collections Distributed Queries SELECT Informix Dynamic Server (shard…n) IDXs IDXs JSON Tables Collections Tables Tables Tables IDXs Logs Enterprise replication + Flexible Grid INSERT UPDATE DELETE Tables IDXs ER + Grid Logs
  • 72. NoSQL Feature Informix Implementation 1. Flexible Schema Use BSON and JSON data type. Complete row is stored in a single column; BSON, JSON are multi-rep types and can store up to 2GB. 2. Accessing KV pairs within JSON. Translate the Mongo queries into SQL expressions to access key-value pairs. Informix has added expressions to extract specific KV pairs. Select bson_new(data, “{id:1, name:1, addr:1}”) from tab; 3. Indexing Support standard B-tree index via functional indexing. create index itid on t(bson_value_int(data, “{id:1}”); Create index itnamestate on t(bson_value_lvarchar(data, “{name,1}”), bson_value_lvarchar(data, “{city,1}”); Informix also supports indexing bsons keys with different data types. 4. Sharding Supports range & hash based sharding.Informix has built-in technology for replication. Create identical tables in multiple nodes. Add meta data for partitioning the data across based on range or hash expressions. 5. SELECT Limited support now. Mongo API helps by disallowing joins. The translated query on a single table is transformed into federated UNION ALL query; includes shard elimination.
  • 73. NoSQL Feature Informix Implementation 6. Updates (single node) INSERT: Simple direct insert. DELETE: DELETE statement with WHERE bson_extract() > ?; or bson_value..() > ? UPDATE: bson_update(bson, “update expr”) will return a new bson after applying the bson expression. Simple updates to non_sharded tables will be direct UPDATE statement in a single transaction. UPDATE tab bsoncol = bson_update(bsoncol, “expr”) where 7. Updates (sharded env) INSERT – All rows are inserted to LOCAL shard, replication threads will read logs and replicate to the right shard and delete from local node (log only inserts underway). DELETE – Do the local delete and replicate the “delete statements” to target node in the background UPDATE – Slow update via select-delete-insert. 8. Transaction Each statement is a distinct transaction in a single node environment. The data and the operation is replicated via enterprise replication. 9. Isolation levels. NoSQL session can use any of Informix isolation levels. -- Examples from Mongo applications. 10. Locking Application control only on the node they’re connected to. Standard Informix locking semantics will apply when data is replicated and applied on the target shard. -- Verify.. What’s the default locking level? -- Any option to change it? Just ONCONFIG??
  • 74. Feature Informix Implementation 11. Hybrid access Mongo APIs can simply access Informix tables, views, virtual tables as if they’re JSON collections. db.foo.find(); foo can be JSON collection, relational table, a view or a virtual table on top of timeseries or Websphere MQ. From MongoAPI to relational tables. 12. Hybrid access 1. Directly get binary BSON or cast to JSON to get in textual form. From SQL to JSON data 2. Use expressions to extract to extract specific key-value pairs. 3. To be done: NoSQL collections will only have one BSON object in the table. We can “imply” the expressions when the SQL refers to a column. SELECT t.c1, t.c2 from t; SELECT bson_extract(t.data, “{c1:1}”), bson_extract(t.data, “{c2:1}”) from t; So, that’s twelve steps for NoSQL Anonymous!
  • 75. 1. Flexible Schema Clients exchange BSON document with the server both for queries and data. Thus, BSON becomes a fundamental data type. The explicit key-value pairs withing the JSON/BSON document will be roughly equivalent to columns in relational tables. However, there are differences! − The type of the KV data encoded within BSON is determined by the client − Server is unaware of data type of each KV pair at table definition time. − No guarantees that data type for each key will remain consistent in the collection. − The keys in the BSON document can be arbitrary; − While customers exploit flexible schema, they’re unlikely to create a single collection and dump everything under the sun into that collection. − Due to the limitations of Mongo/NoSQL API, customers typically denormalize the tables (customer will have customer+customer addr + customer demographics/etc) to avoid joins.
  • 76. 1. Flexible Schema – Informix Implementation • Informix has a new data type BSON to store the data. • Informix also has a JSON data type to convert between binary and text form. • BSON and JSON are abstract data types (like spatial, etc). • BSON and JSON multi-representational types. •Objects up to 4K is stored in data pages. •Larger objects (up to 2GB) are stored out of row, in BLOBs. •MongoDB limits objects to 16MB. •This is all seamless and transparent to applications.
  • 77. 1. Flexible Schema – Informix Implementation CREATE TABLE customer (data BSON); • BSON is the binary represenation of JSON. •It has length and types of the key-value pairs in JSON. • MongoDB drivers send and receive in BSON form.
  • 78. 2. Accessing KV pairs within JSON. • We’ll have number of extract expressions/functions •Expressions returning base type bson_value_bigint(BSON, “key”); bson_value_lvarchar(bsoncol, “key.key2”); bson_value_date(bsoncol, “key.key2.key3”); bson_value_timestamp(bsoncol, “key”) bson_value_double(bsoncol, “key”); bson_value_boolean(bsoncol, “key”); bson_value_array(bsoncol, “key”); bson_keys_exist(bsoncol, “key”); Bson_value_document(bsoncol, “key”) Bson_value_binary(bsonol, “key”) Bson_value_objectid(bsoncol, “key”) •Expression returning BSON subset. Used for bson indices. bson_extract(bsoncol, “projection specification”) •Expressions to project out of SELECT statement. bson_new(bsoncol, “{key1:1, key2:1, key3:1}”); bson_new(bsoncol, “{key5:0}”);
  • 79. 2. Accessing KV pairs within JSON. Mongo Query SQL Query db.customers.find(); SELECT SKIP ? Data::bson db.customers.find({},{num:1,name:1}) ; SELECT SKIP ? bson_new( data, '{ "num" : 1.0 , "name" : 1.0}')::bson FROM customers db.customers.find({}, {_id:0,num:1,name:1}); SELECT SKIP ? bson_new( data, '{_id:0.0, "num" : 1.0 , "name" : 1.0}')::bson FROM customers db.customers.find({status:”A”}) SELECT SKIP ? data FROM customers WHERE bson_extract(data, ‘status') = “A” db.customers.find({status:”A”}, {_id:0,num:1,name:1}); FROM customers SELECT SKIP ? bson_new( data, '{ "_id" : 0.0 , "num" : 1.0 , "name" : 1.0}')::bson FROM customers WHERE bson_extract(data, 'name') = “A”
  • 80. 3. Indexing • • • • • Supports B-Tree indexes on any key-value pairs. Indices could be on simple basic type (int, decimal) or BSON Indices could be created on BSON and use BSON type comparison Listener translates ensureIndex() to CREATE INDEX Listener translates dropIndex() to DROP INDEX Mongo Query SQL Query db.customers.ensureIndex({orderD ate:1}) CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1" , "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({orderD ate:1, zip:-1}) CREATE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" : "c1_1_c2__1" , "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({orderD ate:1}, {unique:true) CREATE UNIQUE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" :"c1_1_c2__1" , "unique" : true , "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64
  • 81. 3. Indexing db.w.find({x:1,z:44},{x:1,y:1,z:1}) Translate to: SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1} FROM w WHERE ( bson_extract(data, 'x') = '{ "x" : 1.0 }'::json::bson ) AND ( bson_extract(data, 'z') = '{ "z" : 44.0 }'::json::bson) Estimated Cost: 2 Estimated # of Rows Returned: 1 1) keshav.w: SEQUENTIAL SCAN Filters: (informix.equal(informix.bson_extract(keshav.w.data ,'z' ),UDT ) AND informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT))
  • 82. 3. Indexing •Functional Index is built on bson expressions CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1" , "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64 •Listener is aware of the available index and therefore generates right predicates. db.w.find({x:1}); gets translated to SELECT SKIP ? data FROM w WHERE bson_extract(data, 'x') = ?
  • 83. 3. Indexing db.w.find({x:5,z:5}, {x:1,y:1,z:1}) Translates to: SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1 .0}')::bson FROM w WHERE ( bson_extract(data, 'x') = '{ "x" : 5.0 }'::json::bson ) AND ( bson_extract(data, 'z') = '{ "z" : 5.0 }'::json::bson ) Estimated Cost: 11 Estimated # of Rows Returned: 1 1) keshav.w: INDEX PATH Filters: informix.equal(informix.bson_extract(keshav.w.data ,'z' ), UDT ) (1) Index Name: keshav.w_x_1 Index Keys: informix.bson_extract(data,'x') (Serial, fragments: ALL) Lower Index Filter: informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT )
  • 84. 84 © 2013 IBM Corporation
  • 86. 4. Mongodb SHARDING (roughly) Shard a single table by range or hashing. Mongos will direct the INSERT to target shard. Mongos tries to eliminate shards for update, delete, selects as well. FIND (SELECT) can happen ONLY a SINGLE table. It also works as coordinator for multi-node ops. Once a row is inserted to a shard, it remains there despite any key update. No transactional support on multi-node updates. − Each document update is unto its own.
  • 87. 4. Informix Sharding App Server App Server App Server App Server Mongo Driver Mongo Driver Mongo Driver Mongo Driver Listener Listener Listener Listener JDBC JDBC JDBC JDBC Customer/1 Customer/2 Customer/3 Customer/4 Customer/5 Customer/6 Sales/1 Sales/2 Sales/3 Sales/4 Sales/5 Sales/6 Location/1 Location/2 Location/3 Location/4 Location/5 Location/6 Informix/2 Informix/3 Informix/1 Informix/4 Informix/5 Enterprise repliation + Flexible Grid Informix/6
  • 88. 4. Informix sharding + High Availability App Server App Server App Server App Server Mongo Driver Mongo Driver Mongo Driver Mongo Driver Listener Listener Listener Listener JDBC JDBC JDBC JDBC Informix/1 Primary Informix/2 Primary Informix/3 Primary Informix/4 Primary Informix/5 Primary Informix/6 Primary Informix/1 SDS/HDR Informix/2 SDS/HDR Informix/3 SDS/HDR Informix/4 SDS/HDR Informix/5 SDS/HDR Informix/6 SDS/HDR Informix/1 RSS Informix/2 RSS Informix/3 RSS Informix/4 RSS Informix/5 RSS Informix/6 RSS Enterprise repliation + Flexible Grid
  • 89. 4. Sharding – Informix Implementation •Shard a single table by range or hashing. cdr define shard myshard mydb:usr1.mytab –type=delete –key=”bson_get(bsoncol, ‘STATE’)” –stragety=expression versionCol=version servA “in (‘TX’, ‘OK’)” servB “in (‘NY’,’NJ’) “ servC “in (‘AL’,’KS’) “ servD remainder cdr define shard myshard mydb:usr1.mytab –type=delete –key=state –stragety=hash --versionCol=version servA servB servC servD •Shard a single table by range or hashing. <<Mongo syntax>>
  • 90. 4. Sharding – Informix Implementation •Shard a single table by range or hashing. •Sharding is transparent to application. •Each CRUD statement will only touch a single table. •Limitation of MongoDB…Makes it easier for Informix. •Lack of joins is a big limitation for SQL applications. •Lacks transactional support for distributed update.
  • 91. 4. Informix Sharding Identical table is created on each node and meta data is replicated on each node. Schema based replication is the foundation for our sharding. CRUD operations can go to any node. − We’ll use replication and other techniques to reflect the data in the target node, eventually. − Right now, replication is asynchronous. − Informix has synchronous replication not used for sharding now.
  • 92. 5. SELECT on sharded tables •The query can be submitted to any of the nodes via the listener. •That node acts as the “coordinator” for the distributed query. •It also does the node elimination based on the query predicate. •After that, the query is transformed to UNION ALL query SELECT SKIP ? bson_new( data, '{"_id":0.0 ,"num":1.0 ,"name" : 1.0}')::bson FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X” is transformed into: SELECT SKIP ? bson_new( data, '{"_id":0.0 FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or UNION ALL SELECT SKIP ? bson_new( data, '{"_id":0.0 FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or ,"num":1.0 ,"name" : 1.0}')::bson bson_extract(data, 'name') = “X” ,"num":1.0 ,"name" : 1.0}')::bson bson_extract(data, 'name') = “X”
  • 93. 6. INSERT: Single node •If necessary, automatically create database & table (collection) on the application INSERT •Collections: CREATE TABLE t(a GUID, d BSON); •GUID column is needed to ensure unique row across the SHARDs. Also used as PK for replication. •Client application inserts JSON, client API converts this into BSON, generates _id (object id) if necessary & sends to server over JDBC/ODBC. •Server saves the data into this table as BSON, with an automatically generated GUID
  • 94. 6. DELETE: Single node •Mongo remove are translated to SQL DELETE •Will always remove all the qualifying rows. •WHERE clause translation is same as SELECT
  • 95. 6. UPDATE: Single node •Simple set, increment, decrement updates are handled directly by the server. Mongo: db.w.update({x: {$gt:55}}, {$set:{z:9595}}); SQL: UPDATE w SET data = bson_update(data, “{$set:{z:9595}}”) WHERE bson_extract(data, 'x') > “{x:55}”::bson ? •bson_update is a built-in expression updating a BSON document with a given set of operations. •Complex updates are handled via select batch, delete, insert. •Always updates all the rows or no rows, under a transaction.
  • 96. 7. INSERT: shard implementation •If the table is sharded: •Insert the data into local table as usual. •Replication threads in the background will evaluate the log record to see which rows will have moved. •Replication thread will move the necessary row to target and delete from the local table. •Work underway to avoid the insert into local table. •For each inserted row ending up in non-local shard, simply generate the logical log and avoid insert into local table & indices.
  • 97. 7. DELETE : shard implementation •Application delete could delete rows from any of the shards in the table. •DELETE will come in as shard_delete() procedure. –Execute procedure shard_delete(tabname, delete_stmt); –This procedure will issue delete locally. –It will then INSERT the delete statement into a “special” shard delete table (single for all tables). –Enterprise replication will propagate the delete to applicable target systems.
  • 98. 7. UPDATE: shard implementation •When the update have to be done on multiple nodes: •Client application does UPDATE and CLIENT API converts that into three operations: SELECT, DELETE & INSERT. •GUID column is needed to ensure unique row across the SHARD •Client application inserts JSON, client API converts this into BSON & sends to server.
  • 99. 8. Transactions (single node) •Mongo does not have the notion of transactions. •Each document update is atomic, but not the app statement •For the first release of Informix-NoSQL •By default, JDBC listener simply uses AUTO COMMIT option •Each server operation INSERT, UPDATE, DELETE, SELECT will be automatically be committed after each operation. •No locks are held across multiple operation. •However, customers can create multi-statement transaction via $sql and issuing begin work, commit work, rollback work
  • 100. 8. Transactions (sharded environment) •In sharded environment, mongo runs database via two different instances: mongos and mongod. •Mongos simply redirects operations to relevant mongod. •No statement level transactoinal support. •Informix •Informix does not have the 2-layer architecture •Informix server the application connected to becomes the transaction coordinator •Informix does have the 2-phase commit transaction support, but is unused for NoSQl, for now. •SELECT statement goes thru distributed query infrastructure •INSERT,UPDATE DELETE goes thru enterprise replication.
  • 101. 9. ISOLATION levels •Default isolation level is DIRTY READ. •Change this directly or sysdbopen() •You can also use USELASTCOMMITTED variable in ONCONFIG. •If you’re using procedures for executing multistatement transaction, you can set it within your procedure.
  • 102. 10. LOCKING •Page level locking is the default. •You can change it to ROW level locks easily. •ALTER TABLE jc MODIFY lock mode (row) • DEF_TABLE_LOCKMODE onconfig variable. • SET LOCK MODE can be set via sysdbopen() •Each statement is executed with auto-commit and locking semantics will apply there.
  • 104. Hybrid Access between relational & JSON Collections Relational Table SQL API Standard ODBC, JDBC, .NET, OData, etc. Language SQL. MongoDB API (NoSQL) ? JSON Collections ? Mongo APIs for Java, Javascript, C++, C#,...
  • 105. Benefits of Hybrid Power Access consistent data from its source Avoid ETL, continuous data sync and conflicts. Exploit the power of SQL, MongoAPI seamlessly Exploit the power of RDBMS technologies in MongoAPI: − Informix Warehouse accelerator, − Cost based Optimizer − R-tree indices for spatial, Lucene text indexes, and more. Access all your data thru any interface: MongoAPI or SQL. Store data in one place and efficiently transform and use them on demand. Existing SQL based tools and APIs can access new data in JSON
  • 106. Why do you need hybrid access? Data model should not restrict Data Access
  • 107. Hybrid Access between relational & JSON Collections Relational Table SQL API MongoDB API (NoSQL) Standard ODBC, JDBC, .NET, OData, etc. Language SQL. Mongo APIs for Java, Javascript, C++, C#,... JSON Collections Direct SQL Access. Dynamic Views Row types Mongo APIs for Java, Javascript, C++, C#,...
  • 108. Hybrid Power MongoAPI to relational data
  • 109. Hybrid access: From MongoAPI to relational tables. You want to develop an application with MongoAPI, but 1. You already have relational tables with data. 2. You have views on relational data 3. You need to join tables 4. You need queries with complex expressions. E.g. OLAP window functions. 5. You need to get results from a stored procedure. 6. You need to exploit Informix stored procedure 7. You need federated access to other data 8. You have timeseries data.
  • 110. How to treat relational data as JSON store. Relational data (relations or resultset) can treated as structured JSON documents; column name-value becomes key-value pair. SELECT partner, pnum, country from partners; partner pnum Country Pronto 1748 Australia Kazer 1746 USA Diester 1472 Spain Consultix 1742 France {parnter: {parnter: {parnter: {parnter: “Pronto”, pnum:”1748”, Country: “Australia”} “Kazer”, pnum:”1746”, Country: “USA”} “Diester”, pnum:”1472”, Country: “Spain”} “Consultix”, pnum:”1742”, Country: “France”} Listner translates the query and the data object between relational and JSON/BSON form.
  • 111. Hybrid access: From MongoAPI to relational tables. Mongo Application JSON JSON db.customer.find({state:”MO”}) db.partners.find({state:”CA”}) IBM Wire Listener JDBC connections Access JSON SELECT bson_new(bson, ‘{}’) FROM customer WHERE bson_value_lvarchar(bson,‘state’)= “MO” Access Relational SELECT * FROM partners WHERE state = “CA” Informix Dynamic Server JSON Collections Customer Distributed Queries Tables IDXs Tables Relational Tables Logs partners Tables IDXs Enterprise replication + Flexible Grid
  • 112. Accessing data in relational tables. Create table partners(pnum int, name varchar(32), country varchar(32)); db.partners.find(); SELECT * from partners; db.partners.find({name:”Pronto”}); SELECT * FROM PARTNERS WHERE name = “Pronto”; db.partners.find({name:”Pronto”}, {pnum:1, country:1}); SELECT a, b FROM t WHERE a = 2.0; db.partners.find({name:”Pronto”}, {pnum:1, country:1}).limit(10); SELECT LIMIT 10 pnum, country FROM WHERE name = “Pronto”; db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:1}) SELECT pnum,country FROM partners WHERE name = “Pronto” ORDER BY b ASC db.partners.find({name:”Pronto”}, {pnum:1, country:1}).sort({b:-1}) SELECT a, b FROM t WHERE a = 2.0 ORDER BY b DESC db.t.find({a:{$gt:1}}, {a:1, b:1}).sort({b:-1}) SELECT SKIP ? a, b FROM t WHERE query > 1.0 ORDER BY b DESC
  • 113. Accessing data in relational tables. db.partners.save({pnum:1632, name:”EuroTop”, Country: “Belgium”}); INSERT into partners(pnum, name, country values(1632, ”EuroTop”, “Belgium”); db.partners.delete({name:”Artics”}); DELETE FROM PARTNERS WHERE name = “Artics”; Db.partners.update({country:”Holland”},{$set:{country:”Netherland”}} , {multi: true}); UPDATE partners SET country = “Netherland” WHERE country = “Holland”; db.partners.drop(); DROP TABLE partners;
  • 114. Views and JOINS A dynamically created relation created from one or more database tables or procedures. create table pcontact(pnum int, name varchar(32), phone varchar(32)); insert into pcontact values(1748, "Joe Smith", "61-123-4821"); insert into pcontact values(1746, "John Kelley", "1-729-284-2893"); insert into pcontact values(1472, "Ken Garcia", "34-829-2842"); insert into pcontact values(1742, "Adam Roy", "33-380-3892"); create view partnerphone(pname, pcontact, pphone) as select a.name, b.name, b.phone FROM pcontact b left outer join partners a on (a.pnum = b.pnum); db.partnerphone.find(); { "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" } { "pname" : "Kazer", "pcontact" : "John Kelley", "pphone" : "1-729-284-2893" } { "pname" : "Diester", "pcontact" : "Ken Garcia", "pphone" : "34-98-829-2842" } { "pname" : "Consultix", "pcontact" : "Adam Roy", "pphone" : "33-82-380-3892" } db.partnerphone.find({pname:"Pronto"}) { "pname" : "Pronto", "pcontact" : "Joe Smith", "pphone" : "61-123-4821" }
  • 115. complex expressions. E.g. OLAP window functions create view contactreport(pname, pcontact, totalcontacts) as select b.name, a.name, count(a.name) over(partition by b.pnum) from pcontact a left outer join partners b on (a.pnum = b.pnum); db.contactreport.find({pname:"Pronto"}) { "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 } { "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 }
  • 116. Seamless federated access 1. 2. create database newdb2; create synonym oldcontactreport for newdb:contactreport; > use newdb2 > db.oldcontactreport.find({pname:"Pronto"}) { "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 } { "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 } SELECT data FROM oldcontactreport WHERE bson_extract(data, 'pname') = “Pronto”; • create synonym oldcontactreport for custdb@nydb:contactreport;
  • 117. Get results from a stored procedure. create function "keshav".p6() returns int, varchar(32); define x int; define y varchar(32); foreach cursor for select tabid, tabname into x,y from systables return x,y with resume; end foreach; end procedure; create view "keshav".v6 (c1,c2) as select x0.c1 ,x0.c2 from table(function p6())x0(c1,c2); db.v6.find().limit(5) { { { { { "c1" "c1" "c1" "c1" "c1" : : : : : 1, 2, 3, 4, 5, "c2" "c2" "c2" "c2" "c2" : : : : : "systables" } "syscolumns" } "sysindices" } "systabauth" } "syscolauth" }
  • 118. Access Timeseries data create table daily_stocks ( stock_id integer, stock_name lvarchar, stock_data timeseries(stock_bar) ); -- Create virtual relational table on top (view) EXECUTE PROCEDURE TSCreateVirtualTab('daily_stocks_virt', 'daily_stocks', 'calendar(daycal),origin(2011-01-03 00:00:00.00000)' ); create table daily_stocks_virt ( stock_id integer, stock_name lvarchar, timestamp datetime year to fraction(5), high smallfloat,
  • 119. Access Timeseries data db.daily_stocks_virt.find() { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0 0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0 0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0 0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 } { "stock_id" : 902, "stock_name" : "HPQ", "timestamp" : ISODate("2011-01-03T06:0 0:00Z"), "high" : 403, "low" : 401, "final" : 402, "vol" : 253 } { "stock_id" : 902, "stock_name" : "HPQ", "timestamp" : ISODate("2011-01-04T06:0 0:00Z"), "high" : 503, "low" : 501, "final" : 502, "vol" : 302 }
  • 120. Access Timeseries data db.daily_stocks_virt.find({stock_name:"IBM"}) { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0 0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0 0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0 0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 } db.daily_stocks_virt.find({stock_name:"IBM"}).sort({final:-1}) { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0 0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0 0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0 0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 } mongos>
  • 121. Hybrid access: From MongoAPI to relational tables. You want to develop an application with MongoAPI, but 1. You already have relational tables with data. 2. You have views on relational data 3. You need to join tables 4. You need queries with complex expressions. E.g. OLAP window functions. 5. You need to get results from a stored procedure. 6. You need to exploit Informix stored procedure 7. You need federated access to other data 8. You have timeseries data. Put all of these together!
  • 123. Hybrid access: From SQL API to JSON COllections. SQL Applications JDBC connections Access JSON Access Relational SELECT bson_new(bson, “{customer:1}’) FROM customer WHERE bson_value_lvarchar(bson,‘state’)= “MO” Select * from patners where state = “CA”; Informix Dynamic Server JSON Collections Customer Distributed Queries Tables IDXs Tables Relational Tables Logs partners Tables IDXs Enterprise replication + Flexible Grid
  • 124. Join JSON collections JSON Collection V: { "_id" : ObjectId("526a1bb1e1819d4d98a5fa4b"), "c1" : 1, "c2" : 2 } JSON Collection w: { "_id" : ObjectId("526b005cfb9d36501fe31605"), "x" : 255, "y" : 265, "z" : 395} { "_id" : ObjectId("52716cdcf278706cb7bf9955"), "x" : 255, "y" : 265, "z" : 395} { "_id" : ObjectId("5271c749fa29acaeba572302"), "x" : 1, "y" : 2, "z" : 3 } SELECT c1, c2, x,y,z FROM V, W WHERE V.c1 = W.x;
  • 125. Join JSON collections -- Wrap aorund the built-in expressions SELECT bson_value_int(jc1.data, 'x'), bson_value_lvarchar(jc1.data, 'y'), bson_value_int(jc1.data, 'z') , bson_value_int(jc2.data, 'c1'), bson_value_lvarchar(jc2.data, 'c2') FROM w jc1, v jc2 WHERE bson_value_int(jc1.data, 'x') = bson_value_int(jc2.data, 'c1');
  • 126. Join JSON collections -- Create a view to make the access simple. create view vwjc(jc1x, jc1y, jc1z, jc2c1, jc2c2) as SELECT bson_value_int(jc1.data, 'x'), bson_value_lvarchar(jc1.data, 'y'), bson_value_int(jc1.data, 'z') , bson_value_int(jc2.data, 'c1'), bson_value_lvarchar(jc2.data, 'c2') FROM w jc1, v jc2 WHERE bson_value_int(jc1.data, 'x') = bson_value_int(jc2.data, 'c1'); db.vwjc.find(); Db.vwjc.find({jc1x:100}); Db.vwjc.find({jc1x:{$gte:10}}).sort(jc1y:1)
  • 127. You want to perform complex analytics on JSON data BI Tools like Cognos, Tableau generate SQL on data sources. Option 1: Do ETL Need to expose JSON data as views so it’s seen as a database object. − We use implicit casting to convert to compatible types − The references to non-existent key-value pair returns NULL Create any combination of views − A view per JSON collection − Multiple views per JSON collection − Views joining JSON collections, relational tables and views. Use these database objects to create reports, graphs, etc.
  • 128. Analytics SQL & BI Applications ODBC, JDBC connections Informix Dynamic Server JSON Collections Customer Tables & views Tables Tables Orders Relational Tables partners Inventory Tables CRM Informix Warehouse Accelerator
  • 129. Hybrid Access between relational & JSON Collections Relational Table JSON Collections SQL API Standard ODBC, JDBC, .NET, OData, etc. Language SQL. Direct SQL Access. Dynamic Views Row types MongoDB API Mongo APIs for Java, (NoSQL) Javascript, C++, C#,... Mongo APIs for Java, Javascript, C++, C#,...
  • 130. Ability for MongoDB API to Access All Data Models node.js Application Traditional SQL MongoDB Drivers NoSQL - JSON NoSQL - JSON TimeSeries TimeSeries MQ Series MQ Series MQ Series 130
  • 131. All three drivers support all data models Applications native Client Informix SQLI Drivers Traditional SQL IBM DRDA Drivers MongoDB Drivers web browser NoSQL - JSON TimeSeries MQ Series Mobile © 2013 IBM Corporation
  • 132. Benefits of Hybrid Power Access consistent data from its source Avoid ETL, continuous data sync and conflicts. Exploit the power of SQL, MongoAPI seamlessly Exploit the power of RDBMS technologies via MongoAPI: − Informix Warehouse accelerator, − Cost based Optimizer & power of SQL − R-tree indices for spatial, Lucene text indexes, and more. Access all your data thru any interface: MongoAPI & SQL Store data in one place and efficiently transform and use them on demand. Existing SQL based tools and APIs can access new data in JSON
  • 134. Simple SQL statements which Operate on the BSON CREATE TABLE mycollection (data BSON); INSERT INTO mycollection (data) VALUES ( '{"name":“John"}‘ ::JSON ); CREATE INDEX myindx ON mycollection(bson_extract(data,“name")) USING BSON SELECT data::JSON FROM mycollection; 134 © 2013 IBM Corporation
  • 135. The SQL to Create a Collection Formal definition of a collection used by the Wire Listener CREATE COLLECTION TABLE mycollection ( id char(128), modcnt integer, data bson, flags integer ); 135 © 2013 IBM Corporation
  • 136. New Built-in BSON Expressions bson_value_array(lvarchar doc, lvarchar key) returns lvarchar as BSON array bson_value_bigint(lvarchar doc, lvarchar key) returns bigint bson_value_binary(lvarchar doc, lvarchar key) returns lvarchar as BSON binary bson_value_boolean(lvarchar doc, lvarchar key) returns boolean bson_value_code(lvarchar doc, lvarchar key) returns lvarchar as string bson_value_date(lvarchar doc, lvarchar key) returns datetime bson_value_document(lvarchar doc, lvarchar key) returns lvarchar as BSON object bson_value_double(lvarchar doc, lvarchar key) returns float bson_value_int(lvarchar doc, lvarchar key) returns bigint bson_value_lvarchar(lvarchar doc, lvarchar key) returns lvarchar as string bson_value_objectid(lvarchar doc, lvarchar key) returns lvarchar as string bson_value_timestamp(lvarchar doc, lvarchar key) returns datetime bson_keys_exist(lvarchar doc, lvarchar key(s)) returns Boolean bson_new(lvarchar doc, lvarchar projection_list) return lvarchar as BSON binary bson_type(lvarchar doc, lvarchar key) return integer bson_size(lvarchar doc, lvarchar key) return integer bson_extract(lvarchar doc, lvarchar) return lvarchar as BSON binary 136
  • 137. Combining Tables and Collections SELECT customer.*, bson_value_lvarchar(data,"customer_code") AS customer_code FROM mycollection, customer WHERE bson_value_lvarchar(data,"fname") = customer.fname AND bson_value_lvarchar(data,"lname") = customer.lname AND customer_num < 102; Query Optimizer QUERY: (OPTIMIZATION TIMESTAMP: 07-21-2013 23:15:31) Estimated Cost: 6 Estimated # of Rows Returned: 10 1) miller3.customer: INDEX PATH (1) Index Name: miller3. 100_1 Index Keys: customer_num (Serial, fragments: ALL) Upper Index Filter: miller3.customer.customer_num < 102 2) miller3.mycollection: INDEX PATH Filters: informix.equal(BSON_VALUE_LVARCHAR (miller3.mycollection.data , 'fname' ) ,miller3.customer.fname ) (1) Index Name: miller3.mycollection_ix1 Index Keys: :informix.bson_value_lvarchar(data,'lname') (Serial, fragments: ALL) Lower Index Filter: equal(BSON_VALUE_LVARCHAR (miller3.mycollection.data , 'lname' ) ,miller3.customer.lname ) NESTED LOOP JOIN 137 © 2013 IBM Corporation
  • 138. Understanding Informix BSON Indexes Indexes are created on BSON data and support – – – – Primary Key (enforced across all nodes) Unique Indexes (enforced at a single node level) Composite Indexes ( 8 parts ) Arrays { "fname":"Sadler", "lname":"Sadler", "company":"Friends LLC", "age":21, "count":27, “phone": [ “408-789-1234”, “408-111-4779” ], } create index fnameix1 on customer(bson_value_lvarchar(bson,"fname")) using bson; create index lnameix2 on customer(bson_value_lvarchar(bson,"lname")) using bson; create index phoneix3 on customer(bson_value_lvarchar(bson,"phone")) using bson; 138 © 2013 IBM Corporation
  • 139. Understanding Informix BSON Indexes create index fnameix1 on customer(bson_value(bson,"fname")) using bson; create index lnameix2 on customer(bson_value(bson,"lname")) using bson; create index phoneix3 on customer(bson_value(bson,"phone")) using bson; select * from customer where bson_value_lvarchar(bson,"fname") = "Ludwig"; -- use fnameix1 select * from customer where bson_value_lvarchar (bson,"lname") = "Sadler"; -- use lnameix2 select * from customer where bson_value_lvarchar(bson,"phone") = "408-789-8091"; -- use phoneix3 select * from customer where bson_value_lvarchar(bson,"phone") = "415-822-1289“ OR bson_value_lvarchar(bson,"phone") = "408-789-8091"; -- use phoneix3 select * from customer where bson_value_lvarchar(bson,"company") = "Los Altos Sports"; -- no index use sequential scan 139 © 2013 IBM Corporation
  • 140. Informix 12.1 Convert Traditional SQL Tables to JSON Mongo API –Leverage the numerous open source drivers for Mongo – Extend the power of the mongo API to execute on • Traditional SQL tables • TimeSeries/Sensor Data 140 © 2013 IBM Corporation © 2013 IBM Corporation
  • 141. Not just a Hybrid Database Hybrid databases only solve half of the problem Hybrid application are hear to stay
  • 142. Scaling Out – Sharding Data Shard Key state= “CA” Shard by either hash or expression When inserting data it is automatically moved to the correct node • 142 Each node in the environment hold a portion of the data • Shard Key state= “OR” • • Shard Key state= “WA” Queries automatically aggregate data from the required node(s)
  • 143. Graphical Administration showing a Sharded System Graphical administration of the shards keep things simple Monitor the entire environment from a single graphical console
  • 144. Simply Administration - OAT Manages JSON Objects 144
  • 145. Simplify the “up and running” Experience Install and setup a typical database instance with only 3 questions: − Where to place the Product? − Where to place the Data? − How many users do you anticipate? Newly installed instance adapts to the resources on the computer Description Auto tuning of CPU VPS Auto Table Placement Auto Buffer pool tuning Auto Physical Log extension Auto Logical Log Add Auto Read Ahead 145
  • 146. SIMPLE POWER – INFORMIX HYBRID DATABASE CAPABILITIES
  • 147. Relational and non-relational data in one system NoSQL/MongoDB Apps can access Informix Relational Tables Distributed Queries Multi-statement Transactions Enterprise Proven Reliability Enterprise Ready Security Enterprise Level Performance Informix provides the capability to leverage the abilities of both relational DBMS and document store systems. MongoDB does not. It is a document store system lacking key abilities like transaction durability. 147
  • 148. Round Peg Square Hole The DBA and/or programmer no longer have to decide upfront if it is better to use a system entirely of JSON documents or a system only of SQL tables, but rather have a single database in which the programmer decides a if JSON document is optimal or a SQL table is optimal for the data and usage. 148
  • 149. Informix Specific Advantages with Mongo Drivers Traditional SQL tables and JSON collections co-existing in the same database Using the MongoDB client drivers Query, insert, update, delete − JSON collections − Traditional SQL tables − Timeseries data Join SQL tables to JSON collections utilizing indexes Execute business logic in stored procedures Provide a view of JSON collections as a SQL table − Allows existing SQL tools to access JSON data Enterprises level functionality 149
  • 150. Scalability Better performance on multi-core, multi-session scenarios − Architecture has finer grain locking – not entire database − Better concurrency because less resources locked Document Compression − 60% to 90% observed Bigger documents – 2GB maximum size − MongoDB caps at 16MB Informix has decades of optimization on single node solution 150 © 2013 IBM Corporation
  • 151. Some NoSQL Use Cases - Mostly Interactive Web/Mobile Online/Mobile Gaming − − − − − Leaderboard (high score table) management Dynamic placement of visual elements Game object management Persisting game/user state information Persisting user generated data (e.g. drawings) Display Advertising on Web Sites − − Ad Serving: match content with profile and present Real-time bidding: match cookie profile with ad inventory, obtain bids, and present ad Dynamic Content Management and Publishing (News & Media) − − 151 Store content from distributed authors, with fast retrieval and placement Manage changing layouts and user generated content E-commerce/Social Commerce – Storing frequently changing product catalogs Social Networking/Online Communities Communications – Device provisioning Logging/message passing – Drop Copy service in Financial Services (streaming copies of trade execution messages into (for example) a risk or back office system)
  • 152. Dual Capability Hybrid databases are a way of the past Hybrid application are hear to stay 152
  • 153. Client Applications Applications native Client All three drivers support all data models Informix SQLI Drivers IBM DRDA Drivers MongoDB Drivers web browser Informix NoSQL Mobile © 2013 IBM Corporation
  • 154. Ability for All Clients to Access All Data Models Informix SQLI Drivers IBM DRDA Drivers MongoDB Drivers Traditional SQL NoSQL - JSON TimeSeries MQ Series 154
  • 155. Ability for All Clients to Access All Data Models node.js Application Traditional SQL MongoDB Drivers NoSQL - JSON TimeSeries MQ Series 155
  • 157. BUILDING A REAL LIFE APPLICATION
  • 158. IOD Demo and Beyond A fun exciting demo application that attendees can interact with that makes people take notice of IBM technologies A social networking photo application utilizing smart phones/tablets that allows the end user to take photos, tag and add photos to an Informix NoSQL hybrid database Random photos can be displayed at sessions, booths or on attendee smart devices
  • 159. IOD Attendee Photo Application Allow conference attendee to take and share photo!
  • 160. Technology Highlights • Create a hybrid application using NoSQL, traditional SQL, timeseries mobile web application • • • Utilizing both JSON collections, SQL tables and timeseries Utilize IBM Dojo Mobile tools to build a mobile application Leverage new mongo client side drivers for fast application delvelopment • Demonstrate sharding with over 100 nodes • Cloud based solution on top of Amazon Cloud • Provide real-time analytics on all forms of data • • Leverage existing popular analytic front-end IBM-Congos Utilize an in-memory columnar database accelerator to provide real-time trending analytics on data
  • 161. Mobile Device Application Architecture Apache Web Server IOD Photo App - UPLOAD Photo Application IBM Dojo Mobile Informix tag Van Gogh Informix JSON Listener Photo collection User Table
  • 162. Application Architecture (Big Picture) Apache web server IOD application Informix JSON Listener Informix Photo collection Amazon Cloud Sharding commands Node1 Aa-Am Node2 An-Az Node3 Ba-Bf Node3 Bg-Bp Node101 “Dalmation” shard farm – 101 sharded nodes on Amazon Cloud Photo collection sharded across these nodes
  • 163. Photo Application Schema TimeSeries NoSQL Collections activity_photos activity_data timeseries(photo_like)
  • 164. Photo Application NoSQL Collections activity_photos activity_data timeseries(photo_like)
  • 165. Application Considerations Photo meta-data varies from camera to camera A Picture and all its meta data are stored in-document Pictures are stored in a NoSQL collection Pre-processing on the phone ensures only reasonable size photos are sent over the network.
  • 166. Example of Live JSON Photo Data {"_id":ObjectId("526157c8112c2fe70cc06a75"), "Make":"NIKON CORPORA TION","Model":"NIKON D60","Orientation":"1","XResolution":"300"," YResolution":"300","ResolutionUnit":"2","Software":"Ver.1.00 ","D ateTime":"2013:05:15 19:46:36","YCbCrPositioning":"2","ExifIFDPoi nter":"216","ExposureTime":"0.005","FNumber":"7.1","ExposureProgr am":"Not defined","ISOSpeedRatings":"100", "Contrast":"Normal","Saturation":"Normal","Sharpness":"Normal", "SubjectDistanceRange":"Unknown","name":"DSC_0078.JPG","img_d ata":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABcQ
  • 167. Sharding NoSQL “Dalmatian” cloud deployment Sharding by hash Heterogeneous Cloud deployment with sharding − Processor Type and Number Intel and ARM processors − Operating System Types and Versions Linux, Windows, AIX, Apple, Solaris, HP,
  • 168. Basic PHP Programming Overview Information List of NoSQL collection names and SQL tables names Function to set the active database and return the Collection private $conn; private private private private private private private $dbname = "photo_demo"; $photoCollectionName = "photos"; $contactsCollectionName = "contacts"; $sqlCollectionName = 'system.sql'; $userTableName = "users"; $tagsTableName = "tags"; $likesTableName = "likes"; private $photoQueryProjection = array("_id" => 1, "tags" => 1, "user_id" => 1, "img_data" => 1); /** * Get collection by name * @param MongoCollection $collectionName */ private function getCollection($collectionName) { return $this->conn->selectDB($this->dbname)->selectCollection ($collectionName); }
  • 169. Insert Example Information is placed in the contacts collection
  • 170. Insert Data into a Collection Very simple to insert JSON data into a collection using the MongoAPIs /** * Insert user's contact information into contacts table. */ public function insertContact( $json ) { if (! is_array ( $json )) { return "Contact info not in JSON format."; } try { $result = $this->getCollection($this->contactsTableName)->insert($json); if ($result ["ok"] != 1) { return $result ["err"]; } } catch ( MongoException $e ) { return $e->getMessage (); } return "ok"; }
  • 171. Retrieve Collection Information Very simple to retrieve data from a collection using the MongoAPIs Data is returned as a JSON document /** * Get contact info */ public function adminContacts() { $photoCollection = $this->getCollection($this->contactsCollectionName); $cursor = $photoCollection->find(); $results = $this->getQueryResults($cursor); return $results; }
  • 172. Retrieve Collection Information /** * Get contact info for admin panel */ public function adminContacts() { $photoCollection = $this->getCollection($this->contactsCollectionName); $cursor = $photoCollection->find(); $results = $this->getQueryResults($cursor); return $results; }
  • 174. Delete a Photo and its Information Deleting from SQL Tables and NoSQL Collection is exactly the same /** * Delete photo */ public function deletePhoto($id) { try { // First delete from likes and tags tables $query = array('photo_id' => $id['_id']); $result = $this->getCollection($this->likesTableName)->remove($query); if ($result ["ok"] != 1) { return $result["err"]; } $result = $this->getCollection($this->tagsTableName)->remove($query); if ($result ["ok"] != 1) { return $result["err"]; } // Then delete the photo from the collection $query = array('_id' => new MongoId($id['_id'])); $result = $this->getCollection ( $this->photoCollectionName )->remove ( $query ); if ($result ["ok"] != 1) { return $result["err"]; } } catch ( MongoException $e ) { return $e->getMessage(); } return "ok"; }
  • 175. Executing a Stored Procedure in MongoAPI /** * Get the user_id for a particular user name (email address). * * Calls a stored procedure that will insert into the users table if the * user does not exist yet and returns the user_id. * * @param string $username * @return int $user_id */ public function getUserId($username) { $username = trim($username); try { $sql = "EXECUTE FUNCTION getUserID('" . $username . "')"; $result = $this->getCollection($this->sqlCollectionName)-> findOne(array('$sql'=>$sql)); if (isset($result['errmsg'])) { return "ERROR. " . $result['errmsg']; } return $result['user_id']; } catch (MongoException $e) { return "ERROR. " . $e->getMessage(); } }
  • 176. Real Time Analytics Customer Issues − Several different models of data (SQL, NoSQL, TimeSeries/Sensor) − NoSQL is not strong building relations between collections − Most valuable analytics combine the results of all data models − Most protonate Analytic system written using standard SQL − ETL & YAS (Yet Another System) Solution Provide a mapping of the required data in SQL form − Enables common tools like Congo's
  • 177. Analytics on a Hybrid Database Photo collection SQL MongoAPI Informix User Table
  • 178. Photo Application SQL Mapping of NoSQL PHOTO Collection activity_photos activity_data timeseries(photo_like)
  • 179. Mapping A Collection To A SQL Table CREATE VIEW photo_metadata (gpslatitude, gpslongitude, make, model, orientation, datetimeoriginal, exposuretime, fnumber, isospeedratings, pixelxdimension, pixelydimension) AS SELECT BSON_VALUE_LVARCHAR ( x0.data , 'GPSLatitude' ), BSON_VALUE_LVARCHAR ( x0.data , 'GPSLongitude' ), BSON_VALUE_LVARCHAR ( x0.data , 'Make' ), BSON_VALUE_LVARCHAR ( x0.data , 'Model' ), BSON_VALUE_LVARCHAR ( x0.data , 'Orientation' ), BSON_VALUE_LVARCHAR ( x0.data , 'DateTimeOriginal' ) , BSON_VALUE_LVARCHAR ( x0.data , 'ExposureTime'), BSON_VALUE_LVARCHAR ( x0.data , 'FNumber' ), BSON_VALUE_LVARCHAR ( x0.data , 'ISOSpeedRatings' ), BSON_VALUE_LVARCHAR ( x0.data , 'PixelXDimension' ) , BSON_VALUE_LVARCHAR ( x0.data , 'PixelYDimension') FROM photos x0;
  • 180. IWA Integration What is trending at IOD? Set up to work on frequently updated snapshots of data to perform analytic queries on the photo meta-data Since BSON/JSON data can not be directly compressed and searched, snapshot process would selectively extract data from the documents for copying to IWA Use Genero to show the power of IWA
  • 181. Mobile Device Application Architecture Photo collection SQL MongoAPI Informix User Table
  • 182. Configure Informix on Amazon Cloud Simple • • • • • Instantiate the Amazon image Setup the storage Install the product Start the system Configure sharding