More Related Content Similar to The role of NoSQL in the Next Generation of Financial Informatics (20) More from Aerospike, Inc. (11) The role of NoSQL in the Next Generation of Financial Informatics1. The Role of NoSQL
In Next Generation Financial Informatics
December 10, 2015
2. 2
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Market Overview
■ The role of NoSQL in Financial Services
■ Customer Use Cases
■ Fraud
■ Order Management
■ How you can achieve Speed at Scale
Agenda
4. 4
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Hours - Days
Analytics Store
Operational Store
(Landing Zone – Fast Ingest)
MarkLogic
ElasticSearch
Deep Archive
Apache Storm
Apache Spark
Apache Ignite
Project Heron
Ingest/
Gateway
Tibco EMS
JMS
60 East
29 West
TCP
FTP
IBM MQ
Data
Warehouse
(Legacy)
Teradata
DB2 WH
Actian(ParAccel)
Pivotal
Astor Data/TD
Neteeza/DB2 WH
Exadata
Mark Logic
Veritca
Data Lake
(100+ Petabyte)
Tiered External Storage
Flash Arrays to
Warm/Cold/Frozen
Hadoop
MapReduce
Redis
IMDGrid
Hazelcast
Aerospike
Graph Db
Large Footprint
Front Office
Real-Time Processing
On The Wire “Fast Data’
Hours Days-Years
No SQL
HBase
MongoDB
Cassandra
Aerospike
Time Series
ETL
In Memory
Cache
Data Operating System – YARN, MESOS
Deeper Analysis
World Class Big Data Ecosystem: “Service Oriented Information”
HDFS Interface
Small Footprint
Surveillance Trading
Analytics
Metrics,
Monitoring and
Alerts
Search ComplianceRT Discovery &
Visualization
Oktay Technology LLC Confidential
RAM-first
Storage
Flash-enabled
Storage
SQL-Based
MemSQL
Volt DB
Kafka
In Memory
Fil;esystem
Tachyon
IgniteFS
6. 6
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
2010
Aerospike 2.0 deploys with first at-scale customers
2011
Funding, company launch
2013
Aerospike 3.0 adds indexes, analytics integration
2014
Open Source
2015
Deployments in Financial Services, Telecom, etc
Aerospike History
7. 7
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Modern Scale Out Architecture
Research
Warehouse
Long-term cold storage
HDFS BASED
App Servers
Fast, stateless
Load Balancer
Simple, stateless
High Performance NoSQL
Operational Key Value
Session, authentication, account status, cookies, deviceID,
IP address, location, segments, trades, debits, billing,
prices...
Real-Time Decisions
Best sellers, top scores, trending tweets
8. 8
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Powering Pioneers Across the Internet
Mobile
Advertising
Omni-Channel
Marketing
Search, Video, Social,
Gaming
Web Advertising
9. 9
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
1 Million Writes/Sec on Google Compute
New results: 20 nodes, and 4M reads per second
• Aerospike hits 1M writes/sec with 6x fewer servers than Cassandra
• Delivers consistent low latency with no jitter for both read and write workloads
10. 10
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Shared-Nothing System, High Availability
• Every node in a cluster is identical, handles both transactions and long running tasks
• Clients give rich semantics in multiple languages and connect over the network
• Data is replicated synchronously with within the local cluster
• Data is replicated asynchronously across data centers
• Primary key hash RIPE MD160 ( 20 byte ) for extreme collision resistance ( DHT ), Red black
tree for records within hash bucket
• Scatter-gather B+ secondary index
12. 12
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Fraud Prevention
4% of online transactions are fraudulent
False positives cost millions in lost sales
Global criminal agents
New opportunities:
Data sources
Algorithms
Speed at Scale
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
13. 13
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Operational Scale Problem
LEGACY DATABASE
(Mainframe)
XDR
Decision Engine
DATA WAREHOUSE/
DATA LAKE
LEGACY RDBMS
HDFS BASED
BUSINESS
TRANSACTIONS
Request
for
Payment
( Mobile Queries ) (
Recommendation )
( And More )
High Performance NoSQL
500
Business Trans per sec
5000
Calculations per sec
X =
“REAL-TIME BIG DATA”
“DECISIONING”
2.5 M
Database Transactions per sec
14. 14
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Advanced Fraud Scoring
Anything can be a “feature”
POS type, item sold, price of item, time of day for given items
Browser type, screen resolution, item + browser
But you’ll need recent data
And cutting edge libraries
( your private algorithms )
15. 15
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Example IP Addresses
2B to 4B addresses
Unknown number of networks
Different risk models for different
network ranges
Recent behavior required
16. 16
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Example – IP based scoring
ID START_IP END_IP Score
1 192.168.4.1 192.168.4.26 20
2 192.168.4.7 192.168.17.0 2
SELECT id, score WHERE START_IP >= this_ip AND END_IP <= this_ip
Not so bad!
Too slow, and
too simple
17. 17
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Using SQL
ID IP TIMESTAMP BEHAVIOR
1 192.168.4.1 2015-06-02 12:00 transaction
2 192.168.4.7 2015-06-02 12:01 tweet
Add the most recent 5 minutes of traffic for each IP address and lookup by previous network
INSERT ip, time, behavior
FOR ( network )
SELECT behavior WHERE ip <= network.start_id AND
ip >= network.end_id AND
timestamp > now - 600
You can do it
in SQL!
Too slow, call
the DBAs.
18. 18
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
With a NoSQL KVS
key Network list
192.168.4.0 [ net1, net2, net3 ]
192.168.5.0 null
ID START_IP END_IP Score
net1 192.168.4.1 192.168.4.26 20
net2 192.168.4.7 192.168.17.0 2
Key on Class C address
List of networks
Batch lookup networks
Filter on client
That’s messy.
Two network round
trips, it’ll go fast.
19. 19
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
With a NoSQL KVS – add behavior
Already have filtered list of networks
For each network, insert into capped or time limited collection
key Network list
192.168.4.0 [ net1, net2, net3 ]
192.168.5.0 null
ID START_IP END_IP Score behavior
net1 192.168.4.1 192.168.4.26 20
{ collection
}
net2 192.168.4.7 192.168.17.0 2
{ collection
}
That’s really
messy! It’s fast and easy
to explain.
20. 20
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Use Cutting Edge Libraries in Your Favorite App Environment
If it’s in SQL, it’s too late
21. 21
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Modern Scale Out Architecture
App Servers
Fast, stateless
Load Balancer
Simple, stateless
High Performance NoSQL
Operational Key Value
Session, authentication, account status, cookies, deviceID,
IP address, location, segments, trades, debits, billing,
prices...
Real-Time Decisions
Best sellers, top scores, trending tweets
Scale and Compute
Data
Enabled by Fast
Networks
Least Vendor Lock-in
23. 23
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Financial Services – Order Management Systems
Challenge
• Must update stock prices, show balances on 300
positions, process 250M transactions, 2M updates/day
• Scale-out requirement
• Multi-asset aggregation for combined:
Trading
Risk
Surveillance
Customer
• Real-time data view and computation
Speed at Scale
24. 24
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Financial Services – Order Management Systems
Individual data silos
• No global, integrated data view
• No simple post-order visibility
• Often built on non-scalable technologies
• Little shared operational infrastructure
DB1 DB2 DB3
Order Management
System
Order Management
System
Order Management
System
FIXED INCOMEEQUITIES DERIVATIVES
25. 25
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Financial Services – Order Management Systems
Shared Data OMS
•True multi-asset aggregation
•Supports real-time risk models
•Immediate customer visibility
of order state
•Natural NoSQL data model
including parent-child and fills
•Microsecond performance with
persistence
•Scale-out architecture
Order Management
System
Order Management
System
Order Management
System
FIXED INCOMEEQUITIES DERIVATIVES
26. 26
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Financial Services – Order Management Systems
Hybrid order management
•Existing order management code
unchanged
•Order states placed onto message bus
•Recorded in unified NoSQL storage
•Achieve unified multi-asset aggregated
view
•Storm and Spark integration
•Beyond RAM-based processing solutions
DB1 DB2 DB3
Order Management
System
Order Management
System
Order Management
System
FIXED INCOMEEQUITIES DERIVATIVES
MESSAGE BUS (KAFKA, etc)
28. 28
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Multiple Engagements
10B to 50B objects
300 T before HA replication
2 M TPS
50 milliseconds per 5,000 queries
Local and remote DR
29. 29
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
1 Million TPS on Flash per server
Options for storage on a database before Aerospike:
RAM, which was fast, but allowed very limited storage
Disk, which allowed for a lot of storage, but was limited in speed
Intel achieved 1M TPS using 4 Intel P3700 SDs with 1.6 TB
capacity on a single Aerospike server. The cost per GB is a
fraction of the cost of RAM, while still having very high
performance.
30. 30
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Aerospike Has the Experience
Flash optimized from the beginning
5 9’s reliability
( longest running cluster: 5 minutes of outage in 4.5 years )
Multi-million TPS clusters in production today
Enterprise Support of your POC & NoSQL design
Used in financial services, telecom, gaming
( and adtech of course )
32. 32
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
High Performance
NoSQL Database
Powering New
Opportunities at Scale
@aerospikedb
NEXT STEPS:
See how much you can save with Aerospike:
http://www.aerospike.com/tco-calculator/
Ready to get started?
http://www.aerospike.com/quick-start/
If you have any questions or want to further
explore if Aerospike is right for you, contact
us:
info@aerospike.com
Editor's Notes Key Points:
All of the above challenges
If you use Relational, must use cache and you compromise the value of RDBMS = Consistency & durability issues as well