1. My mainframe is my business.
DataKinetics Products My business relies on MIPS.
May 5, 2009
2. Our customers
Our products work so well, no mainframe customer has ever replaced them!
and we’ve been in business over 30 years
2
3. Why they chose DataKinetics
Financial • Settle 100M transactions in one day
• Make a credit card process that is market adaptive
• Reduce 5 systems for each of 5 card types to 1 system
with rules for each market – net effect is market
adaptivity and the ability to create 100s of card types
• Create personalized bank statements with net worth
• One time conversion of local currency to Euros; millions
of current and historical records converted in three hours
Insurance & Healthcare
• Have 10000 – 15000 tables in total, 5000 live
• Eliminate 27M I/O’s in 24 hours
• Consolidate 48 systems into 1
Retail
• Monitor sales by item by store in real time
• Very fast access to price lists and product descriptions
3
4. Barriers to performance and flexibility
What slows data down?
• Moving data from disk to memory and back again
• Looping serially through information to find the piece you want
• Waiting for one transaction to complete before starting another
• Repeated retrieval of identical information
What slows down application delivery?
• Relearning complex logic in order to enhance or repair
• Updates that need to be propagated to many programs
• Comprehensive testing cycles
5. Make the most of your MIPS
Increase performance and flexibility – decrease maintenance
• Access data from memory
• 1000 times faster than accessing from disk*
• Easily determine the most effective search methodology
• Replace sorts with virtual sorts
• Replace temporary files with temporary tables
• Place read only reference data (such as price lists) in in-memory tables
• Use in-memory tables as a write through cache
• Use in-memory tables as a message queue
• Use process control tables to decouple processes
• Replace logic trees with decision tables
Our customers have
• Reduced CPU by up to 40%
• Reduced program execution time from hours to minutes
• Reduced the time to implement changes from months to hours
5 *Philip E, Courtney in Z / Journal Feb/Mar 2003 p 13
6. DataKinetics Products
Current Products
tableBASE® - increase performance, flexibility, time to market
tablesONLINE – interactive tool to browse, edit and define tables
Virtual Table Space - share tables among applications
Process Manager – coordinates shared tables, even across LPARs,
providing 7/24/365 transaction processing
Future Products
netTables® – extending the performance enhancing capabilities of
tableBASE® to the distributed environment
7. tableBASE®
In-memory tables
• Define autoload and index tables
• Tables have a fixed row length
• Each row contains a key and structured data of variable format
• Key can be multiple fields
• Data can be values, instructions, locations, rules or decisions
• Create alternate indices on the fly
• Use alternate indices as a virtual sort
• Optimize table search
• Options for table organization and table search
• Extensive set of table manipulation commands
• Security
• tablesONLINE - tool to browse, edit and define tables
•
tableBASE® - define, build, maintain and manage in-memory tables
7
8. Unique to tableBASE
tableBASE® tables support many data formats
• IMS hierarchies
• Header-trailer relationships
• Relational tables
• Grouping of multiple rows from different tables with different formats
into a single table (such as tariff tables which have different formats for different countries)
• Each row can have a different format, with a common key
• Key can be multiple fields
• Rows can be extended
• Data can be values, instructions, locations, rules or decisions
8
9. When to use tableBASE®
To increase performance by eliminating I/Os reading in data more than
once
• Assembling data from multiple sources into one temporary table
for subsequent processing or rendering
• Place reference data and rules in read only tables
• Replace sort with a virtual sort using alternate indices that can be
defined on the fly
• Replace logic trees with decision tables
• Use tables as a write through cache
To reduce time to deliver new applications or updates and reduce
maintenance
• Rules in tables and decision tables are more easily updated
reducing both effort and elapsed time.
• Reduce complex logic
9
10. Reduce I/O, reduce elapsed time
• Each request for data can result in one or
DB2 Calling App
more I/O transactions
CICS
• DB2 can buffer the data, but it still needs to
be reformatted
• DB2 brings back a block, data is extracted
Temporary file
and reformatted
LPAR
• First call for data loads the table, creates
Calling App
index
DB2
CICS • All other calls access in-memory tables
• Returns all data
TSR
• Create alternate indices on the fly
• Virtual sorts using alternate indices
LPAR • Optimize search method
• Rows in table can have different formats
With tableBASE®, the path to data is shorter and much much faster
10
11. How much faster?
CPU time Number per second
Sequential browse Uses 83 to 90% less CPU time 6 to 9 times faster
to read 1M rows up to 781250 reads
Random access hash Uses 93 to 96% less CPU time 13 to 23 times faster
to read 1M rows up to 909091 reads
Random access to Uses 88 to 95% less CPU time 8 to 19 times faster
binary sequential tables to read 1M rows up to 819664 reads
Populating tables Uses 93 to 97% less CPU time 14 to 30 times faster
for 1M inserts up to 662252 inserts
• Tests conducted in IBM zSeries Benchmarking Center comparing DB2 with tableBASE®
• Range in data results reflects differences in tableBASE table organization and
tableBASE® search method; tableBASE® is consistently faster
• Demonstrates the performance optimization possibilities when data is accessed
from tableBASE® as compared to accessed from RDBMS
• In the real world, one of our customers states tableBASE is 25 times faster
11
12. How a bank uses tableBASE®
Populate table in memory with account data
Line of
credit
Financial App
Render contents via virtual sorts
Savings and
chequing
Loans TSR Provide a summary of their net worth at that
moment
Credit card
Insurance Based on what services the customer has,
Investments
provide offers for new services.
Mortgages
And it’s all done in tables.
•Save CPU
Browser •Save I/Os
Statement •Save time
Create temporary tables to gather data for subsequent processing
13. How insurance customers use tableBASE®
Insurance companies have rate tables based
Insurance App on many variables – location (cities, regions,
Rates
provinces/states), demographics (age, sex,
TSR
smoking, driver training) and many more.
Claims DB
Keeping the data in read only tables means:
Customer •Access at the speed of memory
information •Separation of data from programming logic
Read once, keep in memory
•Save I/Os
Agent Customer •Save time
•Save maintenance
Create read only tables of rules, reference data, price lists or
product descriptions
13
14. Replace logic trees with decision tables
The set of conditions becomes the key for
searching the table.
Conditions Actions
Rather than searching through data each
in
time to find a match on each parameter,
ping m
dit OK
mer
ping
mit
mgr
search only once.
pt
pt
l custo
li
< ship
al ship
dit de
dit de
ipping
credit
al cre
rder When there is a match, the table can be set
Specia
Stop o
To cre
To cre
Order
Manu
Manu
To sh
Not >
up to return a program name, or an action, or
Y - - - Y X a set of variables.
Y - - Y N X
Y - - N N X
N Y - N N X Extract application complexity
N N Y Y N X •Save CPU
N N Y N N X
N N N - - X
•Save processing time
•Increased flexibility
•Reduce maintenance
•Reduce testing
Reduce a large sequence of “if, then, else” statements to a
single table lookup
14
15. Process control tables
Program A v1 v2 v3 Program E v4 v5 v6
Program B v3 v8 v9 Program A v12 v13 v14
Program C v3 v4 v5 Program D v7 v8 v9
Program D v15 v16 v17 Program F v10 v11 v18
The key The outputs
Program a sequence of steps and store those steps in a table. Pass program
names and variables as part of outputs. Decouple processes from application
logic.
Table becomes a program switching table, and search becomes very fast
Create a program switching table
•Save processing time
•Save maintenance, no increase in complexity
•Enables new wide range of opportunities
16. tablesONLINE
Flexible, customizable, interactive front end to tableBASE enabling users to create,
update, manipulate, test and process data tables
• Elegant solution to manage and edit tables and data in CICS (simplified version available for ISPF)
• Supports multiple formats of rows
• Many to many editor - can have multiple views and multiple tables
• Integrated access controls and data validation
• User interface programmed by editing tables
• Stores metadata in tableBASE tables
• Rules of how to re-structure tables are captured in the process of editing views, and rules stored in a table
for re-use in restructuring data
Menu – can program
sequence of events at View – has field name,
menu items; identifies the the order and how it should
objects (programs,
be displayed; defines what
transactions, tables or can be edited in the table
views)
Table – row formats can be different
Menu information is stored
in a tableBASE® table View information is stored
in a tableBASE® table
Example of how tableBASE® stores tablesONLINE can create the table used by an
instructions in tables that determine application, or edit a table created by an application
subsequent program actions
17. When to use VTS
To increase performance by eliminating the loading of common tables into
each application region
• When each copy of the application uses the same data in a local TSR
• When there are many tables to be loaded into a local TSR on
initialization
• If the data that is needed can persist (for example business rules) and
be retained in a VTS-TSR
To ensure all copies of the application have the exact same data
To share data between different applications
To overcome processing bottlenecks between applications
To facilitate application integration after a merger or acquisition
• Merging business rules
• Data consolidation and data translation
• Consolidation of outputs
17
18. Increase performance
tableBASE® with local TSRs tableBASE™ with VTS-TSRs
Address Calling App
Address Calling App
space TSR space
CICS/TS CICS/TS
Calling App Calling App V-TSR
TSR
CICS/TS CICS/TS
Calling App Calling App
TSR CICS/TS CICS/TS
LPAR LPAR
Reduce memory
Replace multiple copies of the same data Reduce paging
in local TSRs with a shared VTS-TSR Consistency of data
18
19. Share data between applications
tableBASE® with VTS-TSRs
Address Calling App
space
CICS/TS Ideal for sharing business rules,
reference data, processing rules
and decision tables
Calling App VTS-TSR
IMS TM
Calling App
VTS-TSR
Batch
LPAR
Eliminate the need to synchronize multiple copies of the data
Coordinate the actions of other applications
19
20. Eliminate processing bottlenecks
• If the order entry application is faster than
Calling App VTS-TSR order processing, the order processing can
Order become a bottleneck
Entry
• Set up a VTS-TSR to capture the orders, and
the order processing applications process the
order and delete it from the list with a special
read
Calling App
VTS-TSR • An output of the order processing application
Calling App
Order may be to update inventory, which may also be
Processing
Order an input to either order entry or order
Processing
processing
• The bottleneck of order processing is eliminated
• The table in the VTS-TSR becomes the order
queue
Calling App
Inventory
Create a circular message queue,
Completed add new items to the bottom,
orders
take items off the top
20
21. Mergers and Acquisitions – Report Integration
• Populate the TSRs with the data structure
each company uses – without changes –
Data from
into a single table
company A • Use tableBASE® indices and alternate
indices to transform the data from two
VTS-TSR
companies that have different values and
structures into a structure that can be
used to create integrated reports –
Calling App
Data from Report without moving the data
company B writer
• Underlying applications that are producing
the data do not have to change
May arrive as a flat • The report writer application would need
file, or in other to change to access the data from
formats
tableBASE ® tables
Create consolidated reports without any changes to the underlying
programs, and with minimal changes to a report writing program
22. Mergers and Acquisitions – Business Rules Integration
Data from Data from
Calling App Calling App
company A company B Company
Company
A B
Each company has its own programs, rules, data and data structure
• Create common application, with rules for each
Data from Calling App
Company
company in tableBASE® table(s)
company A
A&B • Data loaded into more tableBASE® tables
VTS-TSR
• Underlying data structure from each company
does not need to change – restructuring can
VTS-TSR
Data from be done with alternate indices
company B • Application applies the appropriate rules to the
appropriate data based on decision tables
One application, with harmonized rules, with existing data and data structure
22
23. When to use Process Manager
Seamlessly support 7/24/365 operations with no disruption to
transaction processing
• Switch all applications to use new data simultaneously
Share tables across LPARs
To simplify the deployment of new software from a test to a
production environment
To decrease the time it takes to load tables on initialization
To coordinate the start up and shut down of shared tables
23
24. Decrease initialization time by mapping to LDS
Process
Manager
• With Process Manager, map VTS-
VTS-TSR
VTS VTS TSR to a VSAM LDS, and on
Manager Manager
initialization, the tables are reloaded
from the LDS, rather than loading the
tables and building the indices.
Calling App Calling App
• Eliminates row by row loading of
tables from a tableBASE® library
VTS-TSR VTS-TSR
LDS
Initialization becomes instantaneous
24
25. Coordinate start up and shut down
VTS-TSR
Process
Manager
• When VTS-TSR A, B and C all
need to be started up together or
VTS
shut down together, Process
VTS
A VTS-TSR
Manager Manager Manager can eliminate the
manual steps
• When combined with the ability to
Calling App map VTS-TSRs to LDS, all the
Calling App
data can be loaded and in place
at a specific pre-determined time
VTS-TSR
VTS-TSR
B
C
LDS
Eliminate manual start up and shut down of VTS-TSRs
25
26. Seamlessly switch all users to new data
Calling App VTS-TSR
Process • When there is new data (such as a new
A
Manager
rate table) that all applications are to
VTS-TSR use at a specific time, Process Manager
Calling App
VTS
Manager
makes a change to the alias table so
that the application has the new VTS-
B VTS-TSR
TSR name
• All new transactions use the new data
• All in-process transactions complete
VTS-TSR
using the existing data
• The switch is instantaneous with no
disruption to ongoing transactions
New transactions use the new data, existing transactions complete
using previous data
26
27. Simplify deployment
Process
Manager
One of the challenges of testing
new software is creating a test
VTS
VTS environment identical to the
Manager
Manager production environment.
With Process Manager and VTS
Production Environment Test Environment
you can:
Calling App Calling App
VTS-TSR
VTS-TSR
• Ensure the data is identical for both
test and production
Production Production
• Use the exact same names for the
Production New software
software
VTS-TSRs in both the test and
Test Environment production environments
Calling App VTS-TSR
• The production environment and
each test environment must be
managed by different VTS
New software New data managers to use the same names
formats
Test Environment
Calling App
Testing environment can use
the exact same names as the
VTS-TSR
production environment
Production New data
27 software formats
28. Share tables across LPARs
Calling App
VTS-TSR
Calling App
• Share read-only tables across LPARs
• Can share across any number of
LDS LPAR1
machines as long as they share DASD
Calling App
VTS-TSR
Calling App
LPAR2
Ensures data consistency and reduces memory usage
28
29. Make the most of your MIPS with DataKinetics
tableBASE® – increase performance, flexibility, time to market
Replace numerous I/Os with in-memory tables
Reduce CPU (by up to 40%)
Reduce program execution time (from hours to minutes)
Reduce maintenance costs to implement changes (from months to days)
Replace logic trees with decision tables
tablesONLINE – tool to browse, edit and define tables
Incorporates use of process tables in our own product
Virtual Table Space - share tables amongst applications
Eliminate multiple copies of data
Improve performance by decreasing loading times
Improve operations by eliminating need to coordinate changes to multiple copies of data
Improve operations by eliminating errors caused by differences amongst the multiple copies of data
Address bottlenecks between processing applications
Facilitate merging of operations from M&A
Process Manager – coordinates VTS-TSRs, even across LPARs
Improve operations
Synchronize simultaneous use of new data
Coordinate start up and shut down
Simplifies testing of new software and new data
Improve loading times using VSAM LDS
29
30. DataKinetics helps you get more for your MIPS
• Phenomenally fast
• Flexible
• Reduce time to market
• Reduce maintenance
Our customers have
• Reduced program execution time from hours to minutes
• Reduced CPU by up to 40%
• Reduced the time to implement changes from months to days
30
31. Thank you for your time.
+1-613-523-5500 x212
azander@dkl.com
31
32. Why our financial customers chose DataKinetics
Financial
• Process 100M transactions in one day
• Make a credit card process that is market adaptive
• Reduce 5 systems for each of 5 card types to 1
system with rules for each market
• Collected all information from ATMs, moving money
from one account to another, needed to have
processing complete within a window to collect daily
interest on $3B in assets – batch window went from
hours to 30 seconds
• Collect information from 24 stock exchanges on
global company and subsidiaries to determine net
worth
• One time conversion of local currency to Euros,in 3
hrs, millions of records changed
32
33. Why our insurance customers chose DataKinetics
Insurance
• Have 10000 – 15000 tables, 5000 live
• Rules processing, lookup processes, decision
tables
• Replace one system for each of 50 states (took 6
months to implement a change across all) with one
application that is rules based and takes a few
hours to update without a recompile and the
associated testing
34. Why some of our other customers chose DataKinetics
Trading house
• Use tables for program trading, examine data
and trigger executes a program
• Stock market reconciliations and settlement
• Change processes that ran sequentially to run in
parallel and stay within batch window
• Stock market ticker table that maintains current
price and is updated as trades occur resulted in
an order of magnitude performance
improvement from using write through cache
and rules tables to manage program logic
Retail
• Monitor sales by item by store in real time
• Very fast access to price lists and product descriptions
Healthcare
•Eliminate 27M I/Os in 24 hours
35. Why governments chose DataKinetics
Government
• Used by tax department
• Used to provide government services
• Used for reference tables
36. Today’s array options
• In-memory array is not searchable – requires programming
population, retrieve only by subscripts
• Can sort arrays in memory using the dataspace, but it’ll
move all the data out and move it back in again
• Tools on searching manipulating sorting arrays not readily
available
• In DB2 there is a concept of cursor
• Can’t search or sort data in cursor, can only iterate through
cursor, can do repetitive iterations
tableBASE® provides a unique capability for in-memory virtual sorts
37. Accelerator for DB2
DB2 and other RDBMS tableBASE™
Raison d’etre Designed to manage vast amounts of data; superior Designed to provide short, very fast path to data; enables elegant and
correlation , indexing and relational storage of data efficient processing enhancing performance and reducing costs
Relations amongst SQL used to assemble relations as needed SQL used to normalize data into a single table for high speed retrievals
data NOTE: tableBASE itself does not use SQL, This describes how the
data is first retrieved, in this example, from DB2 or other RDBMS
Ease of optimization Caching options Allows you to optimize search methods and table organization methods
– data already cached
Handling lots of Very efficient if the application needs to “harden” the Most efficient for stable and semi-stable data that is read only.
updates and data after every row update Databases are more efficient if every row that is read has to be updated.
contention amongst More efficient if application needs to read one or two
the updates rows of a very large table, and doesn’t use them again, tableBASE reads the entire table into memory – which makes it very fast
as it reads only the required rows for all subsequent fetches
Sorts Fetch data according to pre-defined index (reads). A Virtual sorts - Fetch data, load into tables, without indices, create first
complex fetch will require many I/Os index, create indices on the fly, read out data in whatever order is
required – don’t physically sort the data
Path to data DB2 may buffer data, but may not be kept in buffer in Shorter than DB2 (even if data is buffered), returns all data
contiguous format, but still need to re-organize.
Selects part of the data
Brings back a block, and then have to pull out what’s
needed and re-format
Table coupling Gets involved in binding (dynamic and static) and if Don’t have these concepts, therefore much faster as don’t have to
definition changes, need to re-bind search – the interface remembers the table location
Format of rows in a Each row must be the same format Rows do not have to have the same format, although at least one
“table” column must be used for the row identifier; row length is constant
within a table
Security Has security for access Ties in with RACF, ACF2 and TTS
Shortcuts Tables can be opened implicitly, using GET, FETCH and other retrieval
commands,
Meta data Does meta translation on a field by field basis Meta data translation not needed – application knows the format of the
row it is receiving.
37
38. Cost benefit
months
Time to implement
new paradigms
flexibility
maintenance
new paradigms
CPU
weeks flexibility
maintenance
elapsed time
CPU
days
Replace I/O Process control Rules or decision
tables tables
tableBASE capability
39. Return on Implementation
tableBASE Tables to replace I/O Process control tables to Decision tables to replace
capability decouple processes logic trees
Savings Replace temporary files with tables Significantly reduce time to Significantly reduce time to
Eliminate sorting implement new products, time to implement new products, time
implement changes to implement changes
Almost eliminate I/Os to reference
tables Overall program maintenance Overall program maintenance
reduced reduced
Simplifies very sophisticated
decision making
The numbers >80% less CPU time* Time to make a change or create a Competitive advantage in
6 to 23 times faster* new offering reduced from weeks to program trading – the program
hours. logic becomes the table
Batch job reduced from hours to
minutes
Examples Consolidated financial statements. Insurance tables, credit card offerings Settlement in investment
Product lists or price lists in companies
memory
Insurance tables, credit card
offerings
Time to implement 1-5 days to change and verify Can be several weeks to re- architect Can be several weeks to
performance benefits in a test the program. Thereafter changes months to re- architect the
environment take hours. program. Thereafter changes
take days.
How to find the Look at DB2 statistics to identify DataKinetics can perform an audit DataKinetics can perform an
savings tables that are read often and provide consulting services audit and provide consulting
services
Tools ?? ?? ??
Bill has a DOS program that allows you to define the rules, test the decision tables, and generate consolidated decision tables by eliminating all the
“don’t care” cases
39
40. How to know if your applications can be improved
• Batch jobs give I/Os, check the number of I/Os on a
particular file and number of accesses to a table. If these
files are reference data (largely read only), then a good
candidate. Analyse how many I/Os would be eliminated if
the reference information were in memory
• Typical ratio is that 4/5 I/Os are reference
• Customer data is not a good candidate
• If only 1 or 2 are updating the data, good candidate
• Has complex “if then else” logic
• Applications that are cloned – can be combined and table
driven
41. Designed for performance
• Does not do meta data translation on a field by field basis as does DB2
• Avoids OS to do access, eg, link command, or I/O
• Avoid getmains
• Avoid locking
• Efficient algorithms for search
• Choose the search that is the most efficient for the data
• Returns entire rows or portions thereof
• Uses implicit commands to reduce changes to calling application
• Application asks for row, and if table not open tableBASE opens the table –
avoids explicit changes to application to first open and then find row
• Uses shortcuts to reduce path to data
• Searches list of tables first time, and next time, uses a shortcut so doesn’t have
to search again
• Allows dynamic creation of indices; can populate and then create index,
can can create multiple indices after population of tables
41