Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
The WSO2 Business Activity Monitor - Optimizing Enterprise Data Management
1. With WSO2 Business Activity Monitor 2.4.1
Inosh Goonewardena
Associate Technical Lead
Gokul Balakrishnan
Software Engineer
Optimizing Enterprise Data
Management
June 2014
2. **
About the Presenters
● Inosh Goonewardena
Inosh Goonewardena is an associate technical lead
and is a part of the WSO2 Business Activity Monitor
team. He holds a bachelor’s degree in Engineering,
specializing in Computer Science & Engineering from
University of Moratuwa, Sri Lanka.
● Gokul Balakrishnan
Gokul is a software engineer in the Data and APIs
Technology Group focusing on WSO2 Business
Activity Monitor. He has a bachelor’s degree in
Engineering from University of Moratuwa, Sri Lanka,
and has provided consultation services on customer
engagements.
3. **
● Global enterprise, founded in
2005 by acknowledged leaders in
XML, web services technologies,
standards and open source
● Provides only open source
platform-as-a-service for private,
public and hybrid cloud
deployments
● All WSO2 products are 100% open
source and released under the
Apache License Version 2.0.
● Is an Active Member of OASIS,
Cloud Security Alliance, OSGi
Alliance, AMQP Working Group,
OpenID Foundation and W3C.
● Driven by Innovation
○ Launched first open source API
Management solution in 2012
○ Launched App Factory in Q2 2013
○ Launched Enterprise Store and
first open source Mobile solution
in Q4 2013
About WSO2
5. Agenda
● Introduction to Business Activity Monitoring
● WSO2 Business Activity Monitor
● WSO2 BAM Components
● WSO2 BAM Toolboxes
● WSO2 BAM High Availability Distributed Deployment
● Product Demo
6. “The aggregation, analysis, and presentation
of real-time information about activities
inside organizations and involving
customers and partners” - Gartner
Business Activity Monitoring
8. ● Data operations
● Building KPIs
● Operate on large
amounts of historic
data or new data
● Building BI
Analysis
9. ● Visualizing KPIs/BI
● Custom Dashboards
● Visualization tools
● Not just dashboards!
Presentation
10. ● Fully-open source solution for aggregating and analyzing
data and presenting information about business activities.
● Cloud-enabled, lightweight, developer-friendly and easy-to-
deploy
● High performance data capture framework
● Elastically-scalable data analysis powered by Apache Hadoop
● Pre-built Data Agents for WSO2 products
WSO2 Business Activity Monitor
13. ● Compatible with CEP/BAM
● Get data across to BAM
− Service monitoring feature – WSO2 AS, DSS, ESB, API
Manager
− Mediation monitoring feature – BAM Mediator for WSO2
ESB
− Custom data-agents
● Asynchronous & non-blocking
● Thrift for high performance message throughput
Data Agents
14. ● Data agents are meant to be installed/plugged
into the server to be monitored
● A Java SDK is provided
● Through Thrift, different languages can be
supported
Custom Agents
16. ● Receives data and stores it in Cassandra
○ Scalable, big data repository
● Asynchronous & non-blocking
○ Combination of Cassandra, Thrift and the non-blocking nature
results in extremely fast writes
● Shared with WSO2 CEP for real time analysis
● Supports Thrift & REST API
● Supports plugging in of different receiver types
Data Receiver
17. ● Data is sent over using strongly typed Data Streams
{
'name':'phone.retail.shop',
'version':'1.0.0',
'nickName': 'Phone_Retail_Shop',
'description': 'Phone Sales',
'metaData':[
{'name':'clientType','type':'STRING'}
],
'payloadData':[
{'name':'brand','type':'STRING'},
{'name':'quantity','type':'INT'},
{'name':'total','type':'INT'},
{'name':'user','type':'STRING'}
]
}
● Data Streams are versioned
○ Allows for easier analysis
Data Model
19. ● Powered by Apache Hadoop with querying/managing
through Apache Hive
● Data transfer to and from data sources is handled
through custom storage handlers
● Parallel, distributed processing through the MapReduce
programming model
The Analyzer Engine
20. ● Runs on local Hadoop node or delegates to Hadoop cluster
o Scalable analytics
o Cluster can range from a couple of nodes to 1000s
● Analysis is carried out based on analytics scripts
● Scripts are based on an easy-to-learn, SQL-like query language
INSERT OVERWRITE TABLE UserTable SELECT userName, COUNT(DISTINCT
orderID),SUM(quantity) FROM PhoneSalesTable WHERE version=
"1.0.0" GROUP BY userName;
The Analyzer Engine
21. ● Scripts can be scheduled
○ Ex: once a minute, every Wednesday at 4:15 p.m., every
30th
at 12 midnight
● Polyglot data architectures are supported
○ Write summarised information to any kind of datastore
○ Custom written Hive JDBC handler used by default for
output to relational DBs
The Analyzer Engine
23. ● Gadget Portal for the dashboard
● Gadget Generation Wizard – Tool for generating custom
gadgets for the dashboard
● Activity Dashboard for correlating activities
● Message Console
● Plug in your own report server/ dashboard server
The Presentation Layer
30. ● BAM Toolboxes are installable and hot deployable artefacts used for
deploying functionalities to a BAM server
○ Stream definitions
○ Analytics scripts
○ Dashboards
● Supports plugging in of one or more of the above features
● Toolboxes for monitoring and auditing most WSO2 products are
available OOTB
● Toolboxes for custom scenarios can be created easily
BAM Toolboxes
31. ● WSO2 BAM can be clustered and deployed in a distributed manner
to enable high-availability, fail-over scenarios
○ Partially distributed deployment
■ Cassandra (storage) and Hadoop (analyzer) nodes are clustered
○ Fully distributed deployment
■ All components of BAM are clustered (data receiver, storage,
analyzer and presentation)
● Hazelcast in-memory data grids are used for clustering
implementation
HA Distributed BAM
36. WSO2 BAM Demonstration
In this demo:
ESB mediation statistics monitoring with the
mediation data agent
BAM Mediation stats toolbox
BAM dashboard
Gadget gen tool with SQL query and visualisation
37. ● UES integration for the presentation layer
● Enhanced alerting and reporting capabilities
● Enhanced toolbox support for WSO2 product
stack
● Abstract data storage layer
Upcoming Features
