Perficient is a leading IT consulting firm that helps clients implement business-driven technology solutions. It was founded in 1997 and has over 2,000 employees located throughout North America and in global delivery centers. Perficient provides a wide range of services including custom application development, package implementation, architecture planning, and more to support solutions like enterprise portals, SOA, BI, and CRM. The presentation discusses the need for comprehensive application monitoring to prevent costly outages and errors. It covers using logging and complex event processing to monitor systems, transactions, and business events at a high level and in real-time. A demonstration of Perficient's Generic Audit Logger and Generic Exception Handler is also provided to illustrate logging and monitoring solutions
2. About Perficient
Perficient is a leading information technology consulting firm serving clients
throughout North America.
We help clients implement business-driven technology solutions that integrate
business processes, improve worker productivity, increase customer loyalty and create
a more agile enterprise to better respond to new business opportunities.
3. Perficient Profile
Founded in 1997
Public, NASDAQ: PRFT
2011 Revenue of $260 million
Major market locations throughout North America
— Atlanta, Austin, Charlotte, Chicago, Cincinnati, Cleveland,
Columbus, Dallas, Denver, Detroit, Fairfax, Houston,
Indianapolis, Los Angeles, Minneapolis, New Orleans,
Philadelphia, San Francisco, San Jose, Southern California,
St. Louis and Toronto
Global delivery centers in China, Europe and India
2,000+ colleagues
Dedicated solution practices
87% repeat business rate
Alliance partnerships with major technology vendors
Multiple vendor/industry technology and growth awards
4. Our Solutions Expertise & Services
Business-Driven Solutions Perficient Services
• Enterprise Portals End-to-End Solution Delivery
• SOA and Business Process Mgmt IT Strategic Consulting
• Business Intelligence IT Architecture Planning
• User-Centered Custom Applications Business Process & Workflow
• Interactive Design Consulting
• CRM Solutions Usability and UI Consulting
• Enterprise Performance Management Custom Application Development
• Customer Self-Service Offshore Development
• eCommerce & Product Information Package Selection, Implementation
Management and Integration
• Enterprise Content Management Architecture & Application Migrations
• Enterprise Resource Planning Education
• Management Consulting Interactive Design
• Industry-Specific Solutions
• Mobile Technology Perficient brings deep solutions expertise and offers a
complete set of flexible services to help clients
• Security Assessments implement business-driven IT solutions
5. Our Speakers
Eric Roch
• Principal SOA | Mobile | Cloud
• 30+ years of experience in various aspects of
Information Technology including:
• IT executive level management within industry and
consulting
• technical architecture
• application and systems development.
• He has also been an IT industry speaker and
author for many years.
Ben Hahn
• Sr. Technical Architect
• 20+ years of experience in various aspects of
Information Technology.
• Software Solutions Architect
• Enterprise Infrastructure Architect
• Product Management
• Software community contributor
6. Agenda
• Glitches and outages drive the need for
more comprehensive logging and
monitoring
• Big Data and Logging
• Events and Logging
• Logging and exception management as a
service
• Explicit Transaction Monitoring
• Instrumentation for Logging
• Logging and Exception Management
Framework and Demo
6
7. The Need for Application Monitoring
• Systems are distributed, increasingly more complex and we are progressively
more dependent upon them
• Application glitches are costly in dollar terms, customer loyalty and brand image:
– Department of Commerce study found software errors cost U.S. economy
$59.5 billion annually
– Information Week says IT downtime costs $26.5 billion in lost revenue
– The 2003 North America blackout was triggered when a local outage was
not detected by monitoring software
– Glitches in the reporting of prices on the NYSE may have caused the “flash
crash”
– Knight Capital Group Inc. lost $440 million from software trading error –
about $10M a minute for 45 minutes
– Programming error at Visa Debit Processing Services caused a billing error of
$23,148,855,308,184,500
7
8. Humans Monitoring for Glitches
Tourist follows GPS into Bay UK woman follows GPS into river
8
10. Machine Monitoring
• Log analysis is well established for IT operations, security and
compliance
• Enterprise management software standards to detect platform and
network problems
• LogLogic appliance can ingest up to 250,000 events per second with
high speed filtering and routing capabilities
• Splunk provides general-purpose search language for analysis and
reporting for time-series data using the MapReduce framework
10
12. Log Analysis vs. Business Analytics
• Ingest – Versus ETL
• Big Data – Bidirectional integration with Hadoop
• Query language – MapReduce function on unstructured data
• Drill anywhere – Investigate on all the data versus a predefined schema
or cube
• Information discovery – Discover relationships based on patterns in the
data
• Ad-hoc versus dimensional – Log analysis is not based a predefined
structure based a point-in-time set of requirements
Source: splunk Implementation 12
13. Business Events and Event Processing
• Event-driven architecture (EDA) is a
software architecture pattern
promoting the production,
detection, consumption of, and
reaction to events.
• Complex event processing (CEP)
consists in processing many events
happening across all the layers of an
organization, identifying the most
meaningful events within the event
cloud, analyzing their impact, and
taking subsequent action in real
time.
13
14. CEP High-Level Use Cases
• Situation awareness is about
"knowing" the state of the
product, person, document, or
entity of interest at any point in
time.
• Sense and respond is about
detecting some significant fact
about the product, person,
document or entity of interest,
and responding accordingly
• Track and trace is about tracking
the product, person, document or
entity of interest over time and
tracing pertinent facts
Source: TIBCO Software
14
15. CEP Architecture
Fine-grain
• Millions of raw events can CEP
Business Events
Engine
be represented in one
complex event Concept
State
• Component status (fine
grain) – outage (logical Rule
/predictive) Bases
• Events are process with Logical Events – Notifications, Consequences Actions
BPMS ESB
declarative rules and Event
Channel(s)
implicit state management
• Events drive agile business
processes
Flexible Workflows
Integration and
SOA
Business Components
System(s) of Record Business Applications
15
17. Application and Business Event Logging
• Monitoring logical transaction state vs. monitoring transaction through-put
• Exception management vs. error logging
• Explicit application event logging versus machine logging
• Application instrumentation to a framework
• Exception management and log correlation
– Transaction and conversation IDs
– User defined fields
– XML processing - XSD, XSLT and XPath
– Agents and appenders
– Time and dependencies
17 17
18. Logging and Exception Management Design Goals
• Highly configurable and user friendly GUI
• High-speed
• Non-blocking asynchronous calls
• Open architecture / standards support
• Technology agnostic
• Service-oriented
18
19. What is GAL?
Generic Audit Logger
• A configurable message logger to
record, view or reply message
• Fully configurable via a GUI
• Map logs to custom fields
• Filter and query logs
• Supports message query from GEH
• Works standalone and supports our
exception handler
19
23. Transaction Monitoring via Logging
• Leveraging GAL and GEH to define a transaction
Using recognition rules, each log entry can define a
• Transaction demarcation – begin or end
• Transaction context – using a conversation identifier
• Transactional checkpoints – define completed stages in a transaction.
• Transactional errors
Once a transaction is defined we get
• Monitoring
• Instrumentation
• Error tracking via an error handler like GEH
• State and through-put
23
25. Events via Transactions
Now that we have transactions we can also define events
• Each log entry can now also define an event
• Events can be aggregated to make more logical events
• Events can also be generated for “non-events” e.g. if number of
transactions do not reach a certain threshold (sales quotas)
25
Monitoring systems should catch “should not occur” conditions not just machine and platform performance or outage issues
In many cases humans do not recognize the glitch. Or much damage is done before the glitch is noticed,
We need layers of monitoring – a transaction monitor to monitor applications with escalations to humans based on rules
Web Service Distributed Management Common Base Event Simple Network Management Protocol (SNMP)
What’s missing in this picture is business events. I will come back to that.
Splunk – Google for IT data but what about the use case Google for Business Events
Events and big data.A single complex event can be an abstraction of millions of raw events A tweet is an event – but we don’t likely care about individual tweets we can about consumer sentiment – a complex event in time – event processing can drastically reduce the amount of big data Example the real time trending of voter sentiment during the presidential debate – a line graph in time
10s of millions of events logged as 3M complex events per day