Software functional testing can unveil a wide range of potential malfunctions in applications. However, there is a significant fraction of errors that will be hardly detected through a traditional testing process. Problems such as memory corruptions, memory leaks, performance bottlenecks, low-level system call failures and I/O errors might not surface any symptoms in a tester’s machine while causing disasters in production. On the other hand, many handy tools have been emerging in all popular platforms allowing a tester or an analyst to monitor the behavior of an application with respect to these dark areas in order to identify potential fatal problems that would go unnoticed otherwise. Unfortunately, these tools are not yet in widespread use due to few reasons. First, the usage of tools requires a certain amount of expertise on system internals. Furthermore, these monitoring tools generate a vast amount of data even with elegant filtering and thereby demand a significant amount of time for an analysis even from experts. As the end result, using monitoring tools to improve software quality becomes a costly operation. Another facet of this problem is the lack of infrastructure to automate recurring analysis patterns.
This paper describes the current state of an ongoing research in developing a framework that automates a significant part of the process of monitoring various quality aspects of a software application with the utilization of tools and deriving conclusions based on results. According to our knowledge this is the first framework to do this. It formulates infrastructure for analysts to extract relevant data from monitoring tool logs, process those data, make inferences and present analysis results to a wide range of stakeholders in a project.
A software monitoring framework for quality verification
1. JCSSE 2012
A SOFTWARE MONITORING FRAMEWORK
FOR QUALITY VERIFICATION
MANAGED BLACK BOX TESTING
PRESENTED BY:
DILEEPA JAYATHILAKE
99X TECHNOLOGY
2. QUALITY IS PARAMOUNT
Operational Overhead Poor Quality
• Constant re-planning Kills
• Derailing deadlines
• Higher costs
Through-life Costs
• Recalls / Updates
• Warranty Claims
• Litigation
Business Value Decline
• Loss of market share
• Dropped brand equity
3. FUNCTIONAL TESTING
Focuses on spotting any observable deviations of the product
with respect to the desired features and behavior
Both manual and automated
Covers most feature-related malfunctions
Limited to only the behaviors observable directly
4. WHITE BOX TESTING
Supplements functional testing
Directly examines code
Can test cyclometric
complexity, testability, unit test
coverage, potential memory leaks, class
and function level
complexity, deprecated API usage, code
duplication, undocumented code, styling
errors
Many tools exist for this
Does not cover operational errors
5. BLACK BOX TESTING
Certain errors are not captured either by Memory
functional testing or by white box testing corruptions, memory leaks
Disk access errors
They need monitoring application in Failures in certain low level
operation operating system calls
Pitfalls due to insufficient
Black box testing is required for a testing user access rights
process to be completed Unjustifiable performance
bottlenecks
Wide spectrum of tools is available in
each platform
Some tools provide a UI while others
generate a log
6. BLACK BOX TESTING TOOLS
Tool Remarks
Apache - Used for server load testing in client-server applications
JMeter - Supports files, servlets, Perl scripts, Java objects, database queries, ftp servers
- Works with HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, POP3 and IMAP
protocols
- Portable across platforms
- Can replay test results
- Highly extendable via plugins
- Provides simple and intuitive UI
- Does not support client-side scripts
Microsoft - Captures system level problems that occur when a program is run
Application - Can monitor Virtual memory usage, First chance access violation exceptions, Input-
Verifier output transfers, Synchronization objects (e.g. locks), Handle usage, Thread pool
usage, Thread local storage, Dll handling, Interactive service creation, Dangerous
API calls, Driver installations and User privilege issues
- Writes results to an XML log
- Optionally a debugger can be attached
7. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
LeakDiag & - A pair of tools that can be used in conjunction to detect memory leaks in an
LDGrapher application
- Can monitor Virtual memory allocator, Windows heap allocator, COM allocator, C
runtime allocator and TLS slot allocator
- LeakDiag generates an XML log, which can be graphically viewed through
LDGrapher
Process - Windows logging utility, which collects and records most of the system activity
Monitor regarding file system, registry and process/thread activity
- Comes with a rich GUI that displays information in real time
- Offers advanced filtering options to trace specific activity in the interest
- Provides a boot-time logger
- Should be run only for a limited time
8. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
XPerf - Performance-profiling tool for Windows applications
- Can be used for tracking performance bottlenecks in applications as well as for
comparing time taken for various operations in an application
- Writes monitored data into an XML log which can be later explored via XPerfView
- Comes with a rich collection of preset monitoring profiles
Application - This is a Windows application lifecycle management toolset
Compatibility - Toolset comprises Standard User Analyzer, Internet Explorer Test Tool and Setup
Toolkit Analysis Tool
- Can be utilized to
• Analyze portfolio of applications, web sites and computers
• Centrally manage compatibility evaluators and settings, Rationalize and
organize applications, web sites and computers
• Prioritize application compatibility with filtered reporting
• Add and manage issues and solutions for enterprise- computing
environment
• Deploy automated mitigations to known compatibility issues
• Send and receive compatibility information from the Microsoft
Compatibility Exchange
9. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
God - Process-monitoring tool for Linux, BSD and Darwin systems
- Provides watchdog functionality on processes with respect to their CPU and
memory usage
- Can be configured to perform certain actions on a process depending on its CPU
and memory consumption
- Has both a polling mode and an event responder
- Generates a line log
- Capable of sending alerts as email, chat or Twitter messages
Instruments - Standard tool for profiling processes that run under Mac OS X or iOS
- Uses monitoring templates called instruments to collect data on various aspects
- Monitoring areas include CPU consumption, memory usage, memory leaks, I/O
operations, power usage and network traffic
- Data viewers are built in
- Capable of recording a sequence of actions to replay later
10. Tool usage
requires
platform
expertise
Analysis
Generated logs
knowledge not
are bulky
shared
Black box
testing
pitfalls
Difficult to
communicate Difficult to
results to correlate results
different from multiple
stakeholders in tools
a project
Lot of manual
work
Framework is needed to integrate tools and automate analysis
12. CONCLUSIONS
LDEL significantly reduces code for log data extraction
LDEL improves readability of the data extraction code
Introduction of SQLite into data manager enabled
handling high data volumes and persistence
Need a better database for efficiently handling
heterogeneous log data
MML is not appropriate for UI generation
Editor's Notes
I think that every presentation regarding finance or investments has to be grounded in some type of metaphor. The right metaphor allows the audience to feel immediately comfortable with the presentation even though they may not have much prior knowledge on the subject matter. To me, sailing has always seemed to provide just the right terms to describe both the planning and patience required of a successful investor. Whether it be “charting the course” of an investment plan or having the patience required to navigate choppy markets, sailing and investing can make for a compelling presentation. Disclaimer: You understand that Microsoft does not endorse or control the content provided in the following presentation. Microsoft provides this content to you for informational purposes only; it is not intended to be relied upon as business or financial advice. Microsoft does not guarantee or otherwise warrant the accuracy or validity of this information and encourages you to consult with a business or financial professional as appropriate.
Tools are two fold 1. white box testing tools 2. Result integration tools
Tools are two fold 1. white box testing tools 2. Result integration tools
Tools are two fold 1. white box testing tools 2. Result integration tools
Tools are two fold 1. white box testing tools 2. Result integration tools
Tools are two fold 1. white box testing tools 2. Result integration tools
Tools are two fold 1. white box testing tools 2. Result integration tools
This slide lays out the overall objective of the presentation; which is how to achieve or start working towards a “successful retirement”. It starts with a “written financial plan”, and moves clock-wise – all centered around the main goal. On this continuum, buttons can be added or removed, depending upon the financial professional giving the presentation.A successful retirement plan has many moving parts and each of them must serve a unique function in carrying you to retirement. Regardless of your situation, the journey to retirement is likely dotted with some storms and choppy seas along the way. Here, I have laid out what I believe to be the very basic principles necessary for any successful retirement plan.
Tools are two fold 1. white box testing tools 2. Result integration tools