Control model testing

Matthew Sullivan

MANAGING RISK FOR Scott Barber

S o f t wa r e Te s t

SOFTWARE PRODUCTS Professionals
Conference
Fa l l 2 01 1

Copyright © 2011 PerfTestPlus, Inc. All rights
reserved.

“STATE OF THE S/W TESTING PRACTICE”

• Find bugs (identify risks) OR
“Role” of • Check for compliance (V&V)
QA/Testing

• Appears undervalued, BUT
“Value” of • Doesn’t provide nearly the value it
QA/Testing could

• Business goals & value propositions
QA/Testing is • Business risks & risk controls
“out of sync” with • Executive information needs

reserved.

“THE UNDER-INFORMED DIRECTING THE
UNDER-TRAINED TO DO THE UNIMPORTANT”

Executives Testers Artifacts
(the Uninformed): (the Untrained) (the Unimportant)

•Don’t know how to ask •Don’t know what the •Bugs no one wants to fix
for what they need, SO executives need, SO •Metrics no one
•They ask for what they •They do what they are understands
know asked to •Documents no one
reads

reserved.

IMPROVING THE SITUATION (PART 1)

Focus on:
•Delivering business value
•Reducing business risk

At every business layer, identify & balance:
•Responsibility
•Accountability

Get your superiors to read Ch 16:Rightsizing the Cost of Testing:
Tips for Executives of How to Reduce the Cost of Software Testing;
CRC Press 2011

reserved.

IMPROVING THE SITUATION (PART 2)

reserved.

FEELING UNDER SIEGE?

Businesses reduce
allocation of
resources to
testing because of
a perception of
diminished value.

reserved.

WHAT DIMINISHES VALUE FOR TESTING?

1. Lack of insight into
future
2. Redundancy
3. Specification
blocks
4. Lack of
independence
5. Scope
constraint
reserved.

LACK OF INSIGHT INTO THE FUTURE

Why didn’t this
come up in
testing!

reserved.

REDUNDANCY

Sign here, and then sign
the next box attesting to
the authenticity of the
previous signature.

reserved.

SPECIFICATION BLOCK

Honestly I’d love to start testing today, but
first I need detailed requirements. VERY
detailed requirements

reserved.

LACK OF INDEPENDENCE

Its not fun being
the captain’s “no-
man”.

reserved.

SCOPE CONSTRAINT

Someone else was
supposed to be watching
for icebergs.

reserved.

REQUIREMENT-DRIVEN APPROACH

reserved.

THE MEANING OF LIFE (FOR TESTERS)

The purpose of
testing is to reduce
uncertainty about
the future impact
of technology.

reserved.

ALTERNATIVE APPROACH

reserved.

RISK AS A COMMON LANGUAGE

Security
Whether explicitly
or implicitly, all
Compliance Functional forms of testing
Risk revolve around the
reduction and
management of
Usability Performance
risk.

reserved.

THE SECRET TO MANAGING RISK

To effectively
manage risk, you
must effectively
manage
knowledge.

reserved.

WHAT IS CONTROL MODEL TESTING?

Control Model Testing
is a business-aligned
approach to software
testing that derives
“test cases” from
knowledge models
of the system based on
a risk-based
taxonomy .

reserved.

WHAT IS OUR TAXONOMY BASED UPON?

COSO
 Enterprise Risk
Management Integrated
Framework

The Open Group
 Technical Standard on
Risk Taxonomy

PerfTest Plus
 Taxonomy Extensions for

reserved.

WHAT ARE THE BASIC ENTITIES?

reserved.

THE OPEN GROUP’S RISK ASSESSMENT
FRAMEWORK

reserved.

RISK LAYERS

Business
• Financial
• Legal
• Brand or Reputation
Product
• Security
• Performance
• Usability
• Other Qualities
Project
• Budget
• Schedule
• Communication

reserved.

UNADDRESSED RISK

reserved.

HOW CAN TESTS ADDRESS THREATS AND
LEVEL OF RISK?

Controls prevent or
mitigate risk which
may impact business
objectives.

helps identify and
assess these controls.

reserved.

T YPES OF CONTROLS

Systems
• Firewalls
• Encryption
• Load Balancing
Preferences
• Settings
• Security and Access Model
Policies
• Code Standards
• Monitor and Response
• HR
reserved.

CONTROLS CONTEXT

Development
• Development and Test Tools
• Code standards
• Software components
Implementation
• Checklists
• Installation scripts
Maintenance
• Alerts and Triggers
• SOPs
• Configuration Management
reserved.

“SAMSARIC” TEST LIFECYCLE

Analyze

Report Assess

Effort
Evaluate

Knowledge
reserved.

ANALYSIS

Examine
• System
• Users
• Environment
Identify
• Objectives
• Processes
• Threats
• Controls
Output
• Initial Control Model

reserved.

INITIAL CONTROL MODEL

reserved.

ASSESSMENT

Activities
•Identify authorities
•Solicit opinions
•Evaluate exposure
•Determine impact
Outcomes
•Risk assessment
•Assessed Control Model
•Test plan
reserved.

ASSESSED CONTROL MODEL

reserved.

EVALUATION

Activities
•Execute planned and
derivative tests
•Identify discrepancies
•Determine capability
Outcomes
•Tested Control Model
•Test results
•Issues /
recommendations
reserved.

EXECUTED CONTROL MODEL

reserved.

REPORTING

Activities
•Communicate
•Recommend
•Respond
Outcomes
•Implementation plan
•Knowledgebase update
•Confirmation of or
revisions to test plan
reserved.

THE FOUR ROLES IN CONTROL MODEL
TESTING

Leader

Manager

Coordinator

Tester
reserved.

LEADER

Responsibilities:
• Representation
• Roadmaps
Interests
• Information
• Certainty
Talents
• Communication
• Vision
Typical Business Titles
• Director of Testing or Quality Assurance
• Chief Audit Officer (or Assistant to..)
• Principle Consultant

reserved.

MANAGER

Responsibilities:
•Organizing
•Developing
Interests
•Capability
•Consistency
Talents
•Understanding
•Motivating
•Test Manager

reserved.

COORDINATOR

Responsibilities
• Planning
• Oversight
Interests
• Successful outcome
• Thoroughness
Talents
• Teamwork
• Attention
• Test or QA Lead or Senior
• Analyst or Engineer Level 2 or 3
• Manager 1

reserved.

TESTER

Responsibilities
• Execution
• Analysis
Interests
• Discovery
• Experimentation
Talents
• Curiosity
• Skepticism
• Test or QA Analyst or Engineer
• Analyst or Engineer Level 1 or 2

reserved.

RISK LAYERS AND ROLES

Business

Product

Test
Leader Project
Test
Manager
Test Coordinator Tester

reserved.

SUMMARY

 Testing should be an indispensible advisor for leadership
 Testing should not be a convenience or scapegoat for
development
 All types of testing revolve around risk management
 The key to managing risk is managing knowledge
 Testing needs to be a learning discipline in the context of risk
taxonomy
 The test process should be a continuous cycle reducing ef fort
through increased knowledge
 Testing roles should correlate to management or risk, not
resources

reserved.

QUESTIONS?

matthewgsullivan@hotmail.com
sbarber@perftestplus.com

reserved.

RECOURCES

The Open Group (http://www3.opengroup.org/):
Risk Taxonomy Technical Standard -
https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetail
s.jsp?publicationid=12156
The Committee of Sponsoring Organizations of the Treadway
Commission, or COSO (http://www.coso.org/)
Enterprise Risk Management-Integrated Framework -
http://www.coso.org/ERM-IntegratedFramework.htm
PerfTestPlus, Inc. (http://www.perftestplus.com/)
Control-Model Testing – (http://www.perftestplus.com/control-model-
testing)
Rightsizing the Cost of Testing: Tips for Executives of How to Reduce
the Cost of Software Testing; CRC Press 2011

reserved.

ABOUT US

Matthew Sullivan Scott Barber
Quality Control Engineer CTO, PerfTestPlus, Inc
CCH TeamMate
Wolter s Kluwer  Widely regarded exper tise in
per formance.
 Test and Suppor t Engineer for
PricewaterhouseCooper s for 10  Contributor to:
year s  Performance Testing Guidance for Web
Applications– Microsoft Press
 Extensive experience in audit  Beautiful Testing- O’Reilly Press
and risk management industr y
 How to Reduce the Cost of Testing-
 Specialist in testing Microsof t Taylor and Francis
.NET, MS SQL Ser ver, and Lotus  Executive Director of the
Notes applications
Association for Sof tware Testing
 MS in Sof tware Engineering
from Regis University  Co-Founder of the Workshop of
Per formance and Reliability

reserved.

Control model testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Control model testing

Similaire à Control model testing (20)

Dernier

Dernier (20)

Control model testing