In FDA regulated industries, audits are high-stakes, fact-finding exercises required to verify compliance to regulations and an organization’s internal procedures. Although exploratory testing has emerged as a powerful test approach within regulated industries, an audit is the impact point where exploratory testing and regulatory worlds collide. Griffin Jones describes a heuristic model—Congruence, Honesty, Competence, Appropriate Process Model, Willingness, Control, and Evidence—his team used to survive an audit. You can use this model to prepare for an audit or to baseline your current practices for an improvement program.
Griffin highlights the common misconceptions and traps to avoid with exploratory testing in your regulated industry. Avoid mutual misunderstandings that can trigger episodes of incongruous behavior and an unsuccessful audit. Learn how to maintain your composure during a stressful audit and leave with valuable heuristics to help you organize and present your exploratory testing results with confidence.
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Heuristics for Exploratory Testing: Surviving an FDA Audit
1. Griffin Jones – Congruent Compliance LLC 1March 2012
Test Strategy and Design #602
Surviving an FDA Audit:
The Heuristics for Exploratory Testing
Griffin Jones, Consultant, Congruent Compliance
2. The Heuristics for Exploratory Testing
2Griffin Jones – Congruent Compliance LLCMarch 2012
3. Preliminaries
Who is in the room?
My goal:
Stimulate your interest to study the subject more
Leave with a heuristic to help you organize and present
with confidence your ET results to regulatory auditors
Have a conversation and try to meet your needs
Quick Preview
The context
The heuristic and how to apply it
Some of the traps about ET in a regulated industry
Griffin Jones – Congruent Compliance LLC 3
4. Assumptions and Terms
More reference information here than I will present
Follow the for the key points
Much of this can be adapted to other contexts
i.e., not “FDA regulated, Exploratory Testing”
“Schools of Testing” by Bret Pettichord
Analytic , Standard, Quality, Context-Driven, Agile
Exploratory Testing
Simultaneous learning, test design and test execution
Agile Testing
Story completion, test automation: Test Driven Dev., etc.
4Griffin Jones – Congruent Compliance LLC
5. Terms
Congruence
Being balanced between inner feelings & outer actions
Smells
Symptom that possibly indicates a deeper problem
5 Whys
Questions-asking method to investigate root causes
“Mary had a little lamb” heuristic
Emphasize each of the individual words in a statement
Checking: confirming existing beliefs; versus:
Testing - finding new information (Michael Bolton)
Griffin Jones – Congruent Compliance LLC 5
6. The Problem
Let’s assume that you are FDA regulated and trying to
do compliant context-driven or Agile, Exploratory
Testing
You likely have these concerns about passing an audit:
Evidence is not sufficient
Documentation is not sufficient
Process control is not sufficient
Can’t clearly explain what you do and why
Auditors value different things than you, and speak a
different language
Griffin Jones – Congruent Compliance LLC 6
7. Fast Takeaway
The regulator is not your business partner
The regulator has police powers
Pick your battles – Sometimes, “Let the Wookie win”
“Render unto Caesar, that which is Caesar’s …”
Auditors are likely of the “Quality” (gatekeepers) or
“Routine” (traceability matrix) testing school model
You are a different testing school. Deal with it.
Auditors think “testing” is “demonstration and
checking”
Don’t try and convert them. Deal with it.
7Griffin Jones – Congruent Compliance LLC
8. Spoiler
The regulations are not the problem
How you are coping with the regulations is the problem
Give the Auditors what they want:
Clear traceable requirements and description of risks
Description and demonstration of control
Clear objective evidence
The ability to understand their concerns, speak their
language, and explain how you are compliant
Abundant, quality evidence mitigates your other
problems
8Griffin Jones – Congruent Compliance LLC
9. Not going to talk about…
The Fear, Uncertainly, and Doubt swirling in the field
Vendor/Experts: “You should be scared, but I have…”
Silver Bullets and Big Magic
“… so trust me and just buy my wares. By the way, ..”
Persistent Myths
“… IMO the regulators “frown on” ET (… I don’t sell it).”
The “Typical” Regulatory Affairs Presentation
9Griffin Jones – Congruent Compliance LLC
10. Regulatory Overview
Regulations
For the public good - because people died
Regulators
FDA regulates >25% of the Gross Domestic Product
Regulatory Auditors
Police Powers
Industry Auditors
Assessors and valued advisors to management
Audits
10
Details
Griffin Jones – Congruent Compliance LLC
11. Audit Survival Heuristics
CHCMWCE
“Chocolate Mousse”
Congruent
Honest
Competent
Model (Appropriate)
Willing
Control
Evidence
16
Model
Competent
Honest
Evidence
Control
Willing
Congruent
Griffin Jones – Congruent Compliance LLC
12. Let’s take a journey …
17
Practice
Congruent
Theory
Less Stressful
Audits
Griffin Jones – Congruent Compliance LLC
13. The Congruence Triad
Congruence is when you are balanced between inner
feelings and outer actions
The Congruence Triad
Self, Other, Context
Being congruent is a process
A way of communicating with yourself and others
Incongruence is when part of the triad is missing
Placating, Blaming, Super-rational, or Irrelevant?
What is missing and fill it in:
Self, Others, Context
18
Other
Context
Self
Details
Griffin Jones – Congruent Compliance LLC
14. Congruence is like a Sailboat
Because:
It is a vessel or container, like a basket
It requires preparation and maintenance
You don’t “drive” it, and requires skills of crew members
Subject to weather
Is vulnerable to sinking
20Griffin Jones – Congruent Compliance LLC
Tools
15. The Theory Mountains …
Dishonest
Incompetent
Inadequate
22
Honest
Competent
Appropriate Model
Self-Incriminating
Experts and Heroes
Over-Constrained
Griffin Jones – Congruent Compliance LLC
16. Honest
Integrity, Truthful, Trust, Sincerity in:
You and your organization
Words, actions, and documents
Smells
Dishonest
Self-incrimination
Don’t create even the appearance of a problem
Tests
How do you and the organization react to criticism?
Are you a learning organization? (5 Why)
23Griffin Jones – Congruent Compliance LLC
17. Competent
Are you and your organization:
Capable, credible, understands context, speaks the
language; trained in the industry, technology, and
regulatory obligations
Smells
Incompetent
Experts and heroes
Tests
Do you believe you are capable of doing good work?
(5 Why)
24Griffin Jones – Congruent Compliance LLC
18. Appropriate Model
Is the process model:
Complete, reasonable, practical, logical, explainable
Smells
Inadequate model
Over-constrained model
Test:
What problem is this model solving? How will it Fail?
What is required in this model? Missing?
Do you believe this model is sufficient? (5 Why)
25Griffin Jones – Congruent Compliance LLC
19. The Practice Mountains …
Unwilling
Out-of-Control
No Evidence
26
Excessive or Wasteful
Micro-Management
Obsessive-Compulsive
Willing
Under Control
Evidence
Griffin Jones – Congruent Compliance LLC
20. Willing
Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
Smells
Unwilling
Excessive or Wasteful
Test
Do people care? (5 Why)
Is there sufficient resources for the work and
expectations? (5 Why)
27Griffin Jones – Congruent Compliance LLC
21. Under Control
Explain what you are doing and why. Are you living it?
Coherently explain your:
configuration control and authorization
traceability and accountable
organization, preparation, planning, independent review,
prevention, correction, checking and testing
Smells
Out-of control
Micro-managed
Tests
Is the type and level of controls appropriate? (5 Why)
28Griffin Jones – Congruent Compliance LLC
22. Evidence
Auditable evidence:
Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a third party can review or
reconstruct (with minimal outside help); and quickly
reach the same results and conclusions.
Smells
No-evidence
Obsessive-compulsive evidence
Tests
Explain why the specific evidence meets the criteria.
(5 Why)
29Griffin Jones – Congruent Compliance LLC
23. How do you apply this?
Application is as simple as:
30
Remembering
to ask the questions.
Follow the energy
of the answers.
Fix the base, first.
Griffin Jones – Congruent Compliance LLC
24. During an Audit
Choosing a regulatory posture
Manageable issues (within reason)
Evidence
Controls
Willingness (resources and priority)
Unmanageable issues
Broken process model
Lack of competence
Broken trust
Incongruence
31Griffin Jones – Congruent Compliance LLC
25. More Fast Takeaways
The FDA is open to agile processes and realizes that
the current approach to software validation is not
working
At the same time, companies are more concerned
about:
the business risk that the FDA would not accept the
agile process,
than the product or project risk that is associated with
waterfall type development
Find the middle option for your context
32Griffin Jones – Congruent Compliance LLC
26. Natural Evidence
Periodically , take the observer point-of-view and ask:
Is what I see and hear, about the theory and practice of
what we do:
acceptable from both a product qualification and
regulatory compliance point of view?
If yes, what is the most natural, efficient, and strongest
evidence we could collect?
Why not a video/audio recordings w/ paper summary?
Is it being collected? If no, why not? (5 Why)
organizational problem?
33Griffin Jones – Congruent Compliance LLC
28. Smells that lead to …
Stop Shaking the Snow Globe
Hyper-change alongside brittle/heavy formal processes
The “Best Practice” Cargo Cult
We don’t really understand the details of what we do,
why we do it, or how what we do works. But have faith.
Testing Death Spiral
Regulator does not care about testing and management
might only care about regulatory compliance. Spiral.
The Titanic
The gigantic engineered process is perfect – people are
the source of problems, not solutions
35Griffin Jones – Congruent Compliance LLC
29. Organizational Disasters
Pathetic Compliance
Following a regulatory compliant procedure in a way
that does not solve the testing problem for which it was
designed.
Utopian Shelf-ware Procedures
No one reads them. They are not reality.
Close Enough
I don’t have to do it exactly. I know better. No one will
notice or care.
Read My Mind
Because that is the only place where the evidence is.
36Griffin Jones – Congruent Compliance LLC
30. Is the Auditor on Tilt?
37
Maybe it is something we said or did, or are doing?
History
That you are unaware of, and it might be complicated
Notches on the gun
May be making a name for themselves
Making an example of you
May be constructing an example to deter others
Griffin Jones – Congruent Compliance LLC
31. Classic Agile Traps
Mixing informal and formal processes
Start informal - clearly switch to formal when ready
Emphasizing change; light documents = poke the bear
Stokes anxiety: control, process model, and competence
Mistaking team conversation and understanding
For objective documented evidence
Speaking “Crazy Agile Moon Language”
Give the auditor what they want, in their language
Shows empathy and industry competence
38Griffin Jones – Congruent Compliance LLC
Pass
Fail
32. Classic ET Traps
Implementation details identified as requirements
Tighten and simplify your requirements
Documentation lacks detail to support traceability
Require less mind reading
Control is vague or assumed
Summarize and document what control is for you
39Griffin Jones – Congruent Compliance LLC
33. The BIG Trap
Weak Evidence
“Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a third party
can review or reconstruct (with minimal outside help);
and quickly reach the same results and conclusions.”
Check it via “Mary had a little lamb”
Collect it naturally
Weak evidence is likely a symptom of other deeper
issues
Abundant, quality evidence mitigates your other
problems
40Griffin Jones – Congruent Compliance LLC
34. Audits can be Useful
Candor can result in free consulting and insight
Should you take the risk?
Provides motivation – management cares
Provides actionable data
The jiggle that is needed by the organization
A counter-measure to low expectations & poor practices
41
If you can’t be a good example,
you are going to be a stern warning.
Griffin Jones – Congruent Compliance LLC
35. Recap of the Spoiler
The regulations are not the problem.
How you are coping with the regulations is the problem.
Give the Auditors what they want:
Clear traceable requirements and description of risks
Description and demonstration of control
Clear objective evidence
The ability to understand their concerns, speak their
language, and explain how you are compliant
Abundant and quality evidence mitigates your other
problems.
42Griffin Jones – Congruent Compliance LLC
36. The Big Take Away
Understand your regulatory context
Work on your congruence
Work each level of the model, ask the questions
Document how you are under control
Improve your evidence, collect it naturally
Avoid the smells, disasters, and traps
Summarize your regulatory story, practice explaining it
Apply what you learn during the audit
43
1
2
3
Griffin Jones – Congruent Compliance LLC
38. Further Study - A
FDA presentations and resources:
Webinar with FDA's John Murray on Software Validation
in the Field of Medical Devices
Presentation: Preparing for an FDA Medical Device
Sponsor Inspection
Quality System Inspection Technique – Inspection
Guide
General Principles of Software Validation; Final
Guidance for Industry and FDA Staff
45Griffin Jones – Congruent Compliance LLC
39. Further Study - B
Regulatory Compliance
“The Art of Compliance: Turning Compliance into
Sustainable Business Advantage” by Robert Rhoades of
Quintiles
FDA inspections:
“How to Host an FDA Inspection” by SGS – Life Science
Services
“Preparation for FDA Inspection” by
NEMA/ADVAMED/PHILIPS
“FDA Sponsor Inspections: How to Prepare and Survive”
by Medtronic, Inc
46Griffin Jones – Congruent Compliance LLC
40. Further Study - C
Audits
“The ASQ Auditing Handbook” by J. P. Russell
Congruence
“Beyond Blaming” by Jean McLendon and Gerald M.
Weinberg
“The Satir Model: Family Therapy and Beyond” by Virginia M.
Satir
“More Secrets of Consulting: The Consultant's Tool Kit” by
Gerald M. Weinberg
Testers and Auditors
“Testers are like auditors” by James Christie
Evidence
“21 CFR Part 11 Electronic Records …” by the FDA
47Griffin Jones – Congruent Compliance LLC
41. Further Study - D
Agile and the FDA
Business Risk (from the FDA) versus Product Risk
http://blogs.construx.com/forums/t/432.aspx
“What is Exploratory Testing? And How it Differs from
Scripted Testing” by James Bach
“Coping With Complexity: Lessons From a Medical Device
Project” by Yaron Kottler
“Introduction into IEC 62304 Software life cycle for medical
devices” by Christoph Gerber
http://www.spiq.com/abs/JF200809IEC62304%20SPIQ%20
Rev004.pdf
“Who says ET is good for Medical Devices? The FDA!” by
James Bach
http://www.satisfice.com/blog/archives/602
48Griffin Jones – Congruent Compliance LLC
42. Further Study - E
Agile and the FDA
http://rdn-consulting.com/blog/2007/07/25/update-
agile-development-in-a-fda-regulated-setting/
http://www.agilejournal.com/articles/columns/column-
articles/3463-four-reasons-medical-device-companies-
need-agile-development
http://rdn-consulting.com/blog/wp-
content/uploads/2007/07/060703ResMed.pdf
http://scalingsoftwareagility.wordpress.com/2010/11/23/
an-iterative-and-incremental-process-model-for-agile-
development-in-regulated-environments/
http://scalingsoftwareagility.wordpress.com/category/hi
gh-assurance-and-regulated-environments/
49Griffin Jones – Congruent Compliance LLC