What measurements are useful, and which metrics’ baseline makes sense to show the effectiveness of your quality management system? This presentation takes an ISO 13485 and 21 CFR, Part 820 approach to deciding what measurements are important, how trending makes an impact on risk-based decision-making, and why management review is just one of the ways to discuss and document your Quality Data activities.

2. Measuring and Metrics
Tracking and Trending Data
John Gagliardi
MidWest Process Innovation, LLC
June 2017
OMTEC

3. 3
Courtesy of MidWest Process Innovation, LLC
Let’s Add Value
Using Key Performance Indicators
There are valuable and not-so-valuable measurements
to take and metrics to measure against.
There are measurements that have “nothing to
do with nothing” and measurements that might
save your company.

4. 4
Courtesy of MidWest Process Innovation, LLC
Let’s Keep It Simple
Your business and quality goals need to be clear and united.
A essential measurement with a realistic metric can:
 realize progress (improvement)
 identify problems
 be understood by all process owners
 initialize change when change is needed

5. 5
Courtesy of MidWest Process Innovation, LLC
Key Performance Metrics
In today’s performance-focused environment,
medical device companies require performance
metrics to monitor progress against quality
goals and objectives, e.g. to evaluate the
effectiveness and efficiency of Quality
Management System (QMS) and (as far as ISO
is concerned) business process excellence.

6. 6
Courtesy of MidWest Process Innovation, LLC
Measures vs. Metrics
There is overlap between measures and metrics. Both can be
qualitative or quantitative, but what distinguishes them is important.
Measures are concrete, usually measure one thing, and are
quantitative in nature (e.g. I have five apples).
Metrics describe a quality and require a measurement baseline
(I have five more apples than I did yesterday). Measures are useful
for demonstrating workloads and activity, and metrics are useful for
evaluating compliance, processes effectiveness and measuring
success against established Quality Goals and Objectives.

7. 7
Courtesy of MidWest Process Innovation, LLC
How Much Data do I Need
Minimal
Medical device companies often struggle with data collection and usage.
Some companies collect too much data and then do nothing with it.
More “poundage” is not (necessarily) better.
TOO MUCH DATA CAN BE CONFUSING
TOO LITTLE DATA CAN BE DECEIVING
Other companies wait for eye-openers like complaints, medical device reports,
non-conforming products and negative process results, scrap reports and returns.
They anticipate these bits and pieces of bad news to bubble to the top of their
quality overflow scenarios. It’s not a pretty sight, thinking that a Management
Review will miraculously “put out the fires using risk to make decisions about
post-market incidences.”

8. 8
Courtesy of MidWest Process Innovation, LLC
Process Metrics
Sometimes the PFMEA tool can show where measurements should be taken
Use process metrics to:
 evaluate business processes
 establish process improvement goals
 measure progress against those goals
Consider applicable constraints and limitations when developing these
reporting and analysis methods, but realize that process improvement is
meaningless without an associated process metric. (For instance, you
change a process without realizing benefits. What is the cause: ineffective
process change or lack of process adoption?) Without process metrics in
place, you have no visibility into the effectiveness of the changes you
make to business processes. Use process metrics to evaluate the
progress of corrective actions, discover areas of your company
that need more attention or resources, and recognize process
components that are effective or show impressive improvements
over time.

9. 9
Courtesy of MidWest Process Innovation, LLC
Use Objective-Oriented Performance Metrics
Linkages are Important – interrelate the data
Pre-determine how your reporting will support your QMS and your
company’s goals and objectives. Simply collecting information for later
analysis is wasteful…DO SOMETHING WITH IT.
Purchasing
Process
Controls
Incoming
N/C
Product
Risk
Management
Acceptance
ActivitiesCAPA
Evaluation
and
Monitoring

10. 10
Courtesy of MidWest Process Innovation, LLC
Collection Constraints
Company culture, technology level and (human) resources can
influence information collection, how it is collected, resulting
reports, and associated analysis. Quality and Business
Objectives must drive reporting methodologies.
Your reporting can be constrained by some limiting factors:
 Difficulty and expense of collecting information
 Accuracy and timeliness of information
 Whether activities are performed at one site or multiple sites
 Simplicity and ease of interpreting reports
 Organizational culture, structure and priorities

11. 11
Courtesy of MidWest Process Innovation, LLC
Compliance Notes

12. 12
Courtesy of MidWest Process Innovation, LLC
The Quality Review Board can supply
Inputs to Management Review

13. 13
Courtesy of MidWest Process Innovation, LLC
Track and Trend Data
ISO 13485:2016 and 21 CFR, Part 820 include a
commitment to continually monitor and maintain
the effectiveness of a QMS through
improvement efforts.
Improvements that are aligned
with Quality Planning:
Management Review
Quality Review Board
Post Market Activities

14. 14
Courtesy of MidWest Process Innovation, LLC
…and now that I have all of this
data, what do I do with it!?
Critical Point
Your company needs to realize that the data and
parameters which are reviewed (studied) must be:
 Well-thought-out
 Real-time actionable
Fruitless discussions and (maybe no) action taken
are common for companies that are just fakin’ it.
Compliance with ISO may not be enough

15. 15
Courtesy of MidWest Process Innovation, LLC
The New ISO 13485:2016
…wants you to tie these metrics to the Quality
Management System (QMS) to take a barometric
measurement (if you will)effectiveness check
and
…how well “the business” is running
as part of the QMS

16. 16
Courtesy of MidWest Process Innovation, LLC
Management Review is the Minimum
(or it could be just right)
 Feedback
 Complaint Handling
 Reporting to Regulatory Authorities
 Audit Results
 Monitoring and Measuring of Processes
 Monitoring and Measuring of Product
 Corrective Action/Preventive
Action Changes

17. 17
Courtesy of MidWest Process Innovation, LLC
Everything is Not Quantitative
 Quality policy (Clause 5.3)
 Quality objectives (Clause 5.4.1)
 Results of audits – including internal, supplier, certification
audits and FDA inspections (Clause 5.6.2a)
 Customer feedback – including complaints and post-market surveillance (Clause 5.6.2b)
 Process performance (Clause 5.6.2c)
 Product conformity (Clause 5.6.2c)
 Supplier quality performance (Clause 8.4)
 Status of corrective and preventive actions (Clause 5.6.2d)
 Follow-up of action items from the previous Management Review(s) (Clause 5.6.2e)
 Changes that could affect the quality system (Clause 5.6.2f)
 Recommendations for improvement (Clause 5.6.2g)
 New and revised regulatory requirements (Clause 5.6.2h)
 Risk management process (ISO 14971)
 Overall quality system effectiveness (Clause 5.1)

18. 18
Courtesy of MidWest Process Innovation, LLC
Metrics to help make other decisions
(connecting the dots)
 Are all people trained and competent – training is up-to-date
 Number of batches made since last management review
 Number of released finished product
 Number of batches shipped
 Lot acceptance rates
 On-time-delivery to customer/on-time delivery by supplier(s)
 Average quote turn-around time
 Number of returned products because of complaints
 Corrective and Preventive Action – turn-around time
 Number of high-risk Audit Findings
 Number of incoming non-conformances upon receipt
 Number of incoming non-conformances caught during manufacturing
 Number of new suppliers and number of dropped suppliers

19. 19
Courtesy of MidWest Process Innovation, LLC
ISO 13485:2016 Top Level Changes to Section Eight –
Measurement, Analysis and Improvement
A new requirement has been added that information gathered in the feedback
process shall serve as input in the risk management process, as well as the product
realization process to assure that monitoring for the product is being completed.
Analysis of data — Adds the requirement to include determination of appropriate
methods, including statistical techniques and the extent of their use.
 Adds detail to inputs
 Expects the use of trends
 Close linkage with Corrective and Preventive Action(s)
Risk / Planning / Risk / Planning / Risk / Planning / Risk / Planning / Risk / Planning

20. 20
Courtesy of MidWest Process Innovation, LLC
ISO 13485:2016 Top Level Changes to Section Eight –
Measurement, Analysis and Improvement
Section 8 – Measurement, Analysis and Improvement
The final section of the ISO 13485 standard has not significantly changed as many of these
processes have been consistently performed over many years, and the changes are to better
align with other regulatory requirements. There is also a much stronger emphasis that post-
production information needs to serve as an input in the risk management process for
identification of new hazards and confirming current hazard assessment. There is clarification
that a determination needs to be made for any nonconformance, whether detected before or
after delivery, as to what further actions may need to be taken, e.g. investigation, evaluation,
concession or corrective action.
A summary of the changes are as follows:
There has been a clarification that the feedback process it not necessarily just
customer complaints as has been more commonly understood over the last few years.
The feedback process needs to be clearly defined to gather data from production as well
as post-production activities to ensure that the full picture of the product safety and
performance is evaluated.

21. 21
Courtesy of MidWest Process Innovation, LLC
A Risk Tangent
In previous editions of ISO, a clause on preventive
action was separated from the whole.
Now risk is considered and included throughout the standard.

22. 22
Courtesy of MidWest Process Innovation, LLC
Risk-based Thinking
By taking a risk-based approach, an organization becomes
proactive rather than purely reactive, preventing or reducing
undesired effects and promoting continual improvement.
Processes that present uncertainties in the QMS
are candidates for measurement.

23. 23
Courtesy of MidWest Process Innovation, LLC
Risk-based Benefits
Risk-based thinking, therefore:
 Establishes a proactive culture of improvement
 Assures consistency of quality of goods or services
 Improves customer confidence and satisfaction
 Builds a strong knowledge base for planning purposes
 Proactively improves operational efficiency and governance
 Builds stakeholder confidence in the use of risk techniques
 Enables organizations to apply management system controls to
analyze risk and minimize losses
 Improves management system performance and resilience
 Enables organizations to respond to change effectively & protect their business as they grow
 Allows an organization to make informed decisions concerning planning and compliance
Identify the objective evidence you would investigate to show that your company is using risk…

24. 24
Courtesy of MidWest Process Innovation, LLC
Risks and Planning
Risks and opportunities must be integrated into the planning process for
business planning, or for setting organizational goals and objectives.
Measuring against goals and objectives can help mitigate risks
It identifies key processes and conducts risk analysis on them
because they affect the organization’s overall objectives.
 Risk Policy
 Risk Plan
 Risk Procedure
 Risk Report

25. 25
Courtesy of MidWest Process Innovation, LLC
Examples of Data

26. 26
Courtesy of MidWest Process Innovation, LLC
Customer Feedback
• Product failure and/or defect rates
• Customer complaints
• Customer complaint cycle times including “time to closure”
and “average days open”
• Number of adverse events
• Number of recalls
• Customer surveys

27. 27
Courtesy of MidWest Process Innovation, LLC
Progression Thinking
...can be the start of “a story”
Measure number of complaints received
per quarter (Q3FY2017)
491 Simple measurement
What does it tell us? Zip!
The President of your company set a
metric of less than 50 per quarter
Woops!
< 50 Where did < 50 come from?
• Pressure from the Board
• Historical
• “Knee jerk” reaction*
Compare to complaints received over the
last three “quarters”
250 / 301 / 367 Looks like an upward trend?
Okay...now what?
...and we still have > 50!
Compare: last years’ four-QTR average 26.7 per QTR Now What?
Number of Adverse Events (Q3FY2017) vs.
past three quarters
6 vs. 1 / 2 / 2 Looks like an upward trend?
Higher risk!
Number of Returned Products (for further
disposition) – in thousands
12/17+/15+/ 25+ Looks like an upward trend?
% of devices found to be defective
prior to shipping
12 / 9 / 11/ 9 Looks like a downward trend?
*From the tendency of the knee to jerk involuntarily when hit sharply, properly called the patellar reflex.

28. 28
Courtesy of MidWest Process Innovation, LLC
Customer-Related Measurements
Quarterly Measures: Customer-related Picture
Q4 FY2016 Q1 FY2017 Q2 FY2017 Q3 FY2017
Customer Complaints 250 301 367 491
Number of
Adverse Events
1 2 2 6
Number of Recalls 0 0 0 0
Customer Survey
Results – Rating
1= worse rating
10 = best in class
7 6 8 7
Number of Returned
Products (for further
disposition)
12,000 17,500 15,600 25,230
% of devices found to
be defective prior to
shipping
12 9 11 9

29. 29
Courtesy of MidWest Process Innovation, LLC
Is the Quality System
Effectively Operated
• Number of non-conforming events including tracking and
trending of types, sources and impacts
• Non-conforming record or report (NCR) cycle times including
time to closure, average days open, and quantity by department, etc.
• Number of CAPAs opened including tracking and trending of
types, sources and impacts
• CAPA closure cycle time and average days open
• CAPA effectiveness review
• Number of planned deviations opened
• Internal audit status and result summaries
• Internal audit remediation schedules, results and targeted due date status

30. 30
Courtesy of MidWest Process Innovation, LLC
Progression Thinking
...can be the start of “a story”
Measure number of audit findings received
per quarter (Q3FY2017)
19 Simple measurement
What does it tell us? Zip!
The President of your company set a
metric of less than 5 per quarter
Whoops!
< 5 Where did < 5 come from?
• Pressure from ISO
• Historical
• CAPA process is overwhelmed
Compare to audit findings received over the
last three “quarters”
23 / 30 / 17 Looks like an average of 23.3?
Okay…now what?
...and we still have > 5!
Compare: last year’s four-QTR average 7 per QTR Now What?
Number of High Risk Internal audit findings
(Q3FY2017) vs. past three quarters
11 vs. 7 / 16 / 5 Looks like an average of 9.3?
Higher risk!…and we still have > 5!
Number of Returned Products (for
further disposition) – in thousands
12/17+/15+/25+ Looks like an upward trend?
% of devices found to be defective
prior to shipping
12 / 9 / 11/ 9 Looks like a downward trend?

31. 31
Courtesy of MidWest Process Innovation, LLC
Quality System – Process-related
Quarterly Measures: Process-related Picture
Q4 FY2016 Q1 FY2017 Q2 FY2017 Q3 FY2017
Number of Audit
Findings
23 30 17 19
Number of Process -
related CAPAs opened
21 19 12 15
Number of Process-
related corrections
opened
2 11 5 4
Number of High Risk
Internal Audit Findings
7 16 5 11
Employee Turnover % 9 11 5
33*
(Layoff)
% of devices found to
be defective prior to
shipping
12 9 11 11

32. 32
Courtesy of MidWest Process Innovation, LLC
Supplier Performance
• Supplier performance scorecard review, including business risk
analysis as related to key suppliers and/or critical raw materials
and/or services
• Supplier audit status and result summaries including
the status of remediation efforts/projects

33. 33
Courtesy of MidWest Process Innovation, LLC
Supplier-related
Quarterly Measures: Supplier-related Picture
Q4 FY2016 Q1 FY2017 Q2 FY2017 Q3 FY2017

34. 34
Courtesy of MidWest Process Innovation, LLC
Manufacturing Performance
• Percent yields, scrap rates, rework rates, etc.
• Percent of lots manufactured without
a NCR or planned deviation
• Labor utilization (to verify there is sufficient
strategic deployment of resources)
• Production schedule adherence and
backorder rates

35. 35
Courtesy of MidWest Process Innovation, LLC
Manufacturing-related
Quarterly Measures: Manufacturing-related Picture
Q4 FY2016 Q1 FY2017 Q2 FY2017 Q3 FY2017

36. 36
Courtesy of MidWest Process Innovation, LLC
Training Process Impact
Knowledge is Crucial
Qualitative and quantitative metrics to correlate training
process investment with an appropriate return.
For example, is there data that indicates employee training
translates to positive trends in terms of:
 better workmanship (fewer product defects)
 decreases in non-conformance rates and sources
 reduced customer complaints
 improving internal audit results
 greater efficiency

37. 37
Courtesy of MidWest Process Innovation, LLC
Resources Evaluated
Present data that answers the following questions:
• Do all CAPAs, NCRs, planned deviations, internal audits, supplier
audits, and continuous improvement projects have owners
assigned and is progress tangible?
• Are all cycle time metrics associated competent people?
• Are open positions being actively recruited and filled?

38. 38
Courtesy of MidWest Process Innovation, LLC
Opportunities for Improvement
• Review the status of continuous improvement projects
already in-progress…on schedule/on cost
• Use the data reviewed in the meeting to determine
whether additional projects need to be initiated
Corrective And Preventive Action can “kick-start” the process

39. 39
Courtesy of MidWest Process Innovation, LLC
Use Objective-Oriented Performance Metrics
Linkages are Important
Pre-determine how your reporting will support your QMS and your
company’s goals and objectives. Simply collecting information for later
analysis is wasteful…DO SOMETHING WITH IT.
Purchasing
Process
Controls
Incoming
N/C
Product
Risk
Management
Acceptance
ActivitiesCAPA
Evaluation
and
Monitoring

40. 40
Courtesy of MidWest Process Innovation, LLC
What Measurements Could We Look At?
Purchasing Controls and Linkages
Process Name Measurement Measurement Measurement
CAPA (SCAR)
Acceptance
Activities
Risk (Product
and Process)
N/C Product
(incoming)
Evaluation and
Monitoring
Score Card
Quality
Agreements

41. 41
Courtesy of MidWest Process Innovation, LLC
What Metrics Could We Develop?
Purchasing Controls and Linkages
Process Name Metric Process Owner
CAPA (SCAR)
Acceptance Activities
Risk (Product and Process)
N/C Product (incoming)
Evaluation and Monitoring
Score Card
Quality Agreements

42. 42
Courtesy of MidWest Process Innovation, LLC
Remember...
 Having strategic objectives is imperative to realizing success or failure
 By taking a risk-based approach, an organization becomes proactive
rather than purely reactive, preventing or reducing undesired effects
and promoting continual improvement.
 Measuring is only half the story
 Measurements with real-time metrics can be action provoking
 Integrated measurements create a story line (a pulse-rate)
 Management Review is one vehicle to report
on your QMS performance
 Quality Review Boards with cross functional teams is a
good reporting vehicle in between Management Reviews
 Don’t keep changing your measurements…you’ll never see trends
 Keep it simple

43. 43
Courtesy of MidWest Process Innovation, LLC
Thank You
John Gagliardi
MidWest Process Innovation, LLC
7736 Woodside Court
Maineville, Ohio 45039
JGAGL777@One.Net
johngglrd@gmail.com
www.midwestprocessinnovation.com
Cellular Telephone: 513.315.9820
Office Telephone and Fax: 513.573.0085
Alternate Telephone: 513.573.0519