The mission statement sets the direction and priority for developing and implementing the quality plan. It clearly states the nature of the organization’s commitment to quality and should then be tied to the organizational operations through programs, projects, actions and rewards/recognition.

1. Measuring & Understanding Quality
Improvement in Healthcare
Mahboob ali khan, MHA / CPHQ,
Consultant Quality - Operations
Continental Hospitals
Hyderabad, India

2. Continuing the Journey
1) Research Questions
• Experience in Healthcare Operations
2) Theory / Models
•Search and Study
•Develop Questions and Hypotheses
3) Develop / Test
•Develop 2 into something that can
help answer 1
4) Examine Results
• How does 3 answer 1
Focus
Oral Exam
Defense

3. Initial Research Questions
Developed from 8 years of frustration in healthcare
management knowing that I wasn’t equipped to
provide appropriate support to clinicians
What I Could Provide What I Needed to Provide
Leadership
Understanding of Healthcare
Financial Direction
Strategic Direction
Management Capabilities
A More Balanced Approach to
Managing
Assessment
Data Management
Study Design

4. Initial Research Questions
 How do I know something works?
 Continued requests for equipment, supplies and
instruments
 Do patients get better?
 The End of Medicine
 Is there one way to do a procedure that is
better than another?
 Significant variation in preference cards
How do I measure quality?

5. Defining Quality
 IOM – The degree to which health services for
individuals and populations increase the
likelihood of desired health outcomes and are
consistent with current professional knowledge.
 Donabedian - The systematic measurement and
evaluation of the predetermined outcomes of a
process, and the subsequent use of information
to improve the process based on expectations
of the customer.

6. Theories / Models – Oral Exam
 Ernest Amory Codman
 end results idea – 1920s
 Florence Nightingale (late 1800s) and Walter Shewhart
(1920s)
 Statistical Process Control
 Edwards Deming, Joseph Juran and Kauru Ishikawa
 Continuous Quality Improvement (CQI)
 Avedis Donabedian
 structure – process – outcome
 Implicit vs. explicit criteria
 Everett Rogers
 Dissemination of Innovation
 Don Berwick, Paul Batalden, Brent James and Steve
Shortell
 Recent literature

7. Structure – Process - Outcome
Quality of healthcare can be assessed on the basis of
structure, process (how care is delivered), and outcome
(mortality, functional status, quality of life, and patient
satisfaction)
good measures of the first two are those that have a clear
relationship to the third
structure must proceed process which must proceed outcome
 Structure Definition
 Something arranged in a definite pattern of organization
 Organization of parts as dominated by the general
character of the whole

8. Implementing CQI is Largely Structure
Medical Scott CQI
Outcomes
Organization Division of labor HR / Training
Specialty Mix Coordination Customer Focus
Resources Training/Experience Planning
Work Load Power Process
Access Resources Innovation
Buildings Supplier Partner
Information Information
Leadership Leadership
Policies/Procedures
Tasks

9. Structural Dimensions of CQI
Implementation
Strategic Cultural Technical Organiz. Result
No Yes Yes Yes No significant results
on anything important
Yes No Yes Yes Small, temporary
effects
Yes Yes No Yes Frustration & False
starts
Yes Yes Yes No Inability to capture the
learning & diffusion
Yes Yes Yes Yes Lasting process
change
Adapted from Shortell et al. 1996

10. Translating Theory To Research –
Putting Shape to my Frustrations
1. The healthcare system is broken
The IOM reports
2. Appropriate structure – “the forgotten, but important
component of the quality triad” – is wrongly assumed
as present
 JCAHO
3. CQI, if implemented properly, can be the appropriate
structure
Managerial Philosophy
4. Healthcare providers are are finding it difficult to
implement CQI
Problem Solving Methodology

11. Research Question
 Why haven’t healthcare organizations
been able to use CQI to differentiate
themselves in terms of quality?
Answer: Accountability & Assessment

12. Accountability
Individual motivation has not been successful
Midnight at the Waldorf-Astoria
Rhetoric, not Reality (The Halothane Study)
Large scale environmental change is needed
Environment, Organization, Micro-System, Pt
Current Motivators that may force change
Patient Safety – Medical Errors
“report cards” - PA, NY and CA
Increase in Costs and Premiums
Increase in the use of Alternative Medicine
Variation in processes
Increase in litigation

13. Assessment Issues & Research Questions
1. Low CQI knowledge level of senior leadership
Do step by step instructions exist that assists researchers in studying CQI,
and healthcare leaders in implementing CQI?
2. Implementing only a subset of the CQI domains
Does a comprehensive survey and scale of CQI implementation exist?
3. Poor measurement strategies
Are psychometrics examined appropriately in measuring CQI implementation?
4. Rhetoric does not equal the reality
Does a measure exist that can be used to develop a quick snapshot of CQI
implementation efforts in a hospital and is there a corresponding descriptive
scale?

14. Step 1: Search the Literature
Does a comprehensive, valid and easy to administer
measurement tool exist that provides organization leaders
with a descriptive scale and instructions for next steps?
 Methods -
 Used Ovid databases with a focus on business,
psychology, sociology and healthcare
 Keywords: measurement quality, continuous quality
improvement, total quality management,
implementation
 Scrolled through quickly at least 10,000 references
 most focused on implementing only a few domains of CQI
 Articles not deleted included: Measuring Quality;
Domains and Implementation of Quality

15. Quality Overload
Using Medline through PubMed
 from 1995 to the present / English only
 ‘quality improvement’ = 8,848
 ‘continuous quality improvement’ = 1,100
 ‘quality’ in title = 17,466
 ‘quality improvement measure’ in title = 3
 from 2000 to present / English only
 ‘quality improvement’ in title = 350
 50 usable, 30 ‘easily findable’, 15 good, 1
measurement

16. Worldwide Measurement for QA/QI Structure
AWARDS - too time intensive, gold standard domains
 The Malcolm Baldrige Award
Leadership, HR/Training, Process, Business Results, Customer
Focus, Information Systems, Planning, Partnership
 EFQM
 US State-Based Awards
ACCREDITATION & CERTIFICATION - questions on
validity and too time intensive
JCAHO
IS0 9002 - 2000
Inspection, Contract, Public Responsibility, Innovation, Product
Control, Servicing
SURVEY INSTRUMENTS

17. Survey Instruments
Eleven worldwide surveys examining CQI as a
managerial philosophy were analyzed (Tables - pges
1 & 2)
8 from the US, 1 from Canada, the Netherlands, and
Australia
5 were specific to healthcare, including the ‘gold standard’
from Shortell et al.
Most examined psychometrics while very few provided a
scale
The shortest (22 questions) was also the least
comprehensive
The Baldrige Domains dominated

18. Survey Instruments
Weaknesses of the current surveys included:
Relatively few domains other than the Baldrige were
even mentioned
The most comprehensive surveys are much too long
(depth vs. breadth)
Strong potential for respondent bias if survey is only
given to one level employee in an organization
Very few surveys provided a scale
Published in the International Journal for Quality in
Health Care 2001: Volume 13, Number 3: pp 197-207

19. Step 2: Develop and test a comprehensive and
concise measure of CQI implementation
 Initial Survey
 14 domains (Baldrige, EFQM, ISO) and 70 items, with
each domain containing at least 4 items
 All but a very few items were from the 11 surveys
analyzed in Step 1
 Items were chosen by the researchers using a
subjective analysis and whether or not the question
could be answered using a 5 point Likert scale

20. Content Validity
 The benefits of a content validity study for this study
True experts in the field of CQI
Past measures have gone through psychometric testing
Excellent method of data reduction
Methods
Statistical method described in Grant & Davis (1997) and
Lynn (1986)
1. Panel of Experts
All either attend an invite only CQI symposium sponsored by
Dartmouth, have recently taught CQI at a Masters level or are
positional leaders of QI efforts in a healthcare organization
 Best to have between 7 and 10 - this study had 7 from the
US and 1 from England

21. Content Validity
 Methods (cont.)
2. Scoring Grid (See Sample Grid - pge 3)
 Each expert was emailed the scoring grid with
definitions and instructions.
Is the item clear and understandable?
 4 point scale
Does the item represent CQI?
 4 point scale
Match the item with a domain.
 1 through 14 representing each domain & 15 representing
unable to classify

22. Content Validity
 Methods (cont.)
3. Indices
Inter-rater agreement (IR) = # of raters who scored an
item as high / total # of raters
 high defined as a 1 or 2 on both 4 point scales
 acceptable IR > / = .70
Content Validity Index (CVI) = # of items where all
experts rated high / # of items
 acceptable CVI > / = .80
Domain congruence = % of time where experts chose
the same domain as the investigators

23. Content Validity Results
 After 4 analysis iterations where poorly rated items
were deleted, the questionnaire included:
 22 items
 8 domains
 Clarity IR of .91 (range of .85 - 1)
 Representativeness IR of .93 (range of .87 - 1)
 Clarity CVI of .73 using Lynn’s (1986) method
 Representativeness CVI of .91 using Lynn’s (1986)
method
 Overall, the experts chose the same domain as the
investigators in the original measure 76% of the time

24. Content Validity Results
 Investigators added 6 items to ensure that every
domain except for Supplier Partnership contained 3
items
 Investigators changed some wording to increase
clarity
 Final survey was sent back to experts for comments
Other Results
The scoring grid took a lot longer to complete than originally
thought
The leadership domain had the highest rate of agreement
QI must be differentiated from QA
Baldrige criteria dominate

25. Step 3: Develop a corresponding scale of
CQI implementation
 A 5 level corresponding scale (pge 4) was developed by
the investigators from:
 Roger’s Diffusion of Innovations
 agenda setting, matching, redefining / restructuring, confirmation,
clarifying, routinizing
 Deming’s Continuous Quality Improvement
 match domains with appropriate level
 Samsa & Matchar
 CQI as a problem solving methodology vs. a managerial
philosophy
 Characteristics:
 CQI is a developmental process
 Time is important
 Scale provides focus for future quality initiatives

26. Steps 2 & 3 - Content Validity and Scale
Published and Voted Best Student-Led Paper in
the 2002 Business and Health Administration
Proceedings,pges 198-204
Will be further published in Either Quality in
Health Care or Hospital Topics

27. Step 4: Is the survey and scale easy to use
and are the results practical?
Pilot Study
 Worked with the Missouri Hospital Association for contacts.
 83 Missouri hospitals eligible (above 40 beds), 40 participated
 5 responses from each hospital: CEO/COO, Director of
Quality, a non-salaried MD, and 2 managers
 Survey and results disseminated via email
Hypotheses based on Paper 1 Weaknesses
1. There will be measurable differences between and
within hospitals.
2. The survey will have high known-groups validity.
3. The items and domains will differentiate between levels
as hypothesized by the conceptual scale.

28. Pilot Study Methods
 Known Groups Validity
 Examined the relationship between the survey and:
 state quality team winners >/= Level 2
 state quality organization winners >/= Level 3
 national quality award finalists >/= Level 3
 subjective quality assessment at 10 of the 40 hospitals
 question 1 (pge 5) asking the participants to categorize
their quality structure
 Reliability
 Cronbach’s Alpha for each domain, each title, and the
overall measure

29. Pilot Study Methods
 Between Hospital Variation
 One-way ANOVA & Bonferroni
 by size, region and ownership model
 Within Hospital Variation
 Repeated Measures ANOVA & Bonferroni
 by title
 Item and Domain Analysis
 ANOVA & Bonferroni
 determine which items and which domains discriminate
well between different levels of the scale

30. Pilot Study Hospital Total Level
N = 40 (min of 90, max of 130)
16
14
12
10
8
6
4
2
0
Level 1
(</= 100)
Level 2
(101-110)
Level 3
(111-120)
Level 4
(121-130)
Level 5
(131-140)
# of hospitals

31. Pilot Study Results
 Sample Characteristics (pge 6)
 40 hospitals are significantly larger and more likely to be
for profit and part of a system
 Of the 200 returned surveys, there was less than 5%
missing values and ‘I don’t know / NA’
 Known Groups Validity
 2 of the 12 did not score as hypothesized
 hospitals were not significantly different than others
 4 of 9 (44%) similar for the subjective assessment
 40% agreement for question 1 assessment

32. Pilot Study Results
 Reliability
 Cronbach’s Alpha ranged from .54 (HR/Training) to
.84 (Innovation) for the domains
 Information .69; Process, Planning .77; Customer .78;
Leadership .83
 Cronbach’s Alpha ranged from .88 (Director
/Manager and QI Director) to .92 (MDs) for different
groups
 Cronbach’s Alpha was .94 for the overall measure

33. Pilot Study Results
 Between Hospital Variation
 Region was only attribute that was significant
 Within Hospital Variation
 Senior Executives significantly lower than QI Directors
 QI Directors significantly higher than Managers /
Directors
 MDs significantly higher than Managers/ Directors
 supports surveying more than one level employee

34. Pilot Study Results
 Item and Domain Analysis (pge 7)
 Of the 28 items, 4 did not show good differentiation
between any of the levels
 these should be either reworded or changed
 Leadership showed significant differentiation between
all levels studied
 Planning showed significant differentiation between 2 of
the 5 levels

35. Pilot Study Results
Quality Improvement Scale
Level Components after Domain Analysis
1. Quality Assurance
2. QI Low High Focus – Leadership ( visibility) ; Customer Focus
M edium Focus – I nnovation
3. QI Medium High Focus – Process; HR / Training; Planning
M edium Focus – Leadership ( support)
4. QI High M edium Focus – I nformation; Supplier Partnership
Low Focus – Leadership ( consistency) ; Planning
5.QI - Absorbed

36. Findings
 The survey is easy to administer
 The survey provides a reliable and valid snapshot of
CQI implementation in a healthcare organization
 No known group exists
 The scale is a practical method of providing hospital
leaders with a roadmap for CQI implementation
 Leadership is the most important component of
implementing CQI
Submitted to Health Services Research

37. A Likely Future Scenario
1. Patient Safety provides accountability to analyze
quality and outcomes
2. Healthcare leaders see CQI as a methodology to
improve patient outcomes
3. Hospitals use the survey and scale to help
assess & implement CQI appropriately, which in
turn eliminates structure issues discussed
4. Because of this, hospitals can effectively assess
their processes and improve their outcomes

38. Future Research Questions
Can a clearer snapshot of CQI implementation emerge using
line worker responses, and senior leadership interviews?
 Administered the survey to a 40 random line workers at 10
hospitals
 Initial results include:
 50% return rate
 high number of ‘I don’t know / NA’ responses
 all hospitals overall employee score < 100 (Quality Assurance)
 Senior hospital leadership meetings to discuss quality
structure
 Initial results include:
 low knowledge of ‘quality’ among the senior leaders
 structures developed with little statistical or facilitation resources

39. Future Questions
Does a higher level of CQI implementation lead to better
financial, quality and safety outcomes? If not, why?
 Develop and Find Financial, Operational and HR
Effectiveness and Efficiency Measures
Counte & Glandon, 1995
 Build one, clean database with CQI implementation
scores and measures
 Analyze to assess relationships

40. Future Questions
Is CQI, as its described in the literature, an effective
method for improving quality outcomes?
 Change the four items that did not differentiate well
1. How many multi-disciplinary teams currently work to improve the
processes of care in your organization? (Process)
 more statistics, less teams
• Human Factors Research
• Toyota
• Six Sigma
2. Do people in your organization know who their customers are?
(Customer Focus)
 not clear
3. Are employees in this organization encouraged to try new and
better ways of doing things? (Innovation)
4. Is creativity actively encouraged in this organization? (Innovation)
 healthcare has typically not been innovative and so these questions
may need to be more specific

41. THANK YOU