An instrument was developed and validated to measure information technology (IT) sophistication among Canadian hospitals. The instrument measures three dimensions of IT sophistication - technological, functional, and integration. It was validated through interviews with health IT experts, surveys of hospital IT directors, and analysis of reliability and validity. Survey results showed that Canadian hospitals generally had low levels of IT sophistication, especially in integration. The validated instrument can help profile and compare IT adoption among hospitals to identify areas for improvement.

1. An Instrument Validation Study
among Canadian Hospitals
Nawanan Theera-Ampornpunt, M.D.
M.S. & Ph.D. Student in Health Informatics
UMN Health Informatics Journal Club (Oct. 9, 2008)
Based on Paré G, Sicotte C. Information technology sophistication in health care: an instrument
validation study among Canadian hospitals. Int J Med Inform. 2001 Oct;63(3):205‐223.

2. 3 Themes of Focus in Article
Concept: IT Sophistication/Adoption
What, Why, How
Survey Instrument Evaluation
IT Sophistication Profile of Sampled
Hospitals

3. Background
Health care has become increasingly complex
and are in various stages of reform
IT has been shown in MIS to have positive
impacts on organizational performance
Managers and clinicians are realizing the
value of IT in high quality care delivery while
holding down costs

4. Background
To identify effects of IT on health care
delivery, IT needs to be characterized for
operationalization purposes
No existing characterization and validated
instrument of IT in terms of its level of
sophistication in health care organizations
Such instrument would allow profiling and
enable comparison

5. Study Objectives
Primary:
To develop and validate a measurement
instrument of IT sophistication in hospitals
Secondary
To present and compare IT sophistication
profile of Canadian hospitals

6. Existing Models
Nolan’s Stages of EDP Growth (1973, 1979)
Evolution of systems growth in stages
Introduces the concept of “IS Maturity”
The ultimate stage of computing growth in organizations
where information resources are fully developed and
computer‐based systems are fully integrated
Focuses on organizational information systems
Empirical validity has been contested
EDP - Electronic Data Processing IS - Information Systems
Cited in: Paré & Sicotte (2001)

7. Existing Models
Cheney & Dickson (1982)
Investigated the relationship between
“Technological Sophistication” (hardware,
software, nature of application systems),
“Organizational Sophistication”
(information resources management
activities), and system performance
Cited in: Paré & Sicotte (2001)

8. Existing Models
Saunders & Kellers (1984)
IS maturity referred to as the “sophistication of the
mix of applications provided by the IS function”
Raymond & Paré (1992)
IT sophistication defined as “a multi‐dimensional
construct, which includes aspects related to
technological support, information content,
functional support, and IT management practices”
Cited in: Paré & Sicotte (2001)

9. Conceptual Framework
IT Sophistication in Hospitals
“The diversity of technological devices and
software applications used to support patient
management and patient care, clinical
support, and administrative activities” as
well as “the extent to which computer‐based
applications are integrated (electronic and
automatic transfer of information)”

10. Conceptual Framework
3 Dimensions of IT Sophistication
Technological Sophistication
○ Diversity of hardware devices (e.g. medical imaging, bar coding
devices, data warehousing, wireless networks, PACS equipment)
Functional Sophistication
○ Proportion and diversity of processes or activities (e.g. vital sign
recording, medication administration, staff scheduling)
Integration Sophistication
○ Degree to which computer‐based applications are integrated both
internally and externally

11. Conceptual Framework
3 Health Care Activity Domains
Patient Management and Patient Care
○ Patient management, order entry/results reporting, clinical
notes, care planning, vital signs recording, ER, OR, etc.
Clinical Support
○ Laboratory, pharmacy, radiology
Administrative
○ Billing & finance, HR, materials management

12. Conceptual Framework
Source: Paré & Sicotte (2001)

13. Conceptual Framework
Technological Sophistication
Office automation systems
Human‐computer interaction devices
Storage and compression devices
Data distillation systems
Connectivity devices
Each of these can be used in different activities

14. Conceptual Framework
Functional Sophistication
Technology alone doesn’t measure an
organization’s IT sophistication well
One technology can be used for different
activities, each of which has varying impact
on care
Can be viewed as the level of
computerization in the work processes

15. Conceptual Framework
Integration Sophistication
Electronic and automatic transfer of
information
Internal Integration
○ With other systems in the hospital
External Integration
○ With external entities’ systems

16. Instrument Evaluation
Validity
An instrument does what it is intended to
do (Nunnally, 1978)
The degree to which a test measures what it
was designed to measure (Wikipedia, 2008)
The degree to which the measurement
measures what the investigator wants to
measure (Friedman & Wyatt, 2005)

17. Instrument Validation
Reliability
The extent to which measurements are
repeatable, by different persons, on different
occasions , with alternative instruments for
measuring the same thing (Nunnally, 1978)
The consistency of a set of measurements or
measuring instrument (Wikipedia, 2008)
The degree to which measurement is consistent
or reproducible (Friedman & Wyatt, 2005)

18. Validity vs. Reliability
Source: http://ibis.health.state.nm.us/resources/ReliabilityValidity.html

19. Types of Reliability
Test‐retest reliability
Does the measure deliver the same answer when
applied in different time periods?
Internal consistency reliability
Multi‐item instrument
○ Does the measure yield consistent results across
different indicators (items)?
Multiple raters: Inter‐rater reliability
○ Does the measure yield consistent results when different
observers rate the same thing?
Source: Neuman (2005)

20. Types of Validity
Face Validity
On the face of it, do people believe that the
definition and method of measurement fit?
Source: Neuman (2005)

21. Types of Validity
Content Validity
Is the full content of a definition
represented in a measure?
Source: Neuman (2005)

22. Types of Validity
Construct Validity
Do the various indicators operate in a consistent
manner?
Convergent Validity, Discriminant Validity
Source: Neuman
(2005)

23. Types of Validity
Criterion Validity
Does a measure correlate with some
standard or criterion?
Concurrent Validity, Predictive Validity
Source: Neuman (2005)

24. Methods: Content Validity
20 in‐depth interviews with health care and IT
specialists at the Montreal Jewish Hospital (MJH)
Respondents asked to identify technologies and
computer‐based applications in their organization and
elsewhere, processes supported by the systems, &
degree of integration
Inputs used to modify instrument
Pretest with 3 HIS directors with in‐depth interviews
to refine instrument (mostly cosmetic)

25. Methods: Reliability, Construct &
Concurrent Validity
Survey of HIS directors in 2 of the largest
Canadian provinces
Sampling frame: a government list of names,
addresses, and phone numbers of each medical
center in Quebec (n = 80) and Ontario (n = 106)
Subjects contacted by phone to request
participation (10 refused)
Mail survey with 6‐week follow‐up letter
Total period: 4 months

26. Results
Response rate 62.4% (Quebec 73.8%, Ontario 53.8%)
Source: Paré &
Sicotte (2001)

27. Results
Source: Paré &
Sicotte (2001)

28. Results
Reliability: Cronbach’s alpha for the technological,
functional, and integration dimensions of each
activity domain and for overall dimensions
○ Low: items don’t measure the same thing or too few items
○ High (usually > 0.70): items “internally consistent”
Source: Paré & Sicotte (2001)

29. Results
Construct validity: interdimension correlations and correlations
with an adjusted overall dimension score (after removing scales
of that dimension)
○ Interpretation: High & significant correlation means consistent results
between different measures (dimensions + domains) of IT sophistication,
thus construct validity
Source:
Paré & Sicotte (2001)

30. Results
Criterion validity: correlation of each dimension and 6
other variables: the hospital’s present stage of IT maturity
based on Nolan’s 5‐stage model, annual budget, annual IT
budget, number of IT staff, HIS director’s educational level
and IT management experience
Source: Paré & Sicotte (2001)

31. Results
Overall functional sophistication ‐ Not significant between
Quebec & Ontario
Some significant variations exist e.g.
○ Higher percentage of electronic signature for medical chart
documentation in Ontario
○ Higher percentage of OR computerized processes in Ontario
Medication administration, staff scheduling, order
transcription, and historical record keeping are most
computerized nursing activities in both provinces
Vital signs recording and other nursing activities are
among the least computerized ones
Source: Paré & Sicotte (2001)

32. Results
Both provinces exhibit low level of technological
sophistication overall (not significant different)
Telemedicine, expert systems, voice recognition systems
for notes transcription are not available in most hospitals
Smaller percentages of Quebec hospitals installed
workstations in the hallways or at bedside or used
portable devices
PACS are diffused more widely in Ontario
Lower number of Quebec hospitals have a Web site
Source: Paré & Sicotte (2001)

33. Results
Low level of systems integration overall, with Quebec
significantly less integrated.
ER and OR systems are somewhat less integrated than
ADT, scheduling
In many clinical support departments (labs, radiology,
pharmacy), systems are mostly stand‐alone
In general, clear sign of lack of internal and external
integration
Source: Paré & Sicotte (2001)

34. Conclusion
Instrument is reliable and valid
Findings support breakdown of IT sophistication into
technological, functional, and integration dimensions
IT sophistication instrument is helpful for
○ Describing a hospital’s state of IT sophistication
(cross‐sectional, longitudinal)
○ Diagnosis of aspects with low sophistication within a
hospital for future improvements
○ Comparisons between groups of hospitals

35. Limitations/Critique*
IT Adoption has 2 aspects: Depth & Breadth
○ Depth: Level of functionalities computerized
○ Breadth: Extent of adoption across organization
○ IT Sophistication instrument focuses only on depth
Instrument developed and validated in 2001
Internal consistency reliability among items may not be
appropriate for heterogeneous (“formative/causal”)
indicators. Instead, should use inter‐rater reliability or test‐
retest reliability
* Presenter’s own opinion. Not discussed in the original article

36. References
Friedman CP, Wyatt JC. Evaluation methods in biomedical
informatics. 2nd ed. New York (NY): Springer; 2005, c2006.
386 p.
Neuman WL. Social research methods: qualitative and
quantitative approaches. 6th ed. Boston (MA): Allyn & Bacon;
2005, c2006. 592 p.
Nunnally JC. Psychometric theory. 2nd ed. New York (NY):
McGraw‐Hill; c1978. 701 p.
Paré G, Sicotte C. Information technology sophistication in health
care: an instrument validation study among Canadian
hospitals. Int J Med Inform. 2001 Oct;63(3):205‐223.

37. References
Rogers EM. Diffusion of innovations. 5th ed. New York (NY): Free
Press; 2003. 551 p.
Wikipedia [Internet]. San Francisco (CA): Wikimedia Foundation;
c2001‐2008. Reliability (statistics); [modified 2008 Sep 16;
cited 2008 Oct 5]; [about 4 screens]. Available from:
http://en.wikipedia.org/wiki/Reliability_(statistics)
Wikipedia [Internet]. San Francisco (CA): Wikimedia Foundation;
c2001‐2008. Validity (statistics); [modified 2008 Aug 30; cited
2008 Oct 5]; [about 7 screens]. Available from:
http://en.wikipedia.org/wiki/Validity_(statistics)

38. Appendix
What is Health IT?
Can be viewed as a subset of innovations
What is Adoption?
Rogers’ Diffusion of Innovations Theory (1962‐2003)
Implement-
Knowledge Persuasion Decision Confirmation
ation
Rejection Adoption
Source: Rogers (2003)