3. Alberta Innovates Health Solutions
• Research and Innovations System (2010) - Formerly
Alberta Heritage Foundation for Medical Research
• Established in 1980 to support medical/health
research
• AIHS has provided unprecedented opportunities in
research
• Contributed > $1 billion to the scientific community
• Supported > 8,500 researchers approximately
4. AIHS Mission
“to support research and
innovation activities to improve
the health and well-being of
Albertans and create, through
innovation, health related social
and economic benefits for
Albertans”.
8. Pathway to Impact
• # quality of life
• # modifiable risk
factors
• # accessibility
Health Impact
Enabling
Outcomes
Lead to Impact
• # reports of
inventions
• # spin-off companies
and jobs
• Social benefits
Socioeconomi
c Impact
10. Emerging Methodologies – Bibliometrics
Bibliometrics: are a set of methods and procedures
used in the measurement of bibliographic records”
including scientific and technological outputs
(e.g., journal publications and patents; Science
Metrix, 2009)
Uses:
• Impact assessments
• Productivity trends
• Collaborative relationships
• Program innovation
• Patterns and intensity of knowledge dissemination
11. Considerations
• Time lags – time must pass for citations to occur – self
citations
• Need to account for differences in organizations, technical
fields, disciplines
• Bias of databases towards English language
• Contribution - uncertainty whether outputs are result of one
grant or multi-grant
• Multiple Lines of Evidence – complimentary – emerging
indicators
12. An Example:
:
Bibliometric Indicators
• Publications – in top journals, top 20
researchers, comparative publication rates –
Productivity/Activity
• Specialization Index – Contextual and Structural
• Average Relative Impact Factor - Quality
• Average Relative Citation (ARC) – Quality and Uptake
• Collaboration rates: interprovincial, international, top 10, field
comparisons, academic user collaborations - Outreach
April 2011 12
13. Positional Analysis for Canada and Selected Countries, 1985-1989
2
US
UK
France
Canada Canada
ARC
1 Sw eden
Australia
Sw itzerland
Denmark
Finland
Norw ay
0
New -Zealand
0 1 2
SI
The ARC and the SI for the period is estimated using the average ARC and SI over the period
Source – Science Metrix and Graph CIHR
14. Positional Analysis, for Canada and Selected Countries, 1990-1999
2
US
UK
France
Canada
Canada
ARC
Australia
1
Sw eden
Sw itzerland
Denmark
Finland
Norw ay
0
New -Zealand
0 1 2
SI
The ARC and the SI for the period is estimated using the average ARC and SI over the period
Source – Science Metrix and Graph CIHR
15. Positional Analysis for Canada and Selected Countries, 2000-2008
2.0
US
UK
France
Canada Canada
Australia
ARC
1.0 Sw eden
Sw itzerland
Denmark
Finland
Norw ay
0.0 New -Zealand
0.0 1.0 2.0
SI
The ARC and the SI for the period is estimated using the average ARC and SI over the period
Source – Science Metrix and Graph CIHR
16. Emerging Methodologies – Patent Citation
Trees
Patent citation trees show forward citing of the
patents generated by funded completed projects.
Uses:
• Citations can be used to track the dissemination of
technical knowledge to subsequent publications and
patents.
• Help understand the influence of new knowledge
created on others.
• Relevance, extent of dissemination and impact.
17. Current Methodologies – Patent Trees
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 121.
18. An Example: Network Diagrams
Source Time Lags a Feasibility study Rand Europe – Draft 2011
19. Case Studies Using Pay Back
Case studies are in-depth investigations into a
program, project, facility or phenomenon, usually to examine what
happened, to describe the context in which it happened, to explore
how and why, and to consider what would have happened
otherwise.
• Historical Tracing – start with outcomes back
• Forward Tracing – start with research of interest and trace
related events going forward
20. Impact Assessments – Pay Back Case
Studies
Use:
• Identify exemplary or best-practice experiences
• Identify the long term payback from health research
• Explore factors associated with successful research translation
• Provide insights to inform future funding policy
Considerations:
• Results may not be generalizable
22. Outcome Mapping – Developmental Evaluation
Outcome mapping is a participatory method for
planning, monitoring and evaluation; focuses on
changes in behaviour of those with whom the project
or programs works and oriented towards social
change and organizational learning.
Uses:
• Focus on outcomes as changes in behaviour
• Sphere of influence
• Logical path
24. Next Steps - Performance Composite Index
• Using Factor Analysis and Regression Analysis
reduce the number of metrics and construct index in
relation to each program goals.
• Develop weighting system for goals – evidence from
other studies
• Ability to get a clear view of program performance.
Across a portfolio of programs.
• Indicate performance at different stages of the
innovation process
•
25. Promising Practices
• Science of Science Network
• Alliances
• Innovation Platforms – Collective measurement and
Shared Value
• Forum on Research Evaluation Methodologies
26. Resources
Buxton MJ & Hanney S (1996) How can payback from health services research be assessed? Journal of Health
Services Research & Policy, vol 1, no 1, pp 35-43.
Panel on Return on Investment in Health Research. (2009). Making an Impact: A Preferred Framework and Indicators to
Measure Returns on Investment in Health Research. Canadian Academy of Health Sciences: Ottawa, Canada
Earl S. et al. (2001), Outcome Mapping: Building Learning and Reflection into Development Programs. International
Development Research Centre, Canada
Gamble, J. (2008). A developmental evaluation primer. The J.W. McConnell Family Foundation: Montréal.
Michael Quinn Patton (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use.
Guilford Press: New York.
Kaplan, RS & Norton, DP (1992). The Balanced Scorecard: Measures that Drive Performance. Harvard Business
Review, (January-February), pp. 71-79.
Kaplan, RS & Norton, DP (1996). Using the Balanced Scorecard as a Strategic Management System. Harvard Business
Review, (January/February), pp. 75-85
Kaplan, RS & Norton, DP (1996). Balanced Scorecard: Translating Strategy into Action. Harvard Business School Press:
Cambridge, Massachusetts
Kaplan RS & Norton DP (2000). The Strategy Focused Organization. Harvard Business School Press: USA
Kaplan RS & Norton DP (2004). Strategy Maps: Converting intangible assets into tangible outcomes. Harvard Business
School Press: Boston
McLaughlin, JA, & Jordan, JB (2004). Logic Models. In, J. S. Wholey, H. P. Hatry & K. E. Newcomer (Eds). Handbook of
Practical Program Evaluation, 2nd Edition .San Francisco, CA: Jossey-Bass.
Mercure, S, Cote, G, Archambault, E. (2007). Scientific Positioning of Alberta in Health Research Specialties. Science-
Metrix. Montreal, Canada. Retrieved from http://www.ahfmr.ab.ca/download.php/b833a02029ebd9e62fd286168dfc2615
27. Contact information
Kathryn Graham (PhD)
Director, Performance Management
Alberta Innovates Health Solutions
1500 – 10104 103 Avenue N.W.
Edmonton, Alberta, Canada T5J 4A7
Email: kathryn.graham@albertainnovates.ca
THANK YOU
Notes de l'éditeur
Acknowledge, the Performance Management Team: Amy Wong, Heidi Chorzempa, May Kharaghani, Daniel Zhang and Andrew LeJeune (What about me? Liza)
Evaluating complex systems requires using novel approaches as well as multimethod approach to provide an evidence base for policy making as well as informing program and organizational decision making and demonstrating results.
The Edmonton based Alberta Innovates Health Solutions is a not for profit provincial health research funding agency. In early 2010 AHFMR was incorporated under the new realigned research and innovation system designed to strengthen the provinces role as a world leader in using science to seek solutions.
Belief that supporting health research will lead to health, health system benefit.
One of the main stays are surveys and asking researchers for information hence administrative burden
Comprehensive and Integrated:Mixed Methodology – Qualitative and Quantitative collection, analysis and reportingMulti method – case studies to bibliometricsMulti dimensionalIntegrated into the business modelPhased Approach
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology. CAHS : QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Capacity building, advancing knowledge and informing decision making are the enabling outcomes to achieve health and socio-economic impact
Changes in the external environment drive change in AIHS which also impacts how we assess success and what is important to our different key stakeholder groups
MethodsExpert Opinion – Peer ReviewSurveysInterviewsCase StudiesMeasures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology. CAHS : QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology.Difficult to account for organizational S&T mandate, priorities and levels of resources ($, infrastructure and FTEs)
ARC – a direct measure of scientific impact based on paper citation counts (2-year citation window). Normalized by subfields of Science and yearARIF – a proxy for the ‘quality’ of the journal in which papers are published. It is based on symmetric impact factors of journal. Normalized by subfields of science and year.Specialization Index – an indicator of the intensity of research of a given geographic or organizational entity (e.g. a country) in a given research area (e.g. domain, field) relative to the intensity of the reference entity (e.g. the world) in the same research area. CAHS :QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Average Relative Citation and Specialized IndexAll questions addressed1 represents0 is the world levelThe results suggest that within a 20 year window, most countries had made some significant advances in terms of improving the degree of research specialization (SI) and the quality of knowledge ARC produced in the Medical Sciences. The size of the circles is proportional to the number of papers produced by the entity. The position of an entity 1 top right – high impact scientific production, entities specialize in the given domain and their activities have a high impact – their papers are more frequently cites that the world average in this field.
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology.
Figure 8-5 shows a patent tree for one of the first 50 completed ATP projects, A project to develop wafer ion-implantation, carried out by Diamond Semiconductor Group (DSG). Patent trees represent one aspect of knowledge dissemination, the patent trees extend our understanding of the influence of the new knowledge created on others. – relevance, extent of dissemination, and impact.
Advantage - Able to illustrate the density of connectivity, which is important for identifying key papers (from the researcher’s own lens) and the researchers’ most influential periodDisdvantabe3D objects can be difficult to understandDifferent arrangements of the research events could provide quite different impressionWe simplify the highly complex diagram shown previously in Figure 4 by showing only the linkages that are directly or indirectly linked to Patent 312 and move the relevant events inside the circle (Figure 6). Patent 312, granted in 1999, has observable direct linkages to three other key researcher markers: Grant 151 (1999), Paper 16 (1993) and Paper 20 (1993). The latter two have direct linkages to six and four papers respectively. These papers are further connected to four older papers. The oldest paper goes back to 1977 (Paper 0). Therefore, we would conclude that there was a time lag of 22 years, between Paper 0 and Patent 312.
Historical TracingEmphasis on tracing chronologically a series of interrelated developments leading from research to ultimate outcomes or from outcomes back to the factors that spawn them.Forward TracingWhen the objective is to evaluate a given project, forward tracing, where the analyst starts with the research of interest and traces the evolution of related events from the point forward, is generally more manageable and cost-effective than backward tracing….
Mental Health Retrosight investigates the translation of basic, clinical and interventional research into clinical application and community practice.Uses bibliometrics and literature review for hot topics and going forwardUses delphi surveys and clinical guidelines for interventions going backwards
