1. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
April 25, 2012
Patient-Centered Outcomes Research Institute
National Priorities and Research
Agenda

2.  Review Genesis of National Priorities and Research Agenda
 Review Public Comment Process
 Share Public Comment Findings
 Recommend PCORI Response
 Board Vote on Recommended Changes
Objectives for Today
2

3. 3
The What and Why of National Priorities and Research
Agenda
 Mandated in the legislation (including
Public Comment period)
 Pre-requisite for releasing funding
announcements
 Preliminary roadmap for PCORI research
activities
 Envisioned as a living document

4. Developed
Framework to be
used for refining
priorities and for
determining
Research Agenda
and funding
announcements
Reviewed initial
stakeholder input
advising us to not
“reinvent the wheel”
Development of Draft National Priorities and
Research Agenda
4
Initial Stakeholder
feedback
Reviewed prior CER
frameworks (e.g.,
IOM, FCCCER,
National Priorities
Partnership, and
NQF)
Environmental
scan of existing
priorities and
criteria
Identified broad
priorities from prior
frameworks and the
statutory criteria for
PCORI
Candidate
priorities and
criteria identified
Framework to
inter-relate
Priorities and
Criteria

5. Establishing PCORI’s First National Priorities
for Research and Initial Research Agenda
9
Criteria outlined
by law
5
Draft priorities
proposed
Corresponding
agenda drafted
Public input
received and
evaluated
Priorities and
agenda revised
and approved
First primary
funding
announcements
issued
Aug-Dec 2011 Jan-Apr 2012 May 2012
5

6. Patient and Caregiver Focus Groups
6
Columbus, OH (Dec. 7)
• Parents of children with ADD/ADHD
(caregivers)
• Patients with Mental Health
conditions
• Patients who survived Cancer
• Insured and uninsured
• Age: 30-70+
Phoenix, AZ (Dec. 8)
• Patients with Respiratory Disease
(chronic bronchitis, emphysema)
• Hispanic patients with mix of
chronic conditions
• Patients with Heart Disease
• Insured and uninsured
• Age: 21-69
Atlanta, GA (Nov. 21)
• Patients with Diabetes
• Caregivers to Alzheimer’s patients
• Patients with Chronic Pain
• Insured and uninsured
• Age: 21-75+
Baltimore, MD (Nov. 9)
• Patients with Arthritis
• Parents of children with Pediatric
Asthma (caregivers)
• Patients with mix of chronic
conditions
• Insured and uninsured
• Age: 21-69

7. Receiving Public Feedback on the National
Priorities and Research Agenda
7
• Clinician Focus Groups
o Clinician focus groups took place in cities across the US
o Philadelphia
o Birmingham
o California
o Chicago
o Four groups of physicians and four groups of nurses
o Behavioral Health Professionals
o Physician Assistants

8. A commitment to public engagement
8
Formal 53-day Public Comment Period
• Nearly 500 comments received through
website, e-mail or postal mail
• All comments will be posted at pcori.org
Additional Forums
• National Patient and Stakeholder Dialogue
• Patient, caregiver and clinician focus groups
• Individual meetings with diverse mix of
stakeholders

9. Public Comment Yielded Diverse Array of Input
and Feedback
Broad range of comment sources
301 Web survey (64%) 116 Email/ letter (24%)
57 Stakeholder event (12%)
474TotalComments
9

10. Those Who Commented Said They Understand
the National Priorities…
Addressing Disparities Accelerating PCOR and Methodological Research
Assessment of Options Improving Healthcare Systems Communications and Dissemination Research
“How well do you understand
the National Priorities?”
10

11. Rigorous, Systematic Review and Analysis
of Comments
Computer algorithm to identify key terminology
Each narrative comment reviewed and analyzed by
3 people
Stakeholder comments aggregated to 15 key
themes
Themes compared to National Priorities and
Research Agenda to identify gaps
Gaps reviewed to determine options for PCORI
response to themes
11

12. Options for Response
 Change Language Within the Research Agenda
 Embed in PCORI Operations and Processes
 Address in Summary Document
 Future Consideration
12

13. Recommends that PCORI choose a specific condition, disease
area, or other issues in the Research Agenda and National
Priorities
PCORI has proposed a condition-neutral Research Agenda and has introduced
specificity through its comparative nature and emphasis on patient centeredness.
While future funding announcements may specify conditions, the overall mission of
PCORI is not served by excluding any conditions if there is compelling reason for a
patient centered, comparative clinical effectiveness study. We added language to
elaborate on our position.
Recommends that PCORI partner with organizations and
stakeholders to carry out its mission
PCORI is committed to efficient use of its research investments. Where appropriate,
PCORI will partner with other organizations after a transparent decision-making
process and consideration of conflicts of interest. This theme impacts PCORI
processes, rather than funding subjects, so no specific language changes were made
to the document.
Recommends greater focus on the patient, with particular
attention to methods of engagement
PCORI has fully endorsed and appreciates the centrality of patient engagement to its
mission. The National Priorities and Research Agenda reflect the patient centered
focus of PCORI and include many of the themes from the public comments.
Language has been added to the Agenda to specifically reflect the need for study of
self care and to more clearly define personalized medicine.
Recommends a greater focus on care coordination PCORI appreciates the need to study care coordination and has expanded the
language in the Research Agenda to reflect its importance.
Recommends funding towards improving patient and provider
health literacy and education
Improving communication between patient and provider is one of the five PCORI
National Priorities for Research. Language has been added to the Research Agenda
to reflect the importance of health literacy to achieving this goal.
PCORI Response to Key Themes-1
13

14. Recommends funding for and use of health IT infrastructure,
networks, tools and patient data acquisition efforts in and outside
the practice setting
The foundation for performing comparative clinical effectiveness requires substantial
health IT and data infrastructure. The National Priorities and Research Agenda
contain substantial language about this infrastructure. Therefore, no additional
language was added to the document. PCORI will support reusable infrastructure for
comparative clinical effectiveness research.
Recommends that PCORI pay greater attention to the role of
caregivers and other stakeholders in the patient decision making
process
PCORI appreciates the role of caregivers in patient centered care and has mentioned
them in the document and included studies of caregiving in the Research Agenda.
Therefore, no additional language was added to the document.
Recommends that PCORI pay greater attention to access to
care, including the social and environmental determinants that
determine access and use of care
Access to care is a key issue for patients. Language has been added in both the
comparative assessment and the healthcare systems Research Agenda topics to
include the comparative study of access as a determinant of health.
Recommends that PCORI provide greater rationale and
transparency in the public comment, grants, and research
evaluation processes, as well as the performance measurement
process for PCORI as a whole
PCORI is committed to fully transparent processes as it works towards achieving its
mission. The Research Agenda articulates the ongoing engagement that will occur
continuously as PCORI evolves and funds research. Therefore, no additional
changes were made to the document. PCORI intends to roll out a comprehensive
communications and engagement plan that will clearly define when and how
stakeholders can provide input into PCORI decision making.
Recommends that PCORI's research and funding should impact
the practice setting, with particular attention to patient and
provider behavioral change needed to obtain true shared decision
making
PCORI is fully committed to the idea that its research should improve decision making
and help patients at the point of care. Language has been added to the section
“Establishing the Scope of the Research Agenda” to emphasize the importance of
using the evidence developed through PCORI research to change the way medicine
is practiced.
PCORI Response to Key Themes-2
14

15. Recommends that PCORI place stronger emphasis on patients
with multiple conditions, especially chronic conditions
PCORI understands the difficulty of managing multiple chronic conditions when most
evidence is generated in trials that exclude these patients. Language has been added
to emphasis this in the Research Agenda.
Recommends that PCORI study new and expanded roles for
allied health professionals
PCORI recognizes the diverse health professionals involved in patient centered care.
In the Research Agenda, the description of allied health professionals has been
expanded to be more inclusive of all of potential members of a health care team.
Recommends paying attention to international models PCORI recognizes the significant achievements of many countries in developing the
methods and practices of patient engaged comparative clinical effectiveness research
that may inform investigators as they seek PCORI funding. As this is not central to
PCORI research, no change is proposed to the priorities or agenda.
Recommends exploring novel methods to obtain patient centered
focus
PCORI supports the approach of exploring innovative methods for focusing on the
patient. The fundamental basis of PCOR, however, is the science of evidence-based
medicine. PCORI will support and promote approaches that seek rigorous, scientific
results; therefore no changes were made to the document.
Recommends that PCORI study rare diseases PCORI recognizes the challenges faced in studying rare diseases. In the Research
Agenda, language about rare disease has been expanded.
PCORI Response to Key Themes-3
15

16. Discussion

17. Vote on Changes Outlined in Slides 13-15

18.  PCORI Stakeholders
 Thank you for your thoughtful input into the first version
of the PCORI National Priorities for Research and
Research Agenda
 PCORI Program Development Committee
 Thank you for all your hard work in the development and
refinement of these documents
 The detailed Summary Document of changes to the National
Priorities and Research Agenda will be posted on pcori.org
by April 30, 2012
Thanks
18

19. April 25, 2012
PCORI Pilot Projects Selection Committee
Recommendation to Board

20. Selection Committee Members
Kerry Barnett, JD
Carolyn M. Clancy, MD
Arnold Epstein, MD
Sherine Gabriel, MD, MSc
Christine Goertz, DC, PhD (Ex-Officio Committee Member)
Leah Hole-Curry, JD
Gail Hunt
Grayson Norquist, MD, MSPH (Selection Committee Chair)
Joe Selby, MD, MPH (Ex-Officio Committee Member)
Clyde Yancy, MD, MSc

21. Activities to Date:
 Members of a workgroup made up of BOG members met on
a conference call January 25th and arrived at eight potential
balancing criteria
 Selection Committee was appointed and met initially on
March 6th at which time they refined the balancing criteria
and proposed options for selecting applications to be funded
 Applications in potential fundable range were checked for
accuracy of balancing criteria
 The Committee held a conference call on March 19th at which
time they further refined the balancing criteria and decided
on two options for selecting a slate of awards to be funded
 The Committee held a conference call on April 2nd and
developed a final recommendation for the Board.
Background and Updates

22. The Process
Step One:
Determine Universe
Step Two: Determine
Approach
Step Three: Develop
Core Slates
Step Four: Analyze
Balance
Step Five: Make Final
Recommendation
Step Six: Approve
The decision was made to eliminate all applications with a score
above 3.0 from consideration for further funding. This yielded a
potential fundable pool of 80 applications.
The criteria to be used for balancing were refined and two
options for selecting a base slate of applications were proposed.
Staff applied the two selection process options to the top 80
applications, yielding two possible sets of fundable applications
(one with 37 selected; the other with 50).
Using the balancing criteria staff analyzed the balance of each of
the two possible sets to determine if balance was achieved or if
additional balancing activities would be required.
The Selection Committee reviewed the two possible sets of
fundable applications to determine if balancing was needed. The
group decided on two options for the Board and selected one for
recommendation.
The Board of Governors will vote on the recommendation.
This process was completed using only generalized information regarding the applicant. No Selection

23. The selection comm. considered all potential options for selecting the slate of
fundable applications and decided to go with an option that would use priority
score first and then ensure the top two applications were selected in each
review group. All selection committee members were blinded as to the names
and affiliations of the applicants.
• Percentiles (as noted on the NIH web site) are calculated usually for study
sections that have had at least three meetings – traditionally, the last three
rounds during a year. Thus, percentiles for the Pilot Projects Program do
not have the same meaning as NIH percentiles. They reflect the
application’s ranking within a single study section meeting.
• Priority scoring is likely to be the most reliable measure across all groups
since all reviewers had the same training regarding how to score.
• We don’t know (since there is no history to the review group) if a “poor”
score is due to a review group that just scores “harsher” or the grants they
had were just not very “good”.
Rationale for Selection Method

24. Rationale (cont’d)
• NIH attempted to assign applications to reviewers based on areas
of interest, though it was difficult given the large number of
applicants and PI’s did not necessarily self-assign correctly as well
as listing more than more area.
• However, we did want to take into consideration the potential that
some groups could be harsher and give some weight to ranking
within the review group – so, we decided to also pick up the top
two applications in each review group.

25. Balancing Criteria
Balancing
Criteria
Definition Operationalization Source
Area of Interest*
The eight areas of interest listed in
the PFA
At least 2 unique applications but no
more than 50% in any one area
At least two members of the Selection
Committee read each application
abstract and determined the primary
areas of interest.
Population*
Defined as addressing 1) Specific
ethnic or cultural group, 2)
disabled populations, 3) children,
and 4) elderly populations.
At least 1 application in each of the
four categories
PCORI staff reviewed abstracts to
determine if the application had a clear
focus on one or more of the four
populations.
Condition*
The disease or condition used to
demonstrate the approach
No more than 25% in any major
category within the final slate.
Staff reviewed the abstracts to
categorize conditions addressed, if any
Stakeholder/Patient
Involvement*
The average score given by the
three reviewers for this criterion in
the initial merit review group
Any application added to the core
slate will have a score of 1 or 2 (except
methods focused applications)
The average score from the IRG review
was used.
Geography
Geographic location of institutional
affiliation
Will be reported but not used for
balance.
The state or country on the application
face page for the PI.
Method
The innovation of the research
method
Only to be used if balance is needed
within the area of interest related to
methodologies (8).
The methodology committee would
make the determination, if needed.
PI Discipline
Categorization of PI qualifications
based on primary area of expertise
Insufficient data to use
PI Seniority
Whether the PI has received
Insufficient data to use
Eight potential criteria were discussed and
refined. The top four were proposed as the
most appropriate for balancing.

26. Areas of Interest (summary)
 Present recommended slates to BOG in April
 Make Grant Award announcements no later than the
May BoG meeting
 Incorporate Lessons Learned into 4 PFAs to be
released in May
 Create and publish report on selection process to
ensure transparency
 Developing, testing, refining, and/or evaluating new or existing methods and approaches
that can inform the PCORI national priorities.
 Developing, testing, and/or refining existing methods for bringing together patients,
caregivers, and other stakeholders in all stages of a research process.

Developing, refining, testing, and/or evaluating patient-centered approaches, including
decision-support tools, for translating evidence-based care into health care practice in
ways that account for individual patient preferences for various outcomes.
 Developing, refining, testing, and/or evaluating methods to identify gaps in CE knowledge
such as tools collecting and assessing patient- and provider-perceived gaps.
 Identifying, testing, and/or evaluating patient-centered outcomes instruments.
 Identifying, testing, and evaluating methods that can be used to assess the patient
perspective when researching behaviors, lifestyles, and choices.
 Identifying, testing, refining and/or evaluating methods for studying the patient care team
interaction in situations where multiple options exist.
 Advancing analytical methods for CER

27. Option 1: Qualities of Proposed Slate
Applications with a priority score of 25 or better and
ensured the inclusion of all applications in the top two
of each review panel. This resulted in 50 applications.
Underserved PopulationsAreas of Interest*
Stakeholder Engagement Scores
68%
Yes
Chronic Condition 7 14%
Mental Health 5 10%
Cardiac 4 8%
Cancer 3 6%
These conditions
and focuses are
those with the
highest
representation.
2
2
19
0
12
4
6
7
A1
A2
A3
A4
A5
A6
A7
A8
Conditions Categories
4 4
2
9
0
2
4
6
8
10
Pediatric Aging Disabled Racial/Ethnic
Group
Score = 1 11 22%
Score = 2 32 64%
Score = 3+ 7 14%

Balance

Balance

Balance

Balance
Costs: Year 1--$15,843,724; Year 2--$15,005,483
The highest representation is Area 3 at 38%

28. Option 1: Balancing
Option 1 takes the applications with a priority score of 25 or better and ensures
at least the top two applications from each panel are included. This option
yields 50 applications and has only one balancing issue:
 Issue: Area of Interest 4--There are no applications within this slate with that
designation as a primary area of interest.
Developing, refining, testing, and/or evaluating methods to identify gaps in CE knowledge such as
tools for the ongoing collection and assessment of gaps as perceived by patients and providers. Of
special interest are gaps that are particularly relevant to vulnerable populations, including but not
limited to, low-income populations; underserved minorities; children; the elderly; women; and
people with disabilities, chronic, rare, and/or multiple medical conditions.
 Balancing Recommendation: There are no applications within the top 80
that have this designation as a primary area of interest. The committee
recommends we not balance on this area and propose it be a focus in one of
the new PFA’s or use a contract mechanism to address.

29. Option 2: Qualities of Proposed Slate
Underserved populationsAreas of Interest*
Stakeholder Engagement Scores
These conditions
and focuses are
those with the
highest
representation.
1
2
16
0
7
3
4
6
A1
A2
A3
A4
A5
A6
A7
A8
Mental Health 5 13.51%
Chronic Condition 4 10.81%
Cancer 3 8.11%
Conditions and Focus Categories
3 3
2
7
0
2
4
6
8
Pediatric Aging Disabled Racial/Ethnic
Group
Score 1 9 24%
Score 2 24 65%
Score 3+ 4 11%

Balance

Balance

Balance

Balance
Costs: Year 1--$11,803,263; Year 2--$10,944,425
The highest representation is Area 3 at 43%
To remain as close to 40 grants (the number approved
by the BoG), applications with a priority score of 24 or
better were selected and ensured the inclusion of the
top application in each review panel. This resulted in
37 applications.

30. Option 2: Balancing
To remain as close to 40 grants (number approved by the BoG), option 2 takes applications
with a priority score of 24 or better and ensured the inclusion of the top application in each
review panel. This resulted in 37 applications.
1. Issue: Area of interest 4— This is the same issue encountered under Option 1.
Recommendation: This issue should be handled in the same way as in Option 1.
2. Issue: Area of interest 1—There is only 1 application within the slate that addresses this
area.
Developing, testing, refining, and/or evaluating new or existing methods (qualitative and quantitative) and
approaches that can inform the process of establishing and updating national priorities for the conduct of
patient-centered outcomes research (PCOR). This may include research prioritization approaches (such as Value
of Information (VOI), burden of illness, peer review/expert opinion/Delphi approaches) or methods for
incorporating the perspectives of patients or other stakeholders into the development of national priorities.
Recommendation: To add an additional application with a primary area of interest of 1,
the committee looked within those applications with a priority score of 25. Within that
group there is one application with a primary area of interest of 1.

31. Geographic Distribution
AK 1
AR 1
AZ 1
CA 6
CO 2
CT 1
DC 1
FL 1
IA 1
IL 1
MA 5
MD 1
MD 1
MI 1
MN 1
MO 1
NC 3
OH 2
PA 3
RI 1
TN 1
VA 1
WA 1
AK 1
AR 1
AZ 1
CA 8
CO 3
CT 1
DC 1
FL 1
GA 1
IA 1
IL 1
MA 8
MD 3
MI 2
MN 1
MO 1
NC 3
NY 1
OH 2
PA 4
RI 1
SC 1
TN 1
VA 1
WA 1
Option 1 Option 2

32. Discussion and Voting
Selection Committee
Recommendation
• The committee recommends the Board vote for Option 1 and that
we not try to balance on Area 4.