1. Using Rubrics to Enhance Extension
Program Planning and Evaluation:
A Health Insurance Literacy Example
Bonnie Braun, PhD
Professor, Department of Family Science
Specialist, University of Maryland Extension
Faculty Scholar,
Horowitz Center for Health Literacy
University of Maryland
bbraun@umd.edu

2. Rubrics

3. Why Rubrics?
 Encourages consistent and thorough
evaluation
 Assists in evaluation of existing and new
programs, curricula and educational materials
 Permits multiple individuals to make judgments
using common definitions

4. University of Maryland Health Smart
Health Insurance Literacy Initiative
(HILI)
 Emergent area for education
 Strives to enhance the confidence and
competence of consumers in making health
insurance purchasing decisions
 Targets both those who already purchase
insurance and those newly eligible for
health insurance under the Affordable
Care Act
 From Maryland-only to multi-state initiative
 Pilot testing launched April 1, 2013

5. Design Specifications &
Products
Comprehensive, systematic process*
Health literacy principles
Science
Theory
*Coleman, et al, 2011

6. What did we create?
1. Program Assessment Tool©
2. Curriculum Assessment Tool©
3. Materials Assessment Tool©

7. Curriculum Assessment Tool
 Curriculum Assessment Tool -- provides
a standardized set of criteria to evaluate
existing educational curricula and to use in
creating new educational curricula.
 Uses a four-point scale
(Effective, Good, Fair, and Ineffective)
 Based on recommendations of Extension
educators who provided a rationale for
curricula review*
*Coleman et al, (2011)

10. Educational Materials Tool
 Educational Materials Assessment Tool --
provides a standardized set of criteria to judge
educational materials used in programs and in
curricula.
 Can be used both for critiquing existing
educational materials and creating new
materials.
 Based on Doak et al (2007) which provides a
standardized way to analyze materials for
learners with varying levels of prose, document
and quantitative literacy.

13. How did we create them? The role
of theory…
 Both rubrics grounded in theoretical principles of adult
learning, health literacy, health behavior
 WHY?
 Application of theory to educational interventions increases
the likelihood that intended outcomes will be achieved;
 Theories provide the rationale for how the intervention is
strategically structured and delivered; and
 Theories offer the basis for assessment of the program’s
degree of success in achieving intended outcomes.
Braun, et al (forthcoming)

14. How did we create them? The role
of theory…
 Depends on situation, targeted population,
understanding of behavior change and determination
of outcomes that are strategic, measurable,
achievable, relevant and timely*
 Level of intervention—individual, family or other
group, community or policy—will guide the
appropriate programming theories.
 For the Health Insurance Literacy Initiative
curriculum, we used individual-level theories: social
cognitive theory, stages of change or readiness, theory
of planned behavior, communications, adult or youth
development, empowerment, and evaluation and action
research
*Doran, 1981

15. How did we create them? The role
of theory…
 Key theoretical concepts can be combined into a
model that frames the design and measurement of
educational programming impact.
 The theories we selected directed us to:
 Involve the targeted population
 Identify the level of confidence, competence and
sense of control before and after interventions.
 Create messages and deliver via appropriate channels
 Design learning experiences to increase critical
thinking and reflection.
 Assure that evaluation of need, process and outcome
is effectively conducted and reported.

16. How can you use these tools?
 Language of the rubrics could be substituted with
appropriate vocabulary for the curricula content
under review.
 May need revise based on the level of intervention
(family, individual, community) and the intent or goals
of the program.
 May need other theories to guide the development of
programs and assessment tools if not focused on
change-making.

17. Let’s practice…
1. Find your copy of the
Educational Materials
Assessment Tool and the
Scoring Sheet
2. Review the provided document
3. Complete the scoring sheet as it
relates to the material

18. Questions for discussion:
 What are your thoughts about the
material after using the rubric
guidelines to assess it?
 In what ways did you find the
rubric useful in guiding your
evaluation of the material?

19. Coming Soon

20. References
 Braun, B., McCoy, T., & Finkbeiner, N. (In press).
Extension Education Theoretical Guide with Criterion-
Referenced Assessment Tools. College Park:
University of Maryland Extension
 Coleman, G., Byrd-Bredbenner, C., Baker, S., &
Bowen, E. (2011). Best practices for extension
curricula review. Journal of Extension, 49, Article
2T0T1. Available at:
http://www.joe.org/joe/2011april/tt1.php
 Doak, C.C., Doak, L.G., & Root, J.H. (2007). Teaching
patients with low literacy skills. (2nd ed.). Philadelphia:
J.B. Lippincott Company.
 Doran, G. T. (1981). There’s a S.M.A.R.T. way to write
management’s goals and objectives. Management
Review, 70, 35-36.

21. Acknowledgement
This presentation and the rubrics were co-created
with Nicole Finkbeiner, MS, Graduate
Research Assistant, University of Maryland
Department of Family Sciences.