1. ANALYSIS OF OUTREACH
IN THE
RADON AND INDOOR AIR PROGRAMS
Region 10
Policy, Planning and EvaJuation Branch
May, 1993

2. Background
Region 10's Radon and Indoor Air Program managers requested the Policy,
Planning and Evaluation (PP&E) Branch to conduct an analysis of the
effectiveness of their outreach activities. The main goal was for PP&E to help
the Radon and Indoor Air Program (RIAP) determine how they can optimize efforts
to direct their messages to specific target audiences. Analysis success is the
development of recommendations and information that allows RIAP to measure and
demonstrate that they are directing their limited resources in ways that
maximize the reduction of risk.
The RIAP wanted recommendations that would aid efforts to target their outreach
toward geographic areas, audiences, and pollutants with the greatest risk.
Overall Program guidance is established by HQ, but the Region has a wide range
of methods available for disseminating and implementing the guidance. RIAP has
limited resources.
The PP&E Branch was to develop recommendations that can be implemented without
foreseeable increases in resources, .and recommendations for RIAS time
allocation based on quantitative measures of program impact. RIAS requested
preliminary results available in time to incorporate into their planning for
FY94.
2

3. •••
TABLE OF CONTENTS
I. Executive Summary
Page
4
II. Radon and Indoor Air Pollution 5
III. Bennett Model 8
IV. Explanation of Bennett's Hierarchical Steps 10
V. Bennet Model Applied in Washington 13
VI. Outreach in Other Government Agencies 17
VII. Outreach in Region 10 18
VIII. Recommendations 19
Appendices:
Appendix 1 22
Appendix 2 24
Appendix 3 26

4. EXECUTIVE SUMMARY
This analysis is intended to aid Region 10' s Radon and Indoor Air Program
managers in their efforts to determine the effectiveness of their outreach
activities. The Radon and Indoor Air Program (RIAP) strives to optimize their
efforts to successfully target audiences and deliver messages to protect the public.
The analysis presents recommendations and information that will help RIAP to measure
and demonstrate that they are directing their limited resources in ways that
maximize the reduction of risk and public protect.
The dichotomy between outreach that is "shotgunned" and that which is more
precisely targeted need not necessarily be resolved entirely in favor of one or the
other. This analysis will not offer recommendations that support or reject either
approach. A program that includes both approaches to outreach is much more
realistic. These recommendations are intended to suggest different approaches that
are not exclusive of one another. The follow"ing recommendations are offered to
address the analysis goals (all recommendations are discussed in more detail in the
Recommendations section):
1. Maintain a base program of general public information.
2. Build feedback/measurement systems into our grants and contracts.
3. Begin conducting our own feedback/measurement activities.
4. Develop strategic plans with our grantees and/or the states.
5. Develop an outreach capability within the Office of External Affairs.
Recommendations
Recommendation 1: Maintain a base program of general outreach designed to increase
contacts with the general public and with key participants.
Radon/Indoor Air Quality programs are in their infancy. A general dissemination
of information will help to develop an infrastructure of informed people who can
participate in a slow but steady shift of opinions and behavior. This approach is
consistent with the patterns followed by our grantees and contractors -- mainly the
states -- who have the power to reach more people than we do. Too narrow of an
approach reduces the number of potential allies who may at some point, through their
purchasing habits, votes, word-of-mouth, or other behavior, contribute to better
indoor air quality.
Recommendation 2: Build feedback/measurement systems into EPA grants and contracts.
Disbursed funds are a primary lever to achieve better indoor air quality. This
funding is an excellent opportunity for determining how well our outreach is working
within the public sector.
This Recommendation advocates building limited feedback and measurement into
grants. Instead of devoting the entire grant to general public information, the
grantee can craft a proj ect of limited scope that lends itself to design and
monitoring.
This approach requires gai.m.ng t;heapproval and cooperation of the grantee, which
might be a challenge given their limited resources and preference for general public
4

5. information rather than targeted campaigns. EPA might gain cooperation by making
its staff more available.
Recommendation 3:
activities.
EPA Region 10 can conduct its own feedback/ measurement
To date there have been many public presentations, training sessions,
informational inquiries, distribution of materials, and various other people who
might be characterized as members of a radon/indoor air "infrastructure." We could
survey them to find out how useful our information was and how it led to changes in
behavior. More specifically, we could poll real estate brokers to develop a data
base of tested homes. We could also query radon testing and mitigation contractors
to determine their involvement in reducing radon risk.
Recommendation 4: EPA Region 10 can developing strategic plans with its grantees
and/or the states.
While federal, state, and local government may still be struggling through the
start-up phase of radon and indoor air management, we can look ahead. This is a
uniquely federal role, given our four-state perspective, the nation-wide effort at
the national level, and the leverage we possess as grantor. Programs and projects
that lend themselves to feedback and measurement may stretch over longer periods
than the typical one-year grant. We need to persuade others to work with us to
develop a sense of what priorities are needed within our programs and how we can
achieve measurable results.
This proposal is not limited to the states; there are non-profit organizations,
such as the Washington Lung Association, who are potential collaborators. We can
consider forming alliances, in the form of cooperative agreements, with major
collaborators. This encourages a common solution, as well as allows leveraged
resources. Cooperative methods, driven by strategic planning, allow for feedback
and measurement of everyones' effectiveness.
Recommendation 5:
Affairs.
Develop an outreach capability within the Office of External
Neither Radon nor Indoor Air have had much involvement with the Office of
External Affairs. There has been some press interest, but the interest fluctuates.
Press is, by definition, concerned with the news value of an occurrence. Outreach
is a continuing, rather than an episodic, activity. Outreach can be augmented by
professional capabilities (writing, graphics, education) which the Office of
External Affairs could provide as needed by Radon, Indoor Air, and other programs.
The Policy, Planning and Evaluation Branch believes this analysis can be useful
to the Radon and Indoor Air Programs. Further analysis and/or support is available
should the program desire it.
5

6. II. Radon and Indoor Air: National Program Design
The Indoor Air program was initiated by the Radon Gas and Indoor Air Quality Act
of 1986, Title IV of the Superfund Amendment and Reauthorization Act (SARA). The
legislation established a research program to identify, characterize, and monitor
the sources and levels of indoor air pollution, investigate the human health risks,
and to develop control technologies or other mitigation measures. The statute also
provided for disseminating information to the public about the research activities.
EPA has expanded on the statutory mandate through its FY 93 program guidance,
which emphasizes outreach activities at the Headquarters level: research,
preparation of guidance documents for physicians and architects, documents for
targeted segments of the public, an information clearinghouse, written materials on
assessment and mitigation with private sector organizations, and training. The
regional office role includes public information, technical assistance, training,
and state and local program development.
The Indoor Radon Abatement Act, passed in 1988, relies on similar measures. It
provides for the publication of a citizen's guide that outlines the health risk from
radon, mitigation costs and techniques, and testing methods. EPA is further
directed to develop model construction standards and provide technical assistance
to state radon programs, including training, an information clearinghouse,
demonstration programs, and publication of public information materials concerning
radon health risks and methods of radon mitigation.
A key aspect of the radon program, at least from the perspective of EPA regional
offices, is state grant funding which is authorized to states for building and
geographic surveys, and purchase of measurement and analytic equipment. Other
potential activities include development of public information and educational
materials, developing data systems, mitigation demonstration programs, and toll-free
radon hotlines to provide information and technical assistance.
EPA Headquarters guidance for FY 93 identifies five elements to be addressed by
the radon program:
1. Problem Assessment (identify areas with high radon levels)
2. Mitigation and Prevention (methods to reduce or prevent radon contamination)
3. Capability Development (improve state, tribal, private sector, and general
public in dealing with radon)
4. Public Information (provide timely information about radon risk and methods
of risk reduction)
5. Federal Coordination (work with other federal agencies and buildings)
While public information is presented as a separate prong of this five-part
strategy, the last three of these elements arguably constitute outreach activities.
The guidance also iden-tifies regional office roles: overseeing regional training
centers, promoting new building codes; overseeing state grants; promoting real
estate guides and addressing radon problems during real estate transactions;
representing EPA to state, local, tribal, and private sector; providing technical
support; and expanding public outreach activities. This last role includes
developing public education materials (public service announcements, brochures,
posters), providing EPA publications to states and tribes, and promoting model
building codes to state and local governments. Again, public outreach is an
enumerated function, but all of the listed regional office roles probably constitute
outreach.
6

7. r
It is important at the outset to settle on a definition of outreach. An
expansive definition may be best since almost every activity in the radon and indoor
air programs constitutes outreach. The term encompasses more than just public
information activities (e.g. production and distribution of written information,
public service announcements, press releases). Training, grants, promotion of
changes in building codes or real estate agent licensing requirements -- all are
opportunities to educate and persuade, which I believe is what we intend when we
employ the word "outreach".
The national program structures for radon and indoor air invite some conclusions:
• They are voluntary programs that rely on research and outreach rather than
regulatory strictures.
• They reflect a national (some would say Beltway) approach
that emphasizes broad, sweeping measures.
to policy-making
• They provide no guidance on the precise nature of the outreach to be conducted
by the regional offices. They offer none of the precision and narrow focus that
is conducive to gathering useful feedback and measuring program effectiveness.
On the other hand, this leaves the regional offices with enormous latitude in
their outreach practices.
• Both statutes and programs are first-time, start-up efforts that forsake the
regulatory approach in favor of behavior change by promoting "a better way"
through education and persuasion.
EPA has further fleshed out its program goals through a radon program analysis
to determine how well the program has been working. Draft Radon Program Review,
Office of Policy, Planning and Evaluation, Environmental Protection Agency, May
1992. The study found that EPA's radon program had raised public awareness, made
progress in testing and mitigating, supported state programs, and collected data on
radon distribution and health effects. However, the study found that only limited
testing and mitigation had occurred; about six percent of homes had been tested and
even fewer of those needing mitigation had been mitigated. The analysis further
found that, although early program success compared favorably with similar efforts
in anti-smoking, drunk driving, seat belt and crime safety, EPA wanted to have a
greater impact on radon health risks. The study made four major recommendations:
1. EPA should focus on the greatest risks first (high radon potential areas and
smoking).
2. EPA should emphasize the long-term strategies of promoting radon-resistant new
construction and testing/mitigation in connection with real estate transfers.
3. EPA should develop a coordinated research plan.
4. EPA should move beyond a nation-wide public information program and develop
a new strategy for public information, one that emphasizes high radon potential
areas and that focuses on building support for construction of radon-resistant
new homes and for testing/mitigation of existing homes when they are sold.
The fourth recommendation, regarding public information, directly concerns us
here. The study made several findings about public information, including:
7

8. • Most radon testing (70-90%) occurred in the context of real estate
transactions.
• State and local public information and incentive campaigns have yielded
limited responses. A public information effort in Washington, D.C., resulted in
a one-time testing rate of about four percent. Local direct mail and free or
discount testing kit distribution have resulted in one-time testing rates of 15-
30%.
• Government has a primary role of informing the public, particularly because
radon is imperceptible without a public information effort.
• A foundation of public awareness is necessary to reduce radon risk.
• EPA's public information campaign has raised public awareness; 80% of the
population has heard of radon and 65-70% understand that it poses a health risk.
• Nevertheless, public information alone is not sufficient to make a significant
impact on radon risk reduction; EPA should be moving beyond a nation-wide public
information strategy.
• The localized nature of radon limits the federal role; states
but federal energies can complement through research, pilot and
projects, and support for state activities. Other health
organizations, at all levels, also have roles.
have the lead
demonstration
and consumer
The Headquarters analysis sends mixed signals about outreach. On the one hand,
the report finds a need to move beyond a broad-brush public information program and
identifies a need to narrow the focus of outreach and recommends where it can be
done -- real estate transactions. This attempt to isolate an aspect of the overall
radon problem seems consistent with the Bennett Model (see below), which works best
when the results sought are narrowly drawn.
8

9. III. The Bennett Model
A blueprint for planning and measuring outreach activities was developed by
Claude Bennett of the Department of Agriculture in 1974. Bennett was trying to
analyze the effectiveness of the numerous outreach programs administered through
Cooperative Extension. Washington State government has found the model useful; both
the Department of Wildlife and the Puget Sound Water Quality Authority have applied
it to their programs.
In creating the model, Bennett wanted to provide a useful tool for program
managers to know whether their program was succeeding. Bennett's model assumes that
program evaluation (what we are calling measurement of the effectiveness of
outreach) is built into every program from its outset, instead of juryrigging a
measurement system sometime down the road to provide hindsight.
The model consists of seven hierarchical steps. Each step represents a category
of outcomes which, in turn, allows us to develop criteria for evaluating the
outcomes. The seven steps are arranged along a continuum, with "ends" at the higher
end and "means" at the lower end. Each step is a prerequisite to the next, in terms
of program design. However, actual program planning is conducted in reverse -- you
establish your desired results and work backward toward inputs. The essence of the
model is that you cannot design a program unless you know in advance, with
specificity, what your program is supposed to accomplish.
Bennett's Model is a simple one; it organizes obvious program components in a
logical progression in a way that encourages a program planner or manager to see the
connections and integrate them in a result-oriented fashion.
[Bennett's Model is illustrated on the next page.]
9
I

10. Bennett's Model*
Results
(which are reached by)
Behavior Changes
(which result from)
KASA (Knowledge - Attitude - Skill - Awareness
(which are influenced by)
Reactions
(which are a function of)
Participants
(which happens due to)
Activities
(which are offered as a result of)
Inputs
* Designing an Effective Communication Program: A Blueprint for
Success; University of Michigan: School of Natural Resources and
Environment; Prepared for U.S. Environmental Protection Agency, Region
5; September 1992; page 60. '
10

11. IV. Explanation of Bennett's Hierarchical Steps
Results: This step asks the essential question, how do you know when
you have done your job? Someone must identify a problem
and then enunciate a goal so that we know when the problem
is solved. An important question to be resolved is, who
determines the goal (s) -- policymakers or those charged
with conduct of the outreach effort?
There are three kinds of results:
1. Environmental -- a change in the condition of the environment
(e.g., a renewed fish run; a measurable improvement in opacity; a ban
on lead in gasoline).
2. Protective mechanism in place -- compliance with a measure that is
deemed to yield some benefit (e.g., obtaining a Clean Water Act §404 permit;
implementation of the National Contingency Plan for Superfund sites).
3. Institutional capacity -- this is a pseudo-result, where a regulation
is approved or a program is fully funded (e.g. listing a Superfund site on
the National Priority List)
When program design is initiated at the "Results" stage, the goal can be
expressed in terms of qualitative or quantitative measures. Much of EPA's
outreach appears to have considered this step in only the most general way --
if we disseminate information widely then people will smarten up and we will
have a better environment. A glance through the Bennett model reveals how
this method has skipped many steps that provide the logical connections
between program and results. EPA outreach resembles the reproductive methods
of many plant and animal species: scatter enough seeds and some will take
root. While this may work for Douglas fir, it may not reflect the most
careful husbandry of public resources, nor does it protect public health in
the most comprehensive or efficient manner.
Behavior Change:
This involves identification of some durable, persistent institutional or
personal behavior change (e.g., stop smoking; buy recycled materials).
Program designers should ask, what are the cultural enforcers or inhibitors
of behaviors? Behavior change can be difficult to measure it is
impossible to know how many have quit smoking. Then we need to look for
analogue or indicator behavior -- cigarette sales, expansion of anti-smoking
legislation. Follow-up may be necessary to determine behavior changes, such
as telephone surveys.
KASA (Knowledge - Attitude - Skill - Awareness)
This is the information component of program design, where we consider what
people think and know and what they should be thinking and knowing in order
to alter behavior and achieve program results. Ideally, we conduct a pre-
test of KASA to determine what people knew prior to program implementation.
A post-test after the program should then tell us the effect of the program.
These are very resource and time-intensive measures.
11

12. What is essential at this stage of program design is that a clear, readable,
technically accurate body of information be compiled. It may have to
encompass a range of values in order to give everyone targeted what they need
to change behavior.
Often, program design fails to go past this step in determining program
effectiveness Behavior Change and Results are not considered.
Unfortunately, without feedback about what the outreach has accomplished, we
are just "bean-counting" (e.g., we published X brochures and we reached a
potential audience of Y through radio public service announcements).
Reactions
For outreach efforts to work, we have to create a climate of learning:
timing, surroundings, consider the feelings of participants, appropriate
behavior by "teachers." Everything must be structured to achieve "buy-in" by
participants.
Participants
The outreach audience must be targeted with prec1s1on. The larger and more
diverse the audience, the less likely a message can be tailored that achieves
the behavior changes that will produce the result wanted. If the audience is
the general public, we will likely be reduced to a general "shotgunning" of
information accompanied by a leap of faith that appears to be based on the
theory that we know the facts and we understand the health risks, and,
therefore, if we publicize what we know, everyone will know this and act
wisely to protect their health. There is no evidence that broadbrush
outreach accomplishes this.
For an identified audience, we want to know what "pushes their buttons", how
they can best be approached. Perhaps affinity groups (labor unions,
churches, professional societies) can be used as conduits to reach an
audience.
Activities
This step details the tools or levers available to augment a program:
workshops, conferences, technical assistance, toll-free hotlines, brochures,
speeches, testimony, etc. Activities encompasses more than the capabilities
of our own agency; can we leverage additional performance through grants,
contracts, cooperative agreements?
These activities can be arranged into a timeline to organize the entire
program and envision what will occur.
Enputis ; The resources to be applied to the program include FTE, extramural funding,
volunteers, etc. In a program planning sense, this step of the hierarchy
requires budgeting for results. Having designed the program, committing
to outcomes, and identifying the resources will eventually lead to those
results.
12

13. Bennett recognized the limits of quantitative measurement -- the difficulties of
measurement and the substantial resource implications. In response, he also
developed a qualitative measurement system for program evaluation. He called this
Reflective Appraisal of Programs (RAP); program participants "reflect" upon the
changes effected by a program. In essence, participants are asked to respond to a
standard questionnaire keyed to any of the four "higher end" outcomes of Bennett's
hierarchy -- Reactions; K.A.S.A. Change; Behavior Change; and Results.
The RAP approach encourages use of the findings from the interviews to draw
conclusions about and evaluate results of the program. These conclusions then lead
to recommendations about how decision-makers can apply the findings.
The Bennett Model offers a systematic, logical, easily understood framework for
program design. Its emphasis on backwards planning, with project results serving
as the key to project design, encourages a focus on project feedback or measurement.
13

14. V. The Bennett Model Applied in Washington
Two agencies of Washington state government have applied the Bennett Model in
different ways. The Department of Wildlife conducted a retrospective analysis of
its Project WILD, an environmental education effort that had been in operation for
eight years. The goal of Project WILD was to
"...assist learners of any age in developing awareness, knowledge, skills, and
.commitment to result in informed decisions, responsible behavior, and
constructive actions concerning wildlife and the environment upon which all
life depends." Evaluation of Project WILD: The State of Washington 1984 -
1992, by Margaret Tudor, Washington Department of Wildlife, 1992.
When measured against the Bennett Model, Project WILD was found to have many of
the preliminary prerequisites for program success. For example:
Inputs: 1 or 2 FTE throughout program life; 1,000 elementary guides, 300
secondary guides, 300 aquatic guides produced annually.
Activities: 376 introductory and advanced workshops held; 2 conferences held
(total attendance 800).
Participation: 6,300 teachers (15.5% of total) participated.
Beyond this point in Bennett's Hierarchy, program evaluation of Project W+LD tended
to be more anecdotal:
Reactions: Survey found response to workshops favorable; teachers reported
curricular resources useful.
K.A.S.A.: Teachers report change in students' attitude, but students',
knowledge, attitude and skill change not measured.
Behavior change: No strong evidence directly linking Project WILD with
wildlife protection and habitat enhancement; anecdotal evidence that advanced
workshops using Project WILD as resource has led to habitat enhancement by
.teachers.
Results: Project WILD -- no effects measured; anecdotal evidence that habitat
enhancement education program linked to habitat grants protected or enhanced
3,000 acres.
The analysis concluded that Project WILD needed further development of its goals,
and that the goals had to be expressed in terms of measurable outcomes. Targets to
meet the goals needed to be set at the outset of the program.
14

15. The Puget Sound Water Quality Authority (PSWQA), originally established by the
Legislature as an independent agency and now a part of the Department of Ecology,
has conducted an ambitious outreach program for several years. Through the Public
Involvement and Education (PIE) Fund, PSWQA has made an experimental effort to
involve the public in managing Puget Sound through a number of creative outreach
activities.
PSWQA conducted a study of 42 PIE Fund projects, measured in terms of Bennett's
hierarchy. Publicly Financed Voluntary Initiatives for Public Involvement and
Education as a Natural Resource Management Tool, An Evaluation 1989-1991 Washington
State Public Involvement and Education Projects, Funded by the Puget Sound Water
Quality Authority, July 1991. The study include the following findings:
• All projects (100%) fulfilled Levels 1,2 and 3 (Inputs; Activities;
Participation) in Bennett's model.
• 90% of the projects were fulfilled through Level 4 (Reactions).
• At Level 5 (K.A.S.A. changes) 71% were fulfilled.
• 33% reached Level 6 (Behavior/Practice changes).
• 14% reached Level 7 (Results) -- some form of water quality protection or
improvement.
The PSWQA and Department of Wildlife studies cry out for additional analysis --
so here goes. The lower end, early stage steps in Bennett's hierarchy are more
readily quantifiable. At that point in program planning we are measuring resource
levels, numbers of people, workshops, conferences, and hotline calls answered.
As we move up the scale away from "means" toward "ends", our methods of
measurement are few -- we can conduct surveys, send out questionnaires. These
measurements tend to give us qualitative, subjective responses which usually tell
us that we are having some effect; but we have no way of knowing whether we have
orchestrated an efficient use of public resources. Low return rates on these
measurements leave us questioning whether we have a representative sample or whether
we really even know what effect our program has had. It is even more difficult to
determine what behavior changes we have effected; notice how the results in the
PSWQA study drop from 71% (K.A.S.A. changes) to 33% for Behavior changes. The
Project WILD study, while not as quantitatively precise, shows a dramatic change in
the nature of the outcomes at the higher end of Bennett's scale.
In both studies, the evaluations of Results suggest that the projects ran out of
gas; Project WILD had inadequate anecdotal information, and PSWQA found that only
14% of the projects fulfilled their purpose. Some possible explanations for this
dropoff in measurable performance are:
• Program design has not included more precise measurement.
• Environmental and natural resource outreach proj ects are inherently
difficult to measure if they focus on the effects on the natural
environment. Puget Sound or the ambient atmosphere are too big and there
are too many potential intervening factors that may mask our projects,
however well-intentioned, -designed, and -implemented they may be.
15

16. • Environmental outreach may be difficult to measure during the life of a
proj ect. Proj ects tend to reflect funding cycles and budget
appropriations. Perhaps measurable effects are longer term phenomena that
do not readily fall within these timeframes.
• Perhaps it is too early to tell. Many government outreach programs are
still in their infancy. Managers and staff may not have been trained in
the importance of outreach or in its intricacies. Development of outreach
skills may elevate the outcome of future projects.Measurable results may
not be available until some time in the future.
• Measurement at the higher end of Bennett's hierarchy requires substantial
resources. It may involve significant socialfbehavioral science field
work. At the KASA level this involves the art of learning assessment.
Project evaluation of this magnitude is normally beyond the means (funding
and time) of public agencies.
• The lack of quantifiable data at the higher end of the model may simply
reflect a bias of environmental/natural resource agencies: our natural
bent is to look for things to measure. When we cannot find them, we
regard whatever feedback we get as suspect or inadequate. Maybe we need
to reassess our own attitudes and accept qualitative evaluations.
• There may be a form of entropy in human affairs that applies the longer
an undertaking lasts, or the greater the span of control among the
individuals involved. In government agencies, resource cuts, personnel
turnover, and shifting priorities may take the bloom off the rose before
a proj ect is completed. Outreach depends on influencing people and
activities that are not within the control of a government agency and
which have other demands on their time, enthusiasm, and loyalty. Perhaps
there is simply a natural enthusiasm at the outset of a project that is
inevitably drained <}:waybefore completion. Perhaps measuring program
success requires more, or different energies and attitudes, than setting
a project in motion does.
16

17. • Program design, on the macro level, may inhibit the accomplishment of
goals. For example, one of the Radon program's primary tools is grant
funding to the states. However, federal law requires a 50% state match
for federal funding. Scarce radon funding on the state level has resulted
in less than 100% of available federal grant money being awarded in Region
10. If we are not using all of the admittedly low level of program
resources, it should not surprise us that our outcomes fall short.
PSWQA has taken steps to overcome the difficulties of measuring outreach
effectiveness. The Authority has designed a questionnaire for use by PIE Fund grant
recipients to analyze the effectiveness of a grant. (See Appendix 1) PSWQA uses
a pre-proj ect and post-proj ect questionnaire for grant recipients. The two
questionnaires differ essentially only in verb tense. They are closely keyed to the
Bennett Model. The questionnaires are designed to have grantees thinking in terms
of results at the inception of the project, and to reinforce the importance of
thorough program design as the project is completed.
17

18. VI. Outreach in Other Government Agencies
Most of the EPA regional offices, along with a few other government agencies,
were informally surveyed by telephone to determine what outreach techniques they
were using arid their success in measuring outreach effectiveness. (See Appendix 2)
As far as outreach methods, nothing novel emerged -- they were using anticipated
tools to predictable audiences. Almost without exception they claimed no success
in measuring outreach. The Great Lakes Program (EPA, Region 5) relies on a
carefully designed survey of visitors to a ship that sails among Great Lakes ports
to ascertain visitors' knowledge of water quality issues. They are surveying a
captive audience for information purposes only; no behavior change is sought. There
is little applicability of that effort to radon and indoor air. A few of the other
offices surveyed count basic information such as the number of people reached by
various outreach methods, but none has found a way to measure changes in behavior.
Those offices polled expressed an interest in this study as a means of finding
better ways to chart their effectiveness.
18

19. VII. Outreach in Region 10
Region 10 performs an impressive amount of outreach in the Indoor Air and Radon
programs. Staff are fielding public inquiries, developing contacts in public and
private sectors, organizing and presenting training, meeting with key groups,
circulating written materials, appearing at horne shows, etc. There have been
attempts to target audiences, such as appearances before real estate brokers'
associations regarding radon, which is consistent with the recommendation in the
Headquarters Radon Program Review that real estate transactions be a primary focus
of outreach. Staff are maintaining mailing lists and identifying attendees at
various events. There has been no lack of energy in trying to promote both
programs.
What is less apparent is whether the programs are using a measured, result-
oriented approach in their outreach -- the Bennett Model or something similar.
Their efforts seem more concerned with reaching as many different audiences in as
many different ways as possible, rather than targeting an audience with a particular
result in mind and then trying to measure the result. For example, the Indoor Air
Coordinator participated in the presentation of the training program "Orientation
to Indoor Air Quality" in Spokane. If this particular outreach event is analyzed
in light of the Bennett Model, we discover the following:
Inputs: The Region's lone program coordinator (1 FTE) disseminating
program knowledge.
Activities: Three-day interactive training program.
Participants:
Federal, state, and local employees, representing a variety of agencies,
non-profit organizations, and some private sector representatives, many
from Washington and Idaho.
The implicit purpose of this training is to impart information, raise
consciousness, develop networks of information and allies, and generally bring key
individuals to a higher level of understanding of indoor air issues. However, this
effort did not include any attempt to determine the Reactions of the participants,
the Knowledge - Ability - Skills - Attitudes of the participants before and after
the training, their Behavior Changes, and the program Results of the training. An
attempt to measure the effects of the training would have included these steps. (No
criticism of any individual is intended by this analysis; it is offered solely to
suggest program improvement, or at least to illustrate a different approach.)
The traditional broadbrush approach to outreach and the more focused approach
offered by the Bennett Model highlight the fundamental tensions in performing
outreach, which is the main point of the Recommendations offered here.
19

20. VIII. RECOMMENDATIONS
Recommendation 1: Maintain a base program of general outreach designed to
increase contacts with the general public and with key participants.
Government has an educational function. Increasing the number of educated
potential allies who may at some point, through their purchasing habits, votes, or
other behavior, contribute to better indoor air quality, is an educational role.
Radon/Indoor Air Quality programs are in their infancy. A general dissemination
of information to develop an infrastructure of informed people who can participate
in a slow but steady shift of opinions and behavior. This approach is consistent
with the patterns followed by our grantees and contractors -- mainly the states --
who have the power to reach more people than we do.
We can explore ways to make this general information function more efficient than
it is. Perhaps we can make morejbetter use of Senior Executive Employees (SEE) by
giving them primary responsibility for responding to general telephone inquiries.
This frees the program coordinators to perform new duties suggested within these
Recommendations.
An alternative that could produce a similar result would be to use the Public
Information Center in the Office of External Affairs for answering public inquiries.
This would require giving PIC staff more training and more authority in answering
questions, instead of only furnishing the public with publications.
There are other difficulties: these are contractor personnel and we would have
to decide whether contractors should be performing this function. On the other
hand, this might be a preliminary step in creating a more omnibus public information
function in OEA, which would broaden that Office's involvement in Regional affairs
and free program staff for other, more narrowly focused outreach activities.
Recommendation 2:
contracts.
Build feedback/measurement systems into EPA grants and
Disbursed funds are a primary lever to achieve better indoor air quality. This
funding is an excellent opportunity for determining how well our outreach is working
within the public sector.
Appendix 3 is a summary of the Washington State Department of Health's ac tivi ties
under an~PA grant in the Radon Program for a one-year period. Staff made a host
of presentations, attended meetings, drafted and updated fact sheets, operated a
toll-free information line, issued press releases, and published articles.
20
I

21. But very little of their activity shows any results. True, they did have some
involvement in the passage of legislation that requires local building officials to
provide radon testing devices in new home construction. That is a result of sorts;
of course, a more meaningful result would describe the mitigation that occurred as
a consequence of testing.
Other activities suggest tantalizing possibilities of feedback/measurement. They
conducted a testing survey and mailing survey (of what? to what end?). They backed
away from a plan to test and mitigate economically disadvantaged homes when the
contractor failed to produce matching funds. Instead a general public information
effort was introduced.
This Recommendation advocates building limited feedback/measurement into grants.
Instead of devoting the entire grant to general public information, the grantee can
craft a project of limited scope that lends itself to design and monitoring as
envisioned by the Bennett Model. For example, isolation of real estate transactions
within a given area, involving local realtors, offering test kits or services at a
reduced rate, conducting public information campaigns, and then measuring tests at
the time of real estate transactions can create feedback.
This approach requires gaining the approval and cooperation of the grantee, which
might be a challenge given their limited resources and preference for general public
information rather than targeted campaigns. EPA might gain cooperation by making
its staff more available (with the time available through the use of Office of
External Affairs (OEA) personnel to field- inquiries -- see #l above).
Recommendation 3:
activities.
EPA Region 10 can conduct its own feedback/ measurement
To date there have been enough public presentations, training sessions,
informational inquiries, distribution of materials, and various other people who
might be characterized as members of a radon/indoor air "infrastructure." We could
survey them to find out how useful our information was and how it led to changes in
behavior. More specifically, we could poll real estate brokers to develop a data
base of tested homes. We could also query radon testing and mitigation contractors
to determine their involvement in reducing radon risk. Again, the available staff
time through public inquiries fielded by individuals other than program staff could
allow focus on this activity.
21

22. Recommendation 4: EPA Region 10 can developing strategic plans with its grantees
and/or the states.
While federal, state, and local government may still be struggling through the
start-up phase of radon and indoor air management, we can look ahead. This is a
uniquely federal role, given our four-state perspective, the nation-wide effort at
the national level, and the leverage we possess as grantor. Programs and projects
that lend themselves to feedback and measurement may stretch over longer periods
than the typical one-year grant. If we are serious about gathering this
information, we need to persuade others to work with us to develop a sense of what
priorities are needed within our programs and how we can achieve measurable results.
This proposal is not limited to the states; there are non-profit organizations,
such as the Washington Lung Association, who are potential co~aborators. We can
consider forming alliances, in the form of cooperative agreements, with major
collaborators. This encourages a common solution, as well as allows leveraged
resources. Cooperative methods, driven by strategic planning, allow for feedback
and measurement of everyones' effectiveness.
Some of our grantees have continued with us for two or more consecutive years.
The Alaska Health Project (a non-profit) has been a grantee for FY 91-93, as has the
Washington State Department of Health. The Oregon Department of Environmental
Quality has been a grantee for FY 92-93. These patterns suggest some commitment,
assuming continued grant funding. There are potential pitfalls -- grantees are
taking the risk that EPA funding will continue, they will be chosen as grant
recipients, and program priorities will remain constant enough for multi-year
undertakings seen through to conclusions.
Recommendation 5:
External Affairs.
Develop an outreach capability within the Office of
Neither Radon nor Indoor Air have had much involvement with the Office of
External Affairs. There has been some press interest, but not recently. Press is,
by definition, concerned with the news value of an occurrence. Outreach is a
continuing, rather than an episodic, activity. Outreach can be augmented by
professional capabilities (writing, graphics, education) and could be located in OEA
for use as needed by Radon, Indoor Air, and other programs.
22

23. Appendix 1
Puget Sound Water Quality Authority
PIE Fund Questionnaire
Pertinent portions of the post-project questionnaire are listed below.
Section 1: Improving Water Quality in Puget Sound
1.1 What condition has your project helped correct?
1.2 Has that condition been corrected?
1.3 How do you know the condition has been corrected?
1.4 How has your project contributed to correcting the condition?
Section 2: Individual and Group Behavior
2.1 What behaviors (or lack of) on the part of individuals or groups contributed to the
condition described in Section 1?
2.2 What changes in behavior do you hope have occurred as a result of your project?
2.3 How do you know if the behavior of project participants has changed as a result of
your project?
2.4 Do you plan any followup activities to estimate long-term changes in behavior of
individuals or groups involved in your project?
Section 3: Knowledge, Attitudes, Skills, or Awareness
3.1 What information did project participants receive as a result of your project?
3.2 How was that information transmitted to them?
3.3 How did you find out if they received, understood, or used the knowledge, attitudes,
skills or awareness that they received during your project?
Section 4: Reactions Expected from Project Participants
4.1 Under what situations did learning occur in your project?
4.2 How do you hope project participants felt about your project or Puget Sound after
being involved in your program?
4.3 How do you know how they feel?
4.4 Were project teachers (or others presenting intormatlon, if applicable) evaluated by
participants or audience? If so, how?
Section 5: Project Participants
5.1 Who was the audience for your project (be as specific as possible)?
5.2 What was the most effective way you found to publicize your project?
5.3 How many direct participants (workshop attendees, viewers, listeners, publication
readers, site visitors, etc.) were involved in your project?
5.4 How many indirect participants were involved in your project?
5.5 What products (publications, etc.) did your project produce? How many?
5.6 Who participated in technical review or program oversight?
Section 6: Project Activities
6.1 What activities or events were involved in your projects?
6.2 What did participants do at activities or events?
23

24. 6.3 When did these activities occur?
6.4 What externalfactors affected the success of these activities? (bad weather, no
competing events, etc., cooperation of another party or organization)?
Section 7: Resources Put to Work
7.1 What was the total budget for your project?
7.2 How much money was raised from sources other than PSWQA?
7.3 What other kinds of contributions were made to the project?
7.4 How many paid employees worked on the project? For what percentage of their workload?
7.5 How many volunteers worked on the project?
Section 8: Problems and Challenges
What problems or challenges were faced in completing the project and were they dealt with?
Section 9: Unanticipated Results
9.1 Describe unanticipated successes (other than planned goals and objectives) that
you achieved in completing this project.
9.2 To what do you attribute these?
24

25. r~'
I
Appendix 2
Survey of Outreach in Other Government Agencies
Agency Name Staff Outreach Methods Employed Effectiveness How Measured
Army Corps of Eng'rs Press News releases Good questionl Counting beans (# of news Count # of papers that pick up
Ofc. releases, speeches, or public inquiries) not news releases or inquiries that
accurate result from news release; hiring
P.R. firm to research O/R very
expensive
Seattle City Light Com Newsletter; public presentations Trade shows and community events allo~ Customer Service Survey shows
munit them to meet their customers how many customers they inform;
y use signup sheets at presentations
Relati to count # reached; don't know
ons (5 how to measure whether public is
staff) "getting" their Info
WA Dept of Ecology Need to plan measurement
upfront, In program design; easy
to measure # of contacts, harder
to measure changes in people's
behavior; try to measure changes
in attitudes and awareness?
EPA Reg. 1 Presentations for groups; booths at Most effective when they target specific Count # of cases; use signup
fairs; speeches; written groups sheets at presentations - send
publications; show test kit questionnaires; track calls -
correlate w/groups they have
given presentation; clip articles
from newspapers that use their
info
EPA Reg. 2 Respond to calls; offer tech asst; Want to use Am. Lung Assoc. to followup IAQ: program too new to tell
send written materials; bldg. AQ course to avoid "survey" issue;
problem w/O/R is that it leads to more calls
if effective, requiring more staff; no single
form of O/R works -- need combo
25

26. EPA Reg. 3 Presentations to comm'ity grps; Setting up tables at malls not very effective; Don't measure effectiveness, but
developed school curriculum - haven't identified audience have good data re
students take home test kits and testing/mitigation - see direct
must return; hot line for public correlation between D/R and
inquiries; full color maps for testing/mitig
distrib.; use questionnaire for
speakers: who/how many attended
presentations; lots of radio PSA's;
work w/Better Breather's Clubs;
health fairs at hospitals and firms;
work w/states, realtors; combine
w /EED program in work w/schools;
coord w/ALA re Rn Awareness Mo.
EPA Reg. 4 Provide speakers; presentations for Most effective when targeting specific grps. Have no way of measuring
gps.
EPA Reg. 5 Use sophisticated survey Survey measures what people
techniques re Great Lakes project learned from ship - nothing
specific re Rn/IAQ
EPA Reg. 6 1 FTE Presentations to schools Unknown Don't measure
EPA Reg. 7 Brochures, talks in schools, PSA's D/R aimed at children seems effective, based Don't have resources to measure
on experience re smoking and seatbelts;
PSA's waste money - played in dead time
EPA Reg. 8 _ IAQ program just trng state staff; Too early to determine effects of
states do most of Rn D/R; fed. IAQ program; WY has most
effort is hodge-podge: try dif. aggressive Rn D/R and best
things results - even so, awareness level
is 85% while testing level is 16%
"',.
26
"

27. Appendix 3
WSDOH State Indoor Radon Grant Program
Final Report: July 1, 1991 - June 30, 1992
• Report period represent Year 2 of EPA grant
• Staffing during grant period was 7 FTE, plus 1 temporary
• Participation on committees and symposiums
• WSDOH convened meeting of state-wide Radon Task Force
• Attended bimonthly meeting of SBCC tech. adv. gp. during 4-mo. period; attended Energy
Code Comm. and 2 SBCC meetings
• Advise key gps. re Rn
• Clover Park S.D. Innovative Sch. Proj.: numerous meetings w/staff, 2 publ. mtgs.;
presentation to ESD
• Advised and made 1 presentation to CRPDC committees
• 7 presentations to various conferences and committees
• 3 presentations to state gov't bodies; aided in passage of legislation that requires local bldg.
officials to supply testing device in new home construe.
• Continued to augment and update database
• Eliminated 1 fact sheet, updated 4 others
• Updated telephone info service; 1,059 calls received during yr. - residential info was primary request,
then basic Rn info and where to purchase test kits
• Lending library has various materials, which were used by public and contractors
• PSA's
• Issued 3 press releases
• Issued press releases and articles re Clover Park S.D. project
• Published 4 articles in various newsletters
• Rn promotion by Haggen Food Stores -- may have contributed to increase in calls (54 to
133) in Feb.
• Various program staff attended at least 10 training sessions/conferences
• Conducted testing survey and mail survey
• Produced and distributed brochure for builders to distribute to new home occupants, and
collaborated in brochure for local building officials to distribute to new home owners
• Began, but never completed, project to send letters to new home owners re radar detector to be
provided by builders (project superseded by new law requiring local officials to provide detector)
• Gave contract money to NE WA Rural Resources Dev't Assoc. to conduct publ. info. program to test,
diagnose, and mitigate in at least 100 econ. disadv'd homes in 3 counties (project never completed
due to lack of matching funds from contractor); project converted to publ. info. effort through various
media and presentations '
• CSRAT Project compiled materials, offered training re Rn in schools (113 total attendees plus 38
more in Clover Park S.D.)
• 40 buildings te~ted in Clover Park
27
I