Clergy ResponseNet presentation - Deirdre Hainsworth

Clergy ResponseNet:
a social service referral tool
Deirdre King Hainsworth
HCI598 Capstone, Iowa State University
November 20, 2015

Overview of the project
§  Clergy ResponseNet is a web-based tool that
enables religious leaders in Southwestern
Pennsylvania to search for social service referral
information, and enables them to help build and
improve the site by adding information to a
database of social service listings.

Clergy ResponseNet
1. 

Users
and
their
Needs

2. 

Design
and
Prototype

3.  User
Tes8ng
and
Findings

4. 

Future
Work

Users
and
their
needs
Two sources used for determining user needs:
§  Analysis of congregations’ role in social service
referral and provision
§  Interviews with clergy involved in making referrals

aid-seekers congregations
approach congregations because:
•  historic tradition and mission
•  locally accessible, visible
•  more personal, seen as “safer”

approach congregations because:
•  historic tradition and mission
•  locally accessible, visible
•  more personal, seen as “safer”
response can be limited by:
•  lack of internal resources
•  lack of expertise
(Ammerman, 2005)

community
agencies
Clergy serve as “gatekeepers”:
•  agencies offer more specialized
expertise, broader resources
•  many agencies rely on clergy for
verification of aid-seekers’ needs
(VanderWaal, Hernandez, and Sandman, 2012).

Interviews
with
clergy

§  interviewed 3 persons in active congregational ministry, to
learn how they currently handled aid requests
§  all three were within broader potential user group:
§  mid-20’s – early 70’s
§  college educated with some graduate training in ministry
§  experience with at least some online tools (search, e-mail)
§  all reported being frequently asked for help in connecting
to social services from members and non-members; needed
to make referrals

Current
problems
in
making
referrals

users had problems related to information access
§  users had very differently sized personal networks to
tap into, but none were without information gaps
§  very little information shared across congregations
§  clergy newer to the community had special difficulties;
information was lost during clergy transitions
§  could not find information to share on the process of
obtaining services (what to expect, benefits)

Limits
of
current
informa<on
sources

existing sources were unreliable, inaccessible
§  described Google as “too much” and “sketchy”; in the
past, had used it and sent people to places where they
were ineligible for services
§  wanted more contact information that enabled collegial
connection, verification with local agency staff and
other clergy
§  currently using paper brochures, e-mails, text, business
cards, Facebook – often inaccessible when needed

community
agencies
working hypothesis: Information failures are an important
part of how the aid process can break down. An online
system could address current problems in information
access and flow, and could improve congregations’
capacity to make effective referrals and carry out their
“gatekeeping” role.

System requirements 1- 4:
To address current information failures, the system
should display the following information types:
1.  listings of services available for aid-seekers in
particular communities
2.  listing details that are specific and relevant to
responding to the aid-seeker’s needs
3.  information that allows users to connect to
colleagues in the network of social service providers
4.  information that informs users about the process of
making referrals and public program resources

implementation: 4-step search process
step 1: choose zip code
codes were outside of all testers’ areas, to prevent information “bleed”

step 2: choose category

step 3: choose type of service

step 4: obtain listing information
prototype included 70 mock “listings”

implementation: multiple information types

System requirement 5
Users should be able to trust the relevance and reliability
of information they obtain from the system.
design choice:
§  The system will limit access to provider searches and submission
of information to registered users, with more general resources
available to all site visitors.
Dividing these information types will help to prevent over-
saturation of detail in the listings, allowing users to evaluate and
test it. Controlling access to some, but not all, site features has
been shown to support user trust and create a funnel into more
active, registered involvement with websites (Loranger, 2015).

implementation: login for listings access

Users should be able to trust the relevance and reliability
of information they obtain from the system.
design choice:
§  User-contributed information will require approval from the
system manager before being added to the database.
A curated process of information management has worked in
other professional contexts to build trust during the
development phase of online networks for collaborative
knowledge (Hsu, Chang and Yen, 2011; Sharratt and Usoro,
2003; Ye, Feng and Choi, 2015).

Users should be able to obtain referral information in their
work locations regardless of user age or technical skill.
design choice:
§  The system will include affordances for easier navigation,
including large, clear buttons, persistent header and footer
features, on-screen feedback and sequential menus.
These reduce cognitive load and can assist in recovery from
errors and user confidence in site navigation (Nielsen, 2005;
Shneiderman and Plaisant, 2005). Sequential menus tend to be
simpler for users to navigate for simpler tasks (Hochheiser and
Shneiderman, 2000).

Users should be able to obtain referral information in
their work locations, regardless of user age or technical
skill.
design choice:
§  The system will initially be made available as a
webpage, rather than an app.
While a website design can include responsive elements
that aid mobile use, design as an app would potentially
exclude older, poorer, and less technically skilled users,
who are far less likely to use smartphones (Pew Research
Center, 2015)

Why digital rather than paper?
§  Digital testing does better at uncovering the severity of
usability problems (Walker, Takayama and Landay,
2002).
§  Tests with higher-fidelity (digital) prototypes tend to be
more user-driven rather than facilitator-driven (Rudd,
Stern, and Isensee, 1996).
§  A digital prototype enables more direct and pointed
review of the appropriateness of size, color, and
persistence of site elements ( Sefelin, Tscheligi and Giller,
2003). These will be important elements in assessing
usability for older users.

Clergy ResponseNet
1. 

Users
and
their
Needs

2. 

Design
Priori8es

3. 

User
Tes8ng
and
Findings

4. 

Future
Work

Testing participant profile
§  four participants, all members of the potential user
group (clergy currently involved in ministry in
southwestern Pennsylvania)
§  One user was male, and three were female.
§  Each represented a different age bracket (one
each for 30 – 35, 40 – 45, 50 – 55, and 55 – 60).
§  Testing conducted in one user’s office, one user’s
home, and two in quiet rooms of coffee shops during
breaks in users’ work days.

Data collection methods
Used digital and paper tools and direct observation:
§  I chose to do in-person testing in order to limit users on-
screen activities to the use of the prototype itself.
§  Short paper questionnaires were used after each task,
as well as a paper version of the System Usability
Scale instrument (Brooke, 1996) at the end of testing.
§  Using a prototype hosted on my laptop also allowed
me to control for variations in Wi-Fi and equipment
availability, as well as to observe. Data also captured
through Camtasia and a “think-aloud” process.

Testing process
§  Each testing session began with a brief demographic
interview, followed by the reading of an introductory
script and review and signing of a consent form.
§  Users completed four “task scenarios,” related to
potential events they might encounter in ministry.
These required users to perform specific tasks using
the site, responding to an aid-seeking situation in a
particular zipcode, but left them some room in how to
complete them.

Testing process
§  Testing concluded with the SUS and open-ended
interview questions :
§  What did they like or dislike about the site?
§  What suggestions would they make for
improvement?
§  What would they suggest concerning who should
have access to site information and listings
submission?

Task scenario 1
§  Task scenario 1 asked users to log in to the site and
perform a simple search to assist a person seeking
help with utility bills.
§  This task set evaluated:
§  clarity of welcome and login screen
§  user navigation of menus for a single category search
§  the perceived clarity and relevance of listing information
on site.

Task scenario 1- findings
Post-task survey questions u01 u02 u03 u04 average
Task scenario 1, Question 1 (ease
of locating information needed) 7 7 7 7 7
Task scenario 1, Question 2
(usefulness of information found) 7 6 7 7 6.75
(helpful for effective response to
aid seeker) 7 6 7 6 6.5
total on task scenario 1 21 19 21 20 20.25
average for task scenario 1 7.0 6.3 7.0 6.7 6.75
1 = negative, 7 = positive
All users completed all tasks successfully

Task scenario 2
§  Task scenario 2 asked users to complete a more
complex search. They were asked to locate
information to assist a congregation member seeking
meals, social support, and long-term planning help for
an aging parent.
§  user navigation of menus for a multiple-service search
§  support for open-ended searches within a zipcode
§  perceived clarity and relevance of listing information

Task scenario 2 - findings
Task scenario 2, Question 1 (ease of
locating information needed) 6 7 7 6 6.5
(usefulness of information found) 5 7 7 7 6.5
Task scenario 2, Question 3 (helpful
for effective response to aid seeker) 7 7 7 7 7
total on task scenario 2 18 21 21 20 20

Task scenario 3
§  Task scenario 3 asked users to locate area food
pantries, review listings to identify currently unmet
needs for food assistance within a zipcode area, and
locate information about state-level food programs.
§  support for a multiple-purpose search
§  the accessibility and clarity of both information types:
database resources and referral guides.

locations to be searched for task 3 completion:

3: Observations and user comments
§  Users quickly completed the initial stages in this task
set, but were confused when asked to locate contact
information for a state food program.
§  Two users completed the task by locating a link to a
referral guide in the left menu; two could not.
§  One user commented that they had “stopped
seeing” the left menu once they started doing
searches in the center of the screen.

Task scenario 3, Question 1 (ease of
locating information needed) 5 6 6 7 6
Task scenario 3, Question 2 (usefulness
of information found) 7 7 6 7 6.75
Task scenario 3, Question 3 (clarity of
layout) 6 6 7 7 6.5
Two users were unable to complete the final sub-task.

Task scenario 4
§  Task scenario 4 asked users to enter listing data for a new
entry in the site database, and confirm that the system
had received it.
§  user engagement with the site (through the level of detail
they provided in their “listing”)
§  user recognition of system responses (a submission
confirmation screen)
§  users’ understanding of the curation and verification process
for submitted data.

(ease of locating information
needed) 7 7 7 7 7
(clarity of layout) 7 4 7 7 6.25
average for task 4 7.0 5.5 7.0 7.0 6.63

Post-tasks: System Usability Scale
SUS question u01 u02 u03 u04 average
1. I think that I would like to use this system
frequently. 4 4 4 3 3.75
2. I found the system unnecessarily complex. 4 4 4 4 4
3. I thought the system was easy to use. 3 4 4 4 3.75
4. I think that I would need the support of a
technical person to be able to use this system. 3 4 4 4 3.75
5. I found the various functions in this system
were well integrated. 4 3 4 3 3.5
6. I thought there was too much inconsistency in
this system. 4 3 4 4 3.75
7. I would imagine that most people would learn
to use this system very quickly. 4 4 4 4 4
8. I found the system very cumbersome to use. 4 4 4 4 4
9. I felt very confident using the system. 4 4 4 3 3.75
10. I needed to learn a lot of things before I
could get going with this system. 4 4 4 4 4
Sum of questions 1- 10, scaled (multiplied by 2.5) 95 95 100 92.5 95.63
(values converted to uniform range of 1 (negative) to 4 (positive)

Post-tasks: interview findings
All four users commented positively and
enthusiastically about the site:
§  user 1 described it as a “needed resource”
§  user 2 wrote “This is very helpful and needed in our
communities! Individual churches I know keep listings of
these social services but I do not know of any other
online database for collaborative use.” User 4 made a
similar comment.
§  While completing the SUS instrument, user 3 read the
first question aloud, and said “No, I would love to use
this website.”

Necessary usability revisions
§  very serious: the current segregation of listing
information and “process” information is unworkable
solutions:
§  integrate state and federal resources into the
main listings information as a standard menu
item for each category
§  move all information types out from behind
“firewall”, reserving log-in for info submission
§  review and test site more generally to check
that persistent features don’t “disappear” for
users

§  significant: users experienced some confusion in the
use of terms and categories in the listings
solutions
§  add an introduction to the site that includes
sample searches, definitions, and an outline of
the taxonomy used
§  add information windows, activated on hover,
that can provide definitions or guidance

§  minor:
§  users experienced some problems in flow and
movement between site elements
solution: make sure that all links are active and
correct, especially within user feedback areas
§  some users were confused by the different layout of
listings screens and listing submission screens
solution: make sure these match in labeling and layout

Conclusions: ease of use
§  Based on the scores for each task, the SUS scores,
and users’ verbal feedback, users found the system
generally easy to navigate.
§  The serious exception to this was user access to non-
listing information in the site, which failed for 2
users; this will need to be re-designed.

Conclusions: perceived relevance
§  Users gave the site high scores in its support for
helping them do their work effectively (post-task
surveys).
§  In task comments and post-testing interviews, users
commented positively on the usefulness and breadth
of the types of information provided.

Conclusions: perceived trustworthiness
§  During the testing process, users mentioned that the
site “looks official,” and “makes me feel safe.”
§  Three users commented positively on the availability
of contact information that would allow them to
make a connection to agencies and verify services.
§  All four users commented positively on the curation
and verification process for listings information.

Conclusions: user engagement
§  Users were focused and actively engaged in the task
scenarios during the testing process, each choosing
slightly different paths to task completion based on
their own professional understanding of how to
respond to the aid-seeker.
§  SUS scores showed high ratings for user confidence
§  In task 4, three of the four users added more
information than required, thinking aloud about what
would be helpful information for others to have and
making up information to add to what was provided
for text entry.

Conclusions: user engagement
§  While two users initially commented on the
aesthetics of the site (one user in the early 30’s
saying it seemed to be for an older user, another
saying “it makes me feel safe, but it doesn’t make
me feel happy”), the aesthetics of the interface did
not seem to reduce users engagement in tasks or the
value they placed on the site.
§  Users’ comments showed they viewed the site as
relevant and useful to their role as clergy.

Conclusions: stakeholder “fit”
In closing interviews, all four users were open to
broadening the user group to include nonprofit /
agency staff as well as clergy, but not to the general
public. They viewed it as a potential tool within a
broader role:
User 3: “When people are coming for help, they want
to know that someone is there for them, someone is
helping them. Some people can’t use a computer,
some people would be overwhelmed …you don’t
minister by just handing someone a computer.”

Clergy ResponseNet
1. 

Users
and
their
Needs

2. 

Design
and
Prototype

3. 

User
Tes8ng
and
Findings

4. 

Future
Work

Broader issues for implementation
§  information architecture: more research and user
testing will be needed to develop a taxonomy of
listing categories that is useful and intuitive without
creating overwhelming menu structures

§  technical development: this will be relatively simple;
the site can be inexpensively supported using a
basic PHP/MySQL database backend. I have
started work on this.

§  interface refinement: more research and user testing
will be needed to find a balance between universal
design principles and a site design that is
sufficiently engaging for younger users. This will
also need to support responsive design for mobile
use.

§  database population and maintenance: this will be the
major, ongoing issue to resolve.
§  This system will be unmanageable as a single-person effort,
and it will fail if the information quality degrades.
§  My current plan is to form a nonprofit that will enable me to
use the initial build-out of this project as a basis for training
others in community research and technical skills.
§  After initial launch, my hope is that the user community could
play an active role in maintenance.

References
A full list of references for this project is available at:
http://www.datatoserve.com/capref.html

Clergy ResponseNet presentation - Deirdre Hainsworth

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (6)

Similaire à Clergy ResponseNet presentation - Deirdre Hainsworth

Similaire à Clergy ResponseNet presentation - Deirdre Hainsworth (20)

Dernier

Dernier (20)

Clergy ResponseNet presentation - Deirdre Hainsworth