Civic User Testing Group (CUTGroup) presentation for Chicago's Civic Design Camp. Smart Chicago's CUTGroup is a community of over 800 Chicago residents who test a civic websites and apps. This presentation goes over how we run the CUTGroup and the three main components of CUTGroup as UX design, digital skills and community engagement.

1. Sonja Marziano
Smart Chicago Collaborative
April 25, 2015
www.smartchicagocollaborative.org
CIVIC USER TESTING GROUP
(CUTGROUP)

2. SMART CHICAGO COLLABORATIVE
A civic organization devoted to improving
lives in Chicago through technology.
¡ We work on increasing access to the Internet,
improving skills for using the Internet, and
develop meaningful products from data.
¡ CUTGroup is a central program because it cuts
across these three areas

3. PREMISE OF CUTGROUP
The CUTGroup is a set of regular Chicago
residents who get paid to test civic apps
¡ Over 800 Chicago residents signed up, from
all over the city – all 50 wards, all 77
community areas
¡ $5 gift card sign-up
¡ $20 gift card per test

4. 15 tests to date that cover topics such as
health, education, social services,
transportation, permits, and more
¡ All different styles: focus groups, one-on-one
proctored tests, remote
¡ We publish all of our methods, questions and
results online
§ We also wrote a book!
§ www.cutgroupbook.org
CUTGROUP TESTS

5. THREE MAIN COMPONENTS
¡ UX Testing
¡ Digital Skills
¡ Community Engagement

6. UX TESTING
¡ Developers who work with us get
specific, actionable feedback
¡ Both quantitative and qualitative
information
¡ Stray from UX test design principles in a
key way: by requiring the developer to
participate in the test

7. DIGITAL SKILLS
¡ Each test consists of a mix of 10 – 20
people with varying digital skills
¡ Introduce to new websites and
technology
¡ Developers learn how to conduct UX tests
to improve their sites

8. COMMUNITY ENGAGEMENT
¡ Diverse groups of testers from different
neighborhoods
¡ Conversations about technology between
residents and developers

9. Developer Meeting
¡ Who are the current
users?
¡ Who do we want to
target for this CUTGroup
test?
§ What type of screening
questions should we ask?
¡ What type of test?
¡ What do we want to
ask?
¡ What does the
developer want to get
out of this CUTGroup
test?
Example: mRelief
¡ Current users: Southside
¡ Target: Enrolled in social
services benefits, eligible
for benefits, different
devices
¡ Type: One-on-one
proctoring
¡ Questions: Language, ease
of use, text feature, pop-
overs
¡ Wanted to know: language
accessibility, user
experience, importance of
additional info about docs
or process
PRE-TEST

10. ¡ Find location: public computer centers and
libraries
¡ Mailchimp email campaigns to
§ Invite testers
§ Learn more through screening questions
§ Check availability
PRE-TEST

11. ¡ We use Wufoo forms to
capture responses
¡ Also, used as a script
for test questions
¡ Tools:
§ Screencast-o-matic
§ Video cameras
§ Laptops & iPads
§ Other devices
§ Gift cards
§ Candy
DURING TEST

12. Types of Questions
¡ Current behaviors &
experiences
¡ Devices
¡ Homepage review
¡ First item clicked on
¡ Likert scale
¡ Guided/task-related
questions
¡ Quantitative and
qualitative
Example: CPS.edu
¡ Do you search online
about the schools your
children attend?
¡ Review the homepage.
What are 3 things you
notice?
¡ Search for your child’s
school. How did you
search? What did you
learn?
¡ In what ways would you
improve the CPS website?
DURING TEST

13. ¡ Analysis
§ In-depth analysis of each question
§ Publish raw test data
¡ Publish videos and photos from test session
¡ Blog post with overview
¡ Follow-up with Github issues
AFTER THE TEST

14. Example: Foodborne Chicago
¡ How do users feel about
responding to a random
tweet?
¡ Does the form require
too much info?
¡ Would users feel better
with more CDPH or 311
visibility?
¡ Why might residents on
the South side not
respond as often as
residents from other
neighborhoods?
Answers
¡ Tweets should have 3
elements:
compassion, question,
and an official element
¡ Form was not too long
¡ Wanted more visibility
of connection with the
City of Chicago
¡ Twitter is used as a
private communications
network
AFTER THE TEST

15. UPCOMING CUTGROUP CHANGES
¡ Outreach to Cook County
¡ Text messaging system to recruit and notify new
testers
¡ Proctoring program
¡ More post-test work, assisting with implementing
changes

16. ¡ Extensive how to run
CUTGroup
¡ Oakland & Chattanooga
have started CUTGroup
in their cities
www.cutgroupbook.org
CUTGROUP BOOK

17. ¡ Sign-up here:
www.cutgroup.org
¡ Are you a developer? Sign-up here:
www.cutgroup.org/developers.html
BE A TESTER!

18. Sonja Marziano
smarziano@cct.org
@ssmarziano
Smart Chicago
@SmartChicago
www.smartchicagocollaborative.org
www.cutgroup.org
THANK YOU!!!!!!