https://hwpi.harvard.edu/digital-content-connect As communication professionals at Harvard, we are always crafting our messaging with a certain audience in mind. But how do we learn more about our audience, and how can we be sure that our messaging meets their specific needs, especially in this ever-expanding digital landscape? Our Spring Digital Academy event focused on user research. We heard about several methods that digital and content strategists use to learn more about our audiences, from persona development and survey research to usability studies and decision analysis. Speakers included top digital practitioners from Harvard and outside experts with experience in digital media and marketing, tech and public policy, and higher education strategy and consulting.

4. THE DEVELOPMENT OF
HAA EVENT PERSONAS
Emily Dufresne
HAA Engagement Marketing
Helen Lewis
AAD Business Intelligence

5. WHAT WE NEEDED TO KNOW– STRATEGIST POV
•What does our overall alumni engagement look like? Where are we succeeding and
what do we need to work on? Who are NOT engaging? What does this look like
across the different types of events we host?
•What are our demos by: school, age, gender, grad year, eVENTS region, past event
attendance?
•Are we engaging a U-wide audience? What does engagement look like across
schools?
•How many first time event attendees are coming to HAA events?
•We’re rolling out a new event campaign for 2019. How can we better understand our
2018 alumni engagement metrics so we can identify areas of
improvement/strategy/segmentation for 2019?
•Are we seeing any interesting trends that need more research or testing?
5

6. PROCESS CONSIDERATIONS– ANALYST POV
•What are the questions to answer?
•How will this analysis be used?
•What data sources are needed? What demographic data is available/needed?
•What dimensions and metrics answer those questions specifically? (avoid the weeds)
•How can you frame the analysis and leverage metrics?
•Can events and demographics be grouped to build comparative sets (and a larger sample size)?
•How to best visualize the data?
•What other teams can strategically and tactically benefit from this analysis/findings?
•How to best communicate the findings with others? (be creative)
6

7. WHERE WE STARTED
7

8. THIS IS ULTIMATELY WHAT WE NEEDED.
NOW, HOW TO BEST COMMUNICATE IT AND THE TRENDS?
8

9. LEGEND
I = Invitee pool
A = Attendee pool
D = The Delta (difference between pools)
AL = Alumnus/Alumna
AM = HAA Associate Member
Correlation
≠
Causation
KEEP IN MIND
9

10. SELECTED EVENTS & AL/AM ATTENDANCE
A Subset of one year of HAA programming that earlier included HAA Reunions and multi-day events (excludes Club events)
PRESIDENTAL EVENTS (n=5; 999 attended)
An Evening with President Drew Faust in Cincinnati (145 AL/AM attended on 1/10/18);
An Evening with President Drew Faust in Los Angeles (271 AL/AM attended on 2/15/18);
An Evening with President Drew Faust in Naples (193 AL/AM attended on 1/24/18);
An Evening with President Drew Faust in Philadelphia (74 AL/AM attended on 3/9/18); and
Your Harvard San Francisco (316 AL/AM attended on 2/14/18).
HAA WEBINARS | live broadcast (n=3; 3,631 attended)
Acing Your Interview (1,010 AL/AM attended live broadcast);
Getting onto Boards: Recruiting and Preparing Women for the Boardroom (1,088 AL/AM attended live broadcast); and
Successful Career Negotiations for Women (1,533 AL/AM attended live broadcast).
NETWORKING EVENTS (n=1; 5,803 attended)
Global Networking Night (5,803 AL/AM attended in January 2018)
10

11. PRESIDENTIAL EVENTS AL/AM ONLY
= Attended % > Invited %
*Based on data from Your Harvard San
Francisco and Evenings with President
Faust in Cincinnati, Los Angeles, Naples,
and Philadelphia
11

12. PERSONA | FAUST PRESIDENTIAL EVENTS [AL/AM ONLY]*
MORE LIKELY TO ATTEND
Alumni record type I=88% A=94% D= +6%
Alums from COL; KSG; GSD I=50% A=61% D= +11%
Females age 45-54 I=21% A=25% D= +4%
Females age 55-64 I=14% A=24% D= +10%
Females age 65 and older I=14% A=19% D= +5%
Males age 55-64 I=16% A=20% D= +4%
Males age 65 and older I=35% A=41% D= +6%
Donors I=65% A=83% D= +18%
A.H. Community Site Users I=21% A=52% D= +31%
*Based on data from Your Harvard San Francisco and Evenings with President Faust in Cincinnati, Los Angeles, Naples, and Philadelphia 12

13. PERSONA | FAUST PRESIDENTIAL EVENTS [AL/AM ONLY]*
LESS LIKELY TO ATTEND
Associate Member record type I=12% A= 7% D= -3%
Alums from HBS; HLS; EXT I=37% A=30% D= -7%
Females 35 and under I=21% A=14% D= -7%
Females age 35-44 I=25% A=13% D= -12%
Males age 35-44 I=15% A=12% D= -3%
*Based on data from Your Harvard San Francisco and Evenings with President Faust in Cincinnati, Los Angeles, Naples, and Philadelphia 13

14. 1ST
TIME EVENT ATTENDERS
CONSIDERATIONS
1) AL/AM only
2) Based on data in Advance
3) Excludes any club event attendance
What was the percentage of AL/AM
attendees who were at their first event?
14

15. PERSONA DATA|PRESIDENT BACOW EVENTS [AL/AM ONLY]*
*Based on data from Welcoming President Bacow Receptions in
New York, Detroit, San Diego, and Miami.
15

16. GLOBAL NETWORKING NIGHT AL/AM ONLY
= Attended % > Invited %
*Based on data from the January 2018
Global Networking Night
More recently, with the January 2019
GNN event 40% of the invited pool &
45% of attended pool were female ->
+5% delta.
16

17. PERSONA | GLOBAL NETWORKING NIGHT [AL/AM ONLY]*
MORE LIKELY TO ATTEND
“AL” record type I=87% A=90% D= +3%
Alums from COL; KSG; GSD; EXT I=44% A=54% D= +10%
Females age <35 I=24% A=41% D= +17%
Males age <35 I=13% A=25% D= +12%
Males aged 35-44 I=18% A=28% D= +10%
Males aged 45-54 I=17% A=20% D= +3%
International Primary Residence I=16% A=25% D= +9%
A.H. Community Site Users I=24% A=55% D= +31%
LESS LIKELY TO ATTEND
“AM” record type I=13% A=10% D= -3%
Alums from GSA; HBS; RAD; HLS I=46% A=37% D= -9%
Females 55-64 I=14% A=10% D= -4%
Females 65+ I=13% A= 4% D= -9%
Males 65+ I=30% A= 9% D = -21%
Donors I=62% A=59% D= -3%
*Based on data from the January 2018 Global Networking Night
17

18. HAA WEBINARS AL/AM ONLY
= Attended % > Invited %
*Based on data from 3 webinars, Acing
Your Interview, Getting onto Women
Boards, and Successful Career Negotiations
for Women
18

19. PERSONA | HAA WEBINARS [AL/AM ONLY]*
MORE LIKELY TO ATTEND
Alums from HBS; KSG; EXT I=25% A=37% D= +12%
Females age 45-54 I= 8% A=12% D= +4%
Males age 45-54 I=12% A=20% D= +8%
Donors I=59% A=63% D= +4%
A.H. Community Site Users I=31% A=52% D= +21%
*Based on data from 3 webinars, Acing Your Interview, Getting onto Boards: Recruiting and Preparing Women for the Boardroom,
Successful Career Negotiations for Women
LIKELY TO ATTEND
Females age <35 I=38% A=38% D= 0%
Females age 35-44 I=42% A=40% D= +2%
Females age 55-64 I= 6% A= 8% D= +2%
Males age 55-64 I=11% A=13% D= +2%
Intl Primary Residence I=11% A=13% D= +2%
19

20. PERSONA | HAA WEBINARS [AL/AM ONLY]*
LESS LIKELY TO ATTEND
Alums from GSA; HMS; COL I=50% A=43% D= -7%
Females age 65+ I= 6% A= 3% D= -3%
Males age 65+ I=19% A= 6% D= -13%
*Based on data from 3 webinars, Acing Your Interview, Getting onto Boards: Recruiting and Preparing Women for the Boardroom,
Successful Career Negotiations for Women 20

21. TAKEAWAYS
•Things that are measurable are not always the most useful. Use your
data to tell an actionable story.
•This process is iterative and will continue to be refined. Keep asking
questions.
•Your data is going to show you things you were not expecting to see. Be
prepared to adjust your strategy.
•Always consider the end game– what is the bigger question that you
are trying to answer?
21

23. Not for distribution / Desirability Lab /
2019
Introduction to Desirability Lab’s Design
Research Process
Digital Academy
March 20, 2019
Prepared by: Altringer, B. & Delaney, L.

24. Not for distribution / Desirability Lab /
2019
Designing for desirability
in products and services
(research-based)
(with and for diverse others)
(while fully engaged ourselves)

25. Not for distribution / Desirability Lab /
2019
Innovation = constant, uncertain
‘decision-making’
= What is desirable? To whom?
= Constant unknowns & interdependencies of work
= Disengagement risks (internal folks) (politics)
= Product failure risks (external folks) (poor product / market fit)
= All decisions prone to bias
= Structured methods can help (human centered design is one of many
design research methods)

26. Not for distribution / Desirability Lab /
2019
Design research: essential, ongoing, strategic
questioning to help manage inherently
high-uncertainty decision-making

27. Not for distribution / Desirability Lab /
2019
Who are motorcycles
designed best for?

28. Not for distribution / Desirability Lab /
2019

29. Not for distribution / Desirability Lab /
2019
Who is driving
growth?
- Women are the fastest
growing segment in
motorcycles.
- The number of riders doubled
from 2003-2014 (a period of
overall sales decline)
- And that is in a market where they
are still mostly ignored.

30. Not for distribution / Desirability Lab /
2019
Framing Your Initial Strategy
generative
I have an idea!
Or…
Let’s find a better
solution.
contextual
How do
participants
experience this
situation?
explanatory
Why is this
unexpected
thing
happening?
evaluative
Is this thing
working?
Does our
hypothesis hold
up?

31. Not for distribution / Desirability Lab /
2019
Starting with ideas = starting with bias
generative
I have an idea!
Or…
Let’s find a better
solution.
contextual
How do
participants
experience this
situation?
explanatory
Why is this
unexpected
thing
happening?
evaluative
Is this thing
working?
Does our
hypothesis hold
up?

32. Not for distribution / Desirability Lab /
2019
When to use each?
contextual
We’re new, not doing a
good job, or industry is
changing. Need to
check all assumptions.
Larger cultural distance
= more important
ETHNORAPHIC
explanatory
We’ve identified
an interesting
thing but don’t
understand it.
MIXED METHODS
evaluative
We have a hypothesis
about what explains
this but aren’t sure it
is right.
QUANTITATIVE
HYPOTHESIS
TESTING
generative
We have one or more
clear ideas, and criteria to
measure their
effectiveness at given
milestones.
ONGOING ANALYTICS

33. Not for distribution / Desirability Lab /
2019
contextual
We’re new, not doing a
good job, or industry is
changing. Need to
check all assumptions.
Larger cultural distance
= more important
ETHNORAPHIC
explanatory
We’ve identified
an interesting
thing but don’t
understand it.
MIXED METHODS
When you are not your user

34. Not for distribution / Desirability Lab /
2019
Ethnography for Idea Development
Starts with open eyes
Learning from people (while suspending judgement)
Learning from extreme users, regular users, evaluator users
Inferring from the intersection of what people say / do / use
Identifying inconsistencies in what people say and do and
observing workarounds = opportunities
Identifying criteria for an effective solution before
brainstorming solutions

35. Not for distribution / Desirability Lab /
2019
Finding growth opportunities in our blind
spots requires slowing down and
engaging in structured questioning.

36. Not for distribution / Desirability Lab /
2019
Cognitive
biases =
mental shortcuts

37. Not for distribution / Desirability Lab /
2019
Cognitive biases = mental shortcuts
https://medium.com/thinking-is-hard
we fill in gaps with…
Patterns
Generalities
Benefit of doubt
Easier problems
Current mindset
we save space by…
Editing memory
Generalizing
Keeping examples
we assume…
We’re right
We can do this
Nearest is best
Must finish
Easier is better
we only notice…
Change
Bizarreness
Repetition
Confirmation
Not enough
memory
Not enough
time
Too much
info
Not enough
meaning

38. Not for distribution / Desirability Lab /
2019
Examples of common decision-making biases

39. Not for distribution / Desirability Lab /
2019
What assumptions are you
making about the problem
area?
“That’s
too
disruptive
”
“That’s the
way we’ve
always
done it”
“I know the
client, this is
what they
want”
“This way is
easier”
“That idea
is too
crazy”
“That’s what I
thought, so
lets do that!”
“I like the
original
idea
more”
“This way is
easier”
Assumptions can be dangerous if left unchecked.
Assumptions can help frame of your challenge,
background research, and your fieldwork plan.
Biases can often result in accurate thinking, but
also make us prone to errors that can have
significant impacts on overall innovation.
“You like it?
Me to!”
https://www.boardofinnovation.com/blog/2017/08/02/16-cognitive-biases-that-kill-innovative-
thinking/

40. Not for distribution / Desirability Lab /
2019
Identify and analyze your
assumptions makes them a help
instead of hinder.

41. Not for distribution / Desirability Lab /
2019
Make the idea
real
1
Understand the
problem and
brief
Understand
people and their
needs
Figure out what it
means
Ethnographic Fieldwork Process
2 3
Create the idea
4 5

42. Not for distribution / Desirability Lab /
2019
Thanks!
Want to learn more?
www.desirabilitylab.com

43. Not for distribution / Desirability Lab /
2019
Questions?
Thanks!
Want to learn more?
www.desirabilitylab.com

45. WHERE DOES DATA COME FROM?
How Survey Researchers Define, Measure, and
Create Things Around Us
Chase H. Harrison
Program on Survey Research (IQSS)
Department of Government (FAS)
charrison@gov.harvard.edu

46. Data

47. Reporting
and
Publication
AnalysisData
Theories
and
Questions

48. Reporting
and
Publication
AnalysisData
Data
Collection
Theories
and
Questions

49. Reporting
and
Publication
AnalysisData
Survey
Data
Collection
Theories
and
Questions

50. Reporting
and
Publication
AnalysisData
Survey
Data
Collection
Survey
Design
Theories
and
Questions

51. What kind of data comes from surveys….

52. Population statistics…….
Population
Statistics
• Decennial Census
• American Community Survey
Crime • National Crime Victimization Survey
Substance Use • National Survey of Drug Use and Health
Health • National Health Interview Survey
Energy Use • Residential Energy Consumption Survey
Employment
• Current Population Survey
• American Employer Survey
Health Care
Costs
• Medical Expenditure Panel Surveys

53. Other things surveys are used for …
Election
Predictions
Time Use
Field
Experiments
Consumer
Confidence
Customer
Satisfaction
Attitudes
and Values
Public
Opinion
Program
Effectiveness
Social
Surveys
Consumer
Expenditures
TV Ratings
Market
Research

54. What is a survey?

55. Surveys
• Systematic method of data collection
• Usually use samples
• Designed to measure things
– Attitudes
– Behaviors
• Create statistics
– Descriptive
– Analytic
55

56. Survey research generally does two things well…….
Structured,
standardized data
collection
• Consistent equivalent
measures
• Easily quantified and
compared
Samples from
well-defined
frame
• Ability to understand
specific population of
inference
• Samples allow statistical
projection with measurable
or estimable precision

58. Theories About Things
Data
Measures
Concepts
Respondents
Sample
Population
Specification Error
Survey Errors
A Rough Overview of SurveysInternal
Validity
External
Validity
58
Data and Analysis

59. Survey Research:
Sampling, Coverage and Nonresponse
59

60. Inferential Population
Target Population
Sample Frame
Sample Records
Respondents
Random? Or not?

61. Survey Research:
Questionnaires and Questions
61

62. Theories and Surveys
• Concepts/Constructs (Theoretical Ideas)
• Measures (Questions or Scales)
• Responses (i.e. Data)
62

63. Constructs
• Underlying attribute we want to measure
– An element of information
– Sometimes composed of multiple discrete elements of information
• Can end up referring to either a single survey question or a set of questions
– Questionnaires or batteries attempt to cover all aspects of a topic or concept
– Individual items have to consistently measure the discrete thing they are
intended to measure
63

64. Questions
• Think of as Instruments
– Can end up referring to either a single survey question or a
set of questions
– Questionnaires or batteries attempt to cover all aspects of
a topic or concept
64

65. Another way of thinking about a
survey…..

66. How a survey methodologist thinks of
a survey question….
• Respondent receives some stimulus during the interaction
• Respondent reacts to stimulus
– This is data

67. What Respondents Do to Answer a
Question
• Comprehend Question
• Retrieve Information from Memory
• Summarize Information
• Report an Answer
67

68. Modes of Survey Data Collection

69. Traditional Survey Modes
Mode Sample Questionnaires
Face-to-Face ❖ Enumerated by interviewers
❖ Often sampled at household
level
❖ Interviewer administered
in-person
Telephone ❖ Randomly generated telephone
sample
❖ Landlines and mobile
❖ Lists of persons with telephone
numbers
❖ Interviewer administered
telephone survey
Mail ❖ Household addresses
❖ Named persons
❖ Self-administered paper
questionnaire
Internet ❖ Listed people with e-mail ❖ Self-administered web
questionnaire

70. New Technologies
• IVR
• SKYPE
• SMS
• WhatsApp ®
• Social networks
• Twitter
• Smartphones
• Apps

71. Large measurement differences
between….
Aural
Interview
Visual
Interview
Aural
Response
Visual
Response

72. New Electronic Modes
Disruptive Technologies
• The smartphone:
– GPS + Camera + Microphone + Accelerometer
• Wearable devices
• Biometric data / Facial recognition
• The return to “unobtrusive” measures
– Passive electronic measures (big data)
– Video surveillance
– Face recognition
– GPS monitoring
– Satellite imaging
• Administrative records (big data)

73. Dimensions of Approaches
• Level of Interviewer Involvement
• Degree of Human Contact
• Channel of Communication
• Locus of Control
• Degree of Privacy
• Use of Technology

75. What is survey data……
Measurement
Construct
Measure
Measured response
Edited response
Representation
Target population
Sample frame
Sample
Respondents
Postsurvey Adjustments
Survey
Datapoint

76. What is survey data……
Measurement
Construct
Measure
Measured response
Edited response
Representation
Target population
Sample frame
Sample
Respondents
Postsurvey Adjustments
Survey
Data
Specification
Error
Measurement
Error
Processing Error
Coverage Error
Sampling Error
Nonresponse
Error
Adjustment
Error

77. Thank you for your time…
Chase H. Harrison
charrison@gov.harvard.edu

79. The Email Newsletter Guide

81. Why newsletters matter?
• Control the experience
• Mobile first
• Build loyalty

83. Before launching a new newsletter or
refining an existing one, you need to
define:
•Value proposition
•Goals and measures of success
•Resource constraints

84. Here’s how Matt Kiser, founder of
What the Fuck Just Happened
Today?, defined each of these
three questions when thinking
about launching his newsletter…

85. What is the value proposition of your
email/s to your reader? What is the
problem this product solves? By flipping the
question into one that is reader-centered, you’re
forced to align your organization’s incentives (i.e.,
solving a problem readers have) with delivering
value to your reader.

86. What is your primary goal as a media
organization for your email initiative?
What are you trying achieve?
Examples of goals could include: drive reader
consumption habits, increase page views,
increase time on site, or learn more about your
audience. These goals should align with the
value proposition.

87. What are your resource limitations?
Determine what is feasible from a production
workflow perspective and your staffing and
resource levels. Newsletters can require a lot of
work to do well.

90. ● UX INTENT: Link digest for clicking out vs.
editorially driven for reading within product
● PRODUCTION TIME: Automated / RSS
feed vs. time-consuming original writing
● PERMANENCE: Pop-up, ephemeral
newsletter vs. no end in sight

92. Example 1:
The Single Subject News Study

93. Example: Hechinger Report’s Newsletter
1. Audience Surveys -- WHO is our audience and
what do they want? When do they read?
2. Internal Workflow Audit -- HOW does our team
make our newsletter?
3. External Peer Audit -- WHAT do our peers do?
Where is there a gap?

96. Example: The Trace & The Marshall Project
Problem: How can topically focused newsrooms
grow a group of “insider” readers, while also
expanding readership to the general public?

99. Example 2:
Pew
Research
Center
immigration
course

100. “We have a huge body of knowledge on a wide variety of
topics...The idea of boiling down what we know on a big,
complicated, complex topic like that into a series of
digestible, easily understandable emails seemed like it
would be something worth experimenting with.”
— Pew Senior Social Media Editor Andrea Caumont

102. QUESTIONS?
Emily Roseman
eroseman@hbs.edu
@emilyroseman1
Joseph Lichterman
joseph@lenfestinstitute.org
@ylichterman

104. Case Study
User Research &
library.harvard

105. 5 ways we put the
user at the center of
all we did

106. SEPTEMBER 2017
Started work with Velir
JANUARY 2018
Released pilot, Future.Library
CONTINUOUS
Listen, tinker, improve
SPRING 2017
User research with Fresh
Tilled Soil
JULY 2018
Released Library.Harvard.edu
Do user research first

107. Use personas, don’t just
create them
“It’s hard for people who don’t use the
library all of the time or with a ton of
confidence to find what they need. You
really have to ‘game’ the search to get it
to give you the types of results you’re
looking for.”

108. Test along the way

109. Build your team to mirror your
vendor’s team

110. Meet user expectations:
Best in show, not just best in
class

111. library.harvard.edu/
roadmap