A look at how we conducted open, exploratory user research to figure out how we could make significant jumps in conversion rates at CareerFoundry. Analytics and tests only get us so far, but to have a bigger impact we need to explore, and learn something totally new. Something that we don't know we don't know.

1. @DhyanaScarano
Exploratory Research
How to figure out what to test

2. @DhyanaScarano
• To have a drink and meet some cool people?
• To learn some new methods and tools?
• To learn about conversion optimisation and heuristic analysis?
• To hear how other people do explorative research?
• Something else…?
Why are you here?

3. @DhyanaScarano
Who am I?
• Senior UX Designer at CareerFoundry
• Co-organiser of Service Jam Berlin
• Co-organiser of UXcamp Europe
• Head mentor (product strategy),
Google Launchpad startup bootcamp
• Asks a looooot of questions
Dhyana Scarano (Dee)

4. @DhyanaScarano
• A bit of product strategy
• A bit of research methods & tools
• A look at a real exploratory research project underway at
CareerFoundry
• A little bit about how all that feeds into conversion
optimisation
What’s in this talk?

5. @DhyanaScarano
• A little bit of product strategy
• A little bit of research methods & tools
• A hands on exercise (yay!)
• Talk about all that feeds into conversion optimisation
How does CO think about research?

6. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
Define
Testing
ideas

7. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
DefineProblem
Testing
ideas

8. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
DefineProblem
The real
Problem
Testing
ideas

9. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
DefineProblem
The real
Problem
Solution
Testing
ideas

10. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
DefineProblem
The real
Problem
Solution
Defining the problem Coming up with a solution
Testing
ideas

11. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
DefineProblem
The real
Problem
Solution
Defining the problem Coming up with a solution
Figuring out what to test Testing options
Testing
ideas

12. @DhyanaScarano
And what about Design Thinking?
Research
Underst-
anding
Testing
ideas
DefineProblem
The real
Problem
Solution
Defining the problem Coming up with a solution
(what we’ll talk about now) (Conversion optimisation)
Figuring out what to test Testing options

13. @DhyanaScarano
So, it always starts with a problem…
ResearchProblem
Problem:
CareerFoundry needs to
increase it’s conversions

14. @DhyanaScarano
So, it always starts with a problem…
ResearchProblem
So how do we figure out what
makes customers click “enrol”?
Problem:
CareerFoundry needs to
increase it’s conversions

15. @DhyanaScarano
So, it always starts with a problem…
ResearchProblem
So how do we figure out what
makes customers click “enrol”?
Problem:
CareerFoundry needs to
increase it’s conversions
But wait. First, what does CareerFoundry do?

16. @DhyanaScarano
So, it always starts with a problem…

17. @DhyanaScarano
So, it always starts with a problem…
How do we figure
out what makes
people click this?

18. @DhyanaScarano
So, it always starts with a problem…
How do we figure
out what makes
people click this?
But then we need to find the “real problem”
We can test different
things based on what we
already know
We can also try and learn
something totally new
about our users that we
don’t know yet!

19. @DhyanaScarano
So, it always starts with a problem…
How do we figure
out what makes
people click this?
We can also try and learn
something totally new
about our users that we
don’t know yet!
Tweaking based on what
we already know only gets
us so far.
We can test different
things based on what we
already know
But then we need to find the “real problem”

20. @DhyanaScarano
So, it always starts with a problem…
How do we figure
out what makes
people click this?
We can also try and learn
something totally new
about our users that we
don’t know yet!
New insights can lead to
bigger jumps in
conversion
Tweaking based on what
we already know only gets
us so far.
We can test different
things based on what we
already know
But then we need to find the “real problem”

21. @DhyanaScarano
ResearchProblem
Original problem:
CareerFoundry needs to
increase it’s conversions
How to do the research?

22. @DhyanaScarano
Research
Underst-
anding
Testing
ideas
em
The real
Problem
Original problem:
CareerFoundry needs
to increase it’s conversions
more people to click “Enrol”
How to do the research?
We can’t easily find the people who
decided not to enrol with us.
But we can ask the people who did
choose us.
Another challenge: our users are all
over the world, from the U.S. to
China, and even Australia!

23. @DhyanaScarano
Research
Underst-
anding
Testing
ideas
em
The real
Problem
What was in their head just before
they decided to click?
What were they afraid of?
Did they have any doubt or fears?
What was the thing that helped
them decide?
How to do the research?
What can we ask that will help
us figure out what to put on our
landing pages?

24. @DhyanaScarano
Research
Underst-
anding
Testing
ideas
em
The real
Problem
What to ask?
• We’re trying to discover new
information, so…
• Really open questions
• Get them to tell their story
How to do the research?
How to ask it?
• Need to explore and go deeper
(interviews)
• Also need enough data to draw
conclusions (survey)

25. @DhyanaScarano
How to do the research?
The survey
(For enough data)
• Lots of free text responses, for
thoughtful answers
• Open questions. Who, Why,
What, How, Can you tell us…
• Asking them to tell the story of
their decision
• Try to discover things we don’t
know we don’t know
At the end of the survey we asked if
they would be interested in doing a
45 minute interview for a $25
Amazon voucher

26. @DhyanaScarano
How to do the research?
The interviews
(For deeper insights)
• 45 mins
• Field guide / conversation points
• Only seen by the interviewer
• Skype or appear.in used for
video chat
• Topics tailored to go deeper in
to the survey questions

27. @DhyanaScarano
How to do the research?
The interviews
(For deeper insights)
• Recorded audio with a field recorder
• Transcribed notes in Google Docs for
easy sharing
• Highlighted interesting quotes while
typing for easy reference later
Tip: Amazon vouchers were very
difficult in multiple currencies. For the
second round of interviews we didn't
offer an incentive and tried personally
messaging users. This took more
manual effort but a higher percentage
of users agreed to an interview.

28. @DhyanaScarano
How to do the research?
Scheduling interviews
• Services like youcanbook.me and
calendly.com let the interviewee
book a time that suits you.
• About 2 hours to set up and
customise
• Syncs with your calendar and only
shows available time slots to the
customer
• You are notified when someone
makes a booking, and it appears in
your calendar

29. @DhyanaScarano
Underst-
anding
Testing
ideas
Define
The real
Problem
Solution
• Frustration at current job
Very high hopes in CF as a life changer
• Fear of failure & lack of confidence
• Time constraints biggest concern
Spend more time than anticipated
• Longing for community (also offline)
UI & UX students much more so than WebDev
and iOS
General insights
Understanding (the user)

30. @DhyanaScarano
Course specific insights (e.g. WebDev)
From interviews
• Age: 33-53 (older than other courses)
• Diverse professional backgrounds,
often don’t live in a TechHub
• Personal projects or business often a
motivation to learn coding
• Self starters with free code schools,
need in-depth knowledge/skill
polishing
From survey
• 80% interested in WebDev for
some months or years
• 60% considered a career path as
a WebDev before signing up for
the course
• 40 % considered another CF
course
• 70% want to change career or
job, some interested in
freelancing
Understanding (the user)

31. @DhyanaScarano
New insights
“I’m still apprehensive about posting on Slack [...]
I’m worried that if I make a stupid mistake I’ll
feel silly.”
“ One of my biggest concerns was that after
finishing the course my skills wouldn't be job ready.
Since you added the job guarantee it instilled more
confidence that the course is robust.”
“My father was a computer scientist and he
taught me how to code, I realized that I always
liked to code.”
“After reading the reviews, watching the
YouTube channel and all the great feedback
from the staff at CareerFoundry, I've no doubt
that this will be the step I need to take.”
Validation
“I do have a small concern that my work will not
be polished enough in six months to be able to
successfully gain employment as a UI Designer.”
Understanding (the user)

32. @DhyanaScarano
Underst-
anding
Testing
ideas
Define
The real
Problem
Solution
Next?
• Next we’ll update our user personas
(different personas for each course)
• Start to apply changes/tests to our
home page, course landing pages and
other parts of the service based on
these new insights.
• E.g. testing different value propositions
based on what users have told us are
their goals, doubts and fears.
How does this fit into
conversion optimisation?

33. @DhyanaScarano
Research
Underst-
anding
Testing
ideas
DefineProblem
The real
Problem
Solution
Defining the problem Coming up with a solution
(what we’ll talk about now) (Conversion optimisation)
Figuring out what to test Testing options

34. @DhyanaScarano
Tools and acknowledgements
• Surveys: typeform.com
• Video interviews: appear.in (Click-and-go. No install required.)
• Interview scheduling: youcanbook.me, calendly.com
People harmed in the making of this work
Tools
• Clive K Lavery (freelance)
Research lead, chief interviewer, insight gleaner
• Jeff Humble (CF)
Research support, transcriber, Amazon gift card victim
• Claire Rackstraw (CF)
Research support, transcriber, motivator

35. @DhyanaScarano
Thank you :)
Questions?