1. Your
data
is
telling
you
something.
$1B
Who
cares?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
2. Listen
to
it.
$1B
Kevin
Systrom
realizes
customers
only
use
their
product
for
one
thing:
photos.
Burbn
dies,
Instagram
is
born.
Who
cares?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
3. Agenda
• Background
• IdenEfying
opportuniEes
• Using
metrics
to
prioriEze
• TesEng
hypothesis
with
experiments
• Running
“post-‐mortem”
analysis
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
4. Goals
of
“Think
Like
a
PM”
• Introduce
the
idea
of
data
driven
PM’ing
– Focus
on
an
example
using
user
data
• Review
the
“end-‐to-‐end”
process
of
a
data
driven
feature
– Use
Foursquare
as
an
illustraEve
example
• Provide
you
with
another
tool
for
approaching
product
development
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
5. How
do
PMs
decide
what
features
to
build?
• Data
• Talking
to
customers
• Vision
about
the
future
of
the
product
• Beliefs
• Wild-‐ass
guesses
• Looking
at
the
compeEEon
• DirecEon
from
managers
/
execs
• They
don’t
(indecision
strikes!)
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
6. What
types
of
data
do
PMs
focus
on?
• Market
data
– “CompeEtors
that
have
focused
on
Z
approach
have
out-‐
performed
and
we
should
consider
that…”
• Anecdotal
data
– Eg,
“When
we
talk
to
customers,
they
always
complain
about
Y
taking
too
long…”
• User
data
– We
know
25%
of
users
take
X
acEon
in
the
game…”
– Some
famous
examples:
Instagram’s
pivot,
Facebook’s
localizaEon
efforts,
Zynga’s
dominance
of
FB
channels
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
7. Agenda
• Background
• Iden4fying
opportuni4es
• Using
metrics
to
prioriEze
• TesEng
hypothesis
with
experiments
• Running
“post-‐mortem”
analysis
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
8. Why
study
Foursquare?
• Everyone
can
use
it
(it’s
free)
• People
are
familiar
with
it
(25M
users)
• It’s
an
evolving
product
–
you
can
observe
the
Foursquare
team
making
changes
to
the
product
• Clear
defined
user
flows
&
acEons
to
study
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
9. Foursquare
top
level
metrics:
the
“Vanity”
metrics
• 2B
“check-‐ins”
• 25M
registered
users
• 7.2M+
daily
acEve
users
(DAU)
• 20%
of
searches
result
in
a
check-‐in
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012
10. Two
things
to
remember
when
working
with
data
What
is
this?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012
11. Start
with
the
full
picture,
peel
back
layers
of
the
onion
Zoom
out
so
you
can
And
then
you
can
work
on
see
the
whole
picture…
peeling
back
the
layers…
It’s
a
bridge!
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
12. Key
steps
to
idenEfying
opportuniEes
1) Define
a
clear,
measurable
goal
– Eg,
“We
want
to
increase
Foursquare
check-‐ins
/
day”
2) Define
the
relevant
data
set
– Eg
“What
drives
daily
check-‐ins?”
3) Determine
the
status
quo
– Eg,
“What
does
the
current
data
show
about
daily
check-‐
ins?”
4) IdenEfy
opportuniEes
to
improve
the
goal
– Eg,
“What
are
the
inflecEon
points?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
13. Defining
a
clear,
measurable
goal
Foursquare
derives
value
from
loca4on
data
• Check-‐ins
are
a
criEcal
piece
(eg
build
the
database
of
locaEon
data)
• They
have
viral
value
(eg
“Kenton
checked
in
here…)
• Check-‐in
rates
indicate
the
health
of
the
app
/
user
base
(eg,
Check-‐ins
/
day
is
a
good
indicator
of
user
acEvity)
• Result:
Check-‐ins
could
be
a
great
piece
of
data
to
understand
beCer
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
14. Key
steps
to
idenEfying
opportuniEes
1) Define
a
clear,
measurable
goal
– Eg,
“We
want
to
increase
Foursquare
check-‐ins
/
day”
2) Define
the
relevant
data
set
– Eg
“What
drives
daily
check-‐ins?”
3) Determine
the
status
quo
– Eg,
“What
does
the
current
data
show
about
daily
check-‐
ins?”
4) IdenEfy
opportuniEes
to
improve
the
goal
– Eg,
“What
are
the
inflecEon
points?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
15. What’s
the
anatomy
of
a
check-‐in?
iPhone
home
screen
Foursquare
home
Loca4on
picker
Check
in
details
• How
many
• How
many
users
• How
many
users
• How
many
users
users?
reach
it
daily?
reach
it
daily?
reach
it
daily?
• How
many
• How
many
• How
many
• How
many
share
on
decide
to
login
decide
to
click
to
decide
to
select
social
media?
On
on
any
given
iniEate
a
check
an
actual
twiper?
On
day?
in?
locaEon?
facebook?
• How
many
include
a
photo?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
16. Key
steps
to
idenEfying
opportuniEes
1) Define
a
clear,
measurable
goal
– Eg,
“We
want
to
increase
Foursquare
check-‐ins
/
day”
2) Define
the
relevant
data
set
– Eg
“What
drives
daily
check-‐ins?”
3) Determine
the
status
quo
– Eg,
“What
does
the
current
data
show
about
daily
check-‐
ins?”
4) IdenEfy
opportuniEes
to
improve
the
goal
– Eg,
“What
are
the
inflecEon
points?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
17. Foursquare
data:
the
top
level
funnel
of
user
acEvity
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
users
25,000,000
Daily
acEve
users
7,200,000
28.8%
Click
“Check-‐in”
1,800,000
25%
Select
locaEon
900,000
50%
Complete
check-‐in
630,000
70%
Social
Media
sharing
189,000
30%
Share
photo
126,000
20%
No
meta
data
315,000
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
18. Key
steps
to
idenEfying
opportuniEes
1) Define
a
clear,
measurable
goal
– Eg,
“We
want
to
increase
Foursquare
check-‐ins
/
day”
2) Define
the
relevant
data
set
– Eg
“What
drives
daily
check-‐ins?”
3) Determine
the
status
quo
– Eg,
“What
does
the
current
data
show
about
daily
check-‐
ins?”
4) IdenEfy
opportuniEes
to
improve
the
goal
– Eg,
“What
are
the
inflecEon
points?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
19. IdenEfy
opportuniEes
by
understanding
what
the
data
suggests
about
user
behavior
• QuesEons
to
consider:
– What’s
going
on
at
the
top
of
the
funnel?
– At
the
bopom
of
the
funnel?
– Which
acEons
are
we
most
concerned
with?
– Where
do
we
“lose”
the
most
users?
– What’s
working
well?
Why?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
20. Opportunity
#1:
Increase
daily
logins
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
users
25,000,000
Daily
acEve
users
7,200,000
28.8%
Click
“Check-‐in”
1,800,000
25%
1
Select
locaEon
900,000
50%
Only
~29%
of
the
user
base
logs
into
the
app
each
Complete
check-‐in
630,000
day.
One
opportunity
would
be
to
apract
more
70%
users
to
the
app
each
day.
This
would
“widen
the
Social
Media
sharing
189,000
f
the
funnel”
top
o 30%
Share
photo
126,000
20%
No
meta
data
315,000
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
21. Opportunity
#2:
Increase
the
daily
check-‐ins
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
users
25,000,000
Daily
acEve
users
7,200,000
28.8%
Click
“Check-‐in”
1,800,000
25%
Select
locaEon
900,000
50%
2
Complete
check-‐in
630,000
70%
Only
~25%
of
the
user
base
starts
the
“check-‐in”
Social
Media
sharing
process
each
189,000
is
opportunity
to
increase
day.
There
30%
the
number
of
“check-‐ins”
simply
by
gewng
the
Share
photo
126,000
20%
apenEon
of
our
logged
in
users
No
meta
data
315,000
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
22. Opportunity
#3:
Increase
the
%
of
users
selecEng
locaEon
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
users
25,000,000
Daily
acEve
users
7,200,000
28.8%
Click
“Check-‐in”
1,800,000
25%
Select
locaEon
900,000
50%
Complete
check-‐in
630,000
70%
3
Social
Media
sharing
189,000
30%
Only
~50%
of
the
users
that
start
a
“check-‐in”
Share
photo
126,000
20%
actually
select
their
locaEon.
There
is
room
to
opEmize
this
step
of
the
funnel
and
minimize
the
No
meta
data
315,000
drop-‐off
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
23. Opportunity
#4:
Increase
the
number
of
users
compleEng
the
final
check-‐in
step
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
4
sers
u 25,000,000
Daily
acEve
users
7,200,000
28.8%
We
lose
another
30%
of
users
on
the
final
step
of
Click
“Check-‐in”
the
“check-‐in.”
Is
there
anyway
to
prevent
that?
1,800,000
25%
Select
locaEon
900,000
50%
Complete
check-‐in
630,000
70%
Social
Media
sharing
189,000
30%
Share
photo
126,000
20%
No
meta
data
315,000
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
24. Summary:
4
key
steps
to
idenEfying
product
opportuniEes
with
data
Key
things
to
remember:
1) Define
a
clear,
measurable
goal:
“Increasing
check-‐ins”
2) Collect
the
relevant
data
set
&
assemble
it
3) Determine
the
status
quo
4) IdenEfy
opportuniEes
to
improve
the
goal
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
25. Dennis
says:
“I’ve
just
realized
that
…
”
…
for
every
photo
that
gets
shared
on
Twiper
via
Foursquare,
we
acquire
2
new
users.
If
we
could
double
the
amount
of
photos
shared,
we’d
double
our
user
base.
How
many
more
photos
can
we
get
users
sharing
on
Twiper?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
26. Which
of
the
4
opportuniEes
does
Dennis
want
to
take
advantage
of?
Eeeny
…
meeny
…
miny
…
moe
….
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
27. Opportunity
#5:
Increase
the
top
of
the
funnel
by
increasing
the
bopom!
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Total
registered
users
25,000,000
Daily
acEve
users
7,200,000
28.8%
Click
“Check-‐in”
5
1,800,000
25%
Select
locaEon
900,000
50%
Dennis’
insight:
If
we
increase
those
sharing
photos,
We
lose
another
30%
of
users
on
the
final
step
of
we
will
get
more
users
which
will
increase
the
top
of
Complete
check-‐in
the
“check-‐in.”
Is
there
anyway
t70%
630,000
o
prevent
that?
the
funnel
Social
Media
sharing
189,000
30%
Share
photo
126,000
20%
No
meta
data
315,000
50%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Total
registered
users,
DAU
stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012.
All
other
numbers
are
SWAG
at
Foursquare
core
funnel
28. Agenda
• Background
• IdenEfying
opportuniEes
• Using
metrics
to
priori4ze
• TesEng
hypothesis
with
experiments
• Running
“post-‐mortem”
analysis
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
29. Given
Dennis’
goals
of
increasing
photo
shares,
we
need
to
beper
understand
that
data
• QuesEons
to
consider
– What
does
the
photo
sharing
funnel
look
like?
– What
drives
photo
sharing?
– How
do
photos
get
shared
today?
– How
can
we
encourage/discourage
that
behavior
to
achieve
our
goals?
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
30. Zoom
in
on
the
social
media
and
photo
sharing
aspect
of
the
funnel
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
189,000
30%
Shared
to
Twiper
37,800
20%
Shared
to
Twiper
w/
photo
34,020
90%
Shared
to
FB
151,200
80%
Shared
to
FB
w/
photo
15,120
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
31. #1:
Increase
the
%
of
users
who
share
a
photo
ayer
they’ve
decided
to
tweet
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
1
630,000
If
we
increase
the
%
of
users
who
share
a
photo
Social
media
shared
when
they
tweet,
w189,000
that
do
to
our
hat
would
30%
numbers?
Shared
to
Twiper
37,800
20%
Shared
to
Twiper
w/
photo
34,020
90%
Shared
to
FB
151,200
80%
Shared
to
FB
w/
photo
15,120
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
32. By
increasing
Twiper
sharing,
gain
10%+
photo
shares
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
189,000
30%
Shared
to
Twiper
37,800
20%
Shared
to
Twiper
w/
photo
37,800
(+10%)
100%
Shared
to
FB
151,200
80%
Shared
to
FB
w/
photo
15,120
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
33. #2:
Increase
the
%
of
people
sharing
via
social
media
channels
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
189,000
30%
Shared
to
Twiper
37,800
20%
2
Shared
to
Twiper
w/
photo
90%
What
happens
if
we
increase
the
%
of
people
Shared
to
FB
151,200
sharing
via
social
media
from
30%
to
50%?
80%
Shared
to
FB
w/
photo
15,120
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
34. By
increasing
%
of
people
sharing
via
social,
gain
66.6%+
more
photo
shares!
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
315,000
50%
Shared
to
Twiper
63,000
20%
Shared
to
Twiper
w/
photo
56,700
(+66.6%)
90%
Shared
to
FB
252,000
80%
Shared
to
FB
w/
photo
25,200
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
35. #3:
Increase
the
%
of
users
sharing
via
Twiper
vs.
Facebook
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
189,000
30%
Shared
to
Twiper
37,800
20%
Shared
to
Twiper
w/
photo
34,020
90%
3
Shared
to
FB
151,200
80%
What
happens
if
we
increase
the
%
of
users
who
Shared
to
FB
w/
photo
15,120
share
via
Twiper
from
20%
to
50%?
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
36. By
increasing
mix
of
social
shares
to
Twiper,
gain
125%+
photo
–
holy
cow!!
Funnel
Step
Users
hiGng
that
step
%
proceeding
from
previous
Compete
check-‐ins
630,000
Social
media
shared
189,000
30%
Shared
to
Twiper
94,500
50%
Shared
to
Twiper
w/
photo
85,050
(+125%)
90%
Shared
to
FB
94,500
50%
Shared
to
FB
w/
photo
9,450
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
37. If
you
were
forced
to
only
make
1
change,
which
would
it
be?
• Increase
the
%
of
users
who
share
a
photo
when
TweeEng
their
check-‐in
– Expected
impact:
+10%
increase
in
Tweets
w/
photo
• Increase
the
%
of
users
who
decide
to
share
his/her
check-‐in
on
social
media
– Expected
impact:
+66%
increase
in
Tweets
w/
photo
• Increase
%
of
users
who
share
his/her
check-‐in
on
Twiper
vs.
Facebook
– Expected
impact:
+125%
increase
in
Tweets
w/
photo
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
38. If
you
were
forced
to
only
make
1
change,
which
would
it
be?
• Increase
the
%
of
users
who
share
a
photo
when
TweeEng
their
check-‐in
– Expected
impact:
+10%
increase
in
Tweets
w/
photo
• Increase
the
%
of
users
who
decide
to
share
his/her
check-‐in
on
social
media
– Expected
impact:
+66%
increase
in
Tweets
w/
photo
• Increase
%
of
users
who
share
his/her
check-‐in
on
Twiper
vs.
Facebook
– Expected
impact:
+125%
increase
in
Tweets
w/
photo
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
39. How
could
you
increase
%
of
users
sharing
via
Twiper
vs.
Facebook?
Op4ons
to
increase
%
of
TwiCer
shares
• Remove
FB
as
an
opEon
• Make
Twiper
“Opt-‐out”
• Provide
incenEve
to
“Tweet”
(eg,
“Extra
Foursquare
points”
• Make
it
mandatory
for
any
user
w/
a
linked
Twiper
account
• Move
it
“up”
in
the
funnel
• Move
it
“down”
in
the
funnel
and
make
it
“opt-‐out”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
40. How
could
you
increase
%
of
users
sharing
via
Twiper
vs.
Facebook?
Op4ons
to
increase
%
of
TwiCer
shares
• Remove
FB
as
an
opEon
• Make
Twiper
“Opt-‐out”
• Provide
incenEve
to
“Tweet”
(eg,
“Extra
Foursquare
points”
• Make
it
mandatory
for
any
user
w/
a
linked
Twiper
account
• Move
it
“up”
in
the
funnel
• Move
it
“down”
in
the
funnel
and
make
it
“opt-‐out”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
41. AddiEonal
consideraEons
when
prioriEzing
• What
if
we
did
mul4ple
features
together?
– Sure!
That
could
increase
the
expected
impacts
even
further
– NOTE:
Must
be
careful
w/
experiment
design
here
so
results
aren’t
muddled
• What
is
the
maximum
%
of
social
media
shares
that
TwiCer
could
get?
– Data
needed:
What
%
of
users
have
linked
Twiper
accounts?
• What
if
20%
is
the
maximum
share
percentage
(because
only
20%
of
users
have
TwiCer
linked)
– You
need
to
apack
a
different
part
of
the
funnel
– Build
a
feature
that
encourages
users
to
link
Twiper
accounts
• But
there
must
be
more!?
– Could
be
even
*more*
aggressive
by
puwng
social
media
and
photo
sharing
higher
in
the
funnel
– Or
could
make
social
media
sharing
“opt-‐out”
vs.
“opt-‐in”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
42. Agenda
• Background
• IdenEfying
opportuniEes
• Using
metrics
to
prioriEze
• Tes4ng
hypothesis
with
experiments
• Running
“post-‐mortem”
analysis
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
43. A
good
experiment
begins
with
a
clear
hypothesis
• Our
hypothesis:
– We
can
increase
the
%
of
users
sharing
to
Twiper
vs.
Facebook
to
50%
by
making
Twiper
“opt-‐out”
– This
will,
in
turn,
drive
the
number
of
Tweeted
photos
up
125%+
– For
every
addiEonal
Tweeted
photo,
Foursquare
will
gain
2
new
users
/
day
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
44. The
goal:
Prove
the
criEcal
aspects
of
our
hypothesis
• CriEcal
aspects:
– Get
50%
of
social
media
sharers
to
use
Twiper
– Drive
up
Tweeted
photos
+125%
– Acquire
2
new
users
for
each
addiEonal
photo
• To
prove:
– Run
a
controlled
A/B
test
– Setup
a
test
where
50%
of
users
get
status
quo
flow
– The
other
50%
get
the
new
Twiper
“opt-‐out”
flow
– Make
sure
you
have
staEsEcally
significant
sample
sizes
(eg
here
were
using
50%,
~300K
check-‐ins)
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
45. Agenda
• Background
• IdenEfying
opportuniEes
• Using
metrics
to
prioriEze
• TesEng
hypothesis
with
experiments
• Running
“post-‐mortem”
analysis
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
46. Key
steps
to
assembling
the
post-‐mortem
analysis
1) Collect
&
assemble
data
from
test
vs.
control
– Eg,
“What
is
the
core
data
from
the
experiment”
2) Compare
test
results
vs.
expected
results
– Eg
“What
exceeded
or
missed
expectaEons?”
3) What
are
the
next
steps
– Eg,
“Should
we
invest
more
Eme/effort?
If
so,
on
what?
What
will
be
the
impact?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
47. Test
shows
40%
sharing
on
Twiper,
resulEng
in
+78%
in
tweeted
photos
Funnel
Step
Control
(50%
of
Test
(50%
of
users)
users)
Compete
check-‐ins
315,000
315,000
Social
media
shared
94,500
30%
94,500
30%
Shared
to
Twiper
18,900
20%
37,800
40%
Shared
to
Twiper
w/
17,000
90%
30,240
(+78%)
80%
photo
Shared
to
FB
75,600
80%
56,700
60%
Shared
to
FB
w/
photo
7,560
10%
5,670
10%
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
NOTE:
Stats
from
TechCrunch,
“Foursquare
looks
into
a
4th
round”,
Nov.
2,
2012
48. Key
steps
to
assembling
the
post-‐mortem
analysis
1) Collect
&
assemble
data
from
test
vs.
control
– Eg,
“What
is
the
core
data
from
the
experiment”
2) Compare
test
results
vs.
expected
results
– Eg
“What
exceeded
or
missed
expectaEons?”
3) What
are
the
next
steps
– Eg,
“Should
we
invest
more
Eme/effort?
If
so,
on
what?
What
will
be
the
impact?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
49. How
does
this
compare
to
expectaEons?
Why
did
this
happen?
Funnel
Step
Expecta4ons
%
proceeding
Test
(50%
of
users)
%
proceeding
Delta
Compete
check-‐ins
315,000
315,000
Social
media
shared
94,500
30%
94,500
30%
~
Shared
to
Twiper
47,250
50%
37,800
40%
-‐10%
Shared
to
Twiper
w/
42,525
(+125%)
90%
30,240
(+78%)
80%
-‐10%
photo
Shared
to
FB
47,250
50%
56,700
60%
+10%
Shared
to
FB
w/
photo
4,725
10%
5,670
10%
~
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
50. Key
steps
to
assembling
the
post-‐mortem
analysis
1) Collect
&
assemble
data
from
test
vs.
control
– Eg,
“What
is
the
core
data
from
the
experiment”
2) Compare
test
results
vs.
expected
results
– Eg
“What
exceeded
or
missed
expectaEons?”
3) What
are
take
aways
&
next
steps
– Eg,
“Should
we
invest
more
Eme/effort?
If
so,
on
what?
What
will
be
the
impact?”
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
51. Key
quesEons
&
take-‐aways
Key
ques4on
Result
Why?
Next
steps
Did
we
get
50%
of
users
No.
We
got
40%
• Maybe
hit
a
natural
limit
(%
of
• Determine
natural
limit
to
share
on
Twiper?
users
w/
Twiper
accounts)
• Consider
encouraging
account
linking
Did
we
get
+125%
No.
We
got
78%
• Photo
sharing
%
dropped
to
• Can
we
increase
photo
increase
in
photo
80%
sharing
%?
sharing?
• We
only
got
40%
sharing
via
Twiper
(vs.
expected
50%)
What’s
the
upside
ley?
12,525
photo
• If
we
can
tweak
to
hit
goals
of
• What
%
of
that
upside
is
shares
/
day
50%
and
90%
*truly*
achievable
given
our
results?
Was
the
test
a
success?
Yes!
• Proved
that
tweaking
Twiper
• Evaluate
above
opEons,
opEon
can
drive
photo
shares
determine
prioriEes
&
repeat!
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
52. Conclusions
• OrganizaEon
is
key
– Start
with
the
big
picture,
peel
back
the
layers
• Define
clear
goals,
hypothesis
– You
won’t
know
if
your
tests
or
features
worked
if
you
don’t
pre-‐define
a
good
goal
and
hypothesis
• Data
driven
PM’ing
is
applicable
to
all
aspects
– We
focused
on
internal
data
but
you
could
use
it
on
market
data,
with
surveys,
with
organizaEonal
issues,
almost
anything…
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu
53. Thanks
&
final
notes
• Slides
will
be
sent
out
• Contact
info:
– @kivestu
– kivestu@gmail.com
– kentonkivestu.com
(thoughts
on
product
development,
mobile)
SkillShare:
Think
Like
a
PM,
Kenton
Kivestu