Setting up a correct plan with statistics can be hard for a number of reasons. This presentation is a primer about good use of statistics and setting up a correct plan for using statistics in a UX environment.
7. 77
2. Measure and Compare
Is this already being measured?
› If you are measuring it, chances are, someone else
already has
• For UX on websites: SUS
› Review published literature to see how other researchers
measure
• How are items phrased and compared
• What scales are being used
• How close do our methods match published findings
11. 11
3. Precision
Continuous statistics
• Google Analytics
measures everything
continuously
• This results in a different
approach to define a “test
group”
• When do we have enough
data to decide on an A/B
test?
13. 13
3. Precision
Estimate example
• Idea: let’s set up a service
to sell t-shirts printed with
memes from 9gag.
• How can we estimate the
sales of this?
15. 15
3. Precision
Estimate: Fermi equation
› Famously used by Drake to estimate the number of
extraterrestrial life
› Estimating things through the relation of entities
16. 16
3. Precision
Fermi equation: 9gag T-shirt
› How many users are on 9gag each month
› What percentage of users requires a shirt that
month
› What percentage of shirts sold are shirts with
memes
› How long are users on the site each month
› How many memes are exposed per time frame
› What percentage of meme are liked enough to want
a shirt
17. 17
3. Precision
Fermi equation: 9gag T-shirt
Shirts Sold = Market Size x Need x Niche-interest x Time x Exposure x Selection
18. 18
3. Precision
Fermi equation: 9gag T-shirt
10 000 000 visitors/month (in the US)
times 5/12 shirts needed/visitor/month
Times 5/30 000 meme-shirts bought/ shirts needed (global)
Times 297 view-minutes / user / month
Times 11 memes-seen / view-minutes
Times 1/300 memes-liked-to-buy / memes-seen
Equals 7 562 average number of meme shirts bought per month by the users
Psychology today notes 10 reasons why we are bad with odds
https://www.psychologytoday.com/articles/200712/10-ways-we-get-the-odds-wrong
Risk and emotion are inseparable
Current situation
literature for keywords that worked well in discriminating between high and low luxury items
see if they have been specifically tested in cars
reference database that can help us interpret the results when we do collect data
User Experience context factors
Tom Van de Zande
So when is a measurement good enough to use it for any type of action?
How long? One day? Two days? A week?
How often? Based on number of qualitative or percentual vistors
Does it depend on the action you want to take?
Results for a treejack they tend to stabilize around 40-60 users
All too often we see budgets blown on unnecessarily large sample sizes. You'll often reach the same conclusions with margins of errors three to five times as wide
So how are we estimating how many shirts we are going to sell?
7562 t-shirts estimated
What if only sell 700
What if we sell 70 000
So we have measurements but how are we now going to take action
Recap on the current situation
We can actually drop the website, because in any statistical process it actually starts with the stats (not what generated the stats)
The problem is that this is emotional, you could quickly invest into certain areas without really knowing what went wrong.
Do you need to invest to increase certain numbers or do you simply need to drop the prodcut, without a plan or estimate up front this is a choice!
Is it the order that is wrong?
Is it the website that created (continues) statistics that does not fit into this type of standardized working?
What’s wrong with this proces?
This makes sense: your estimate gives you a rough idea and depending on different estimates and different course of actions certain action plans can be executed
Because we have have used a Fermi equation we can also look back at the equation and decide
Action plans should focus on the “hard to guess” entities of the fermi equation