Design for Continuous Experimentation

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Publicité

Design for Continuous
Experimentation
Dan McKinley
...

www. .com

Sunday, December 2, 12

and I’m here from etsy.com

The world’s handmade
and vintage marketplace.

Sunday, December 2, 12

Etsy is the world’s handmade and vintage m...

Sunday, December 2, 12

Etsy’s a place where you can buy all kinds of things, including handmade crafts like this
sampler

Sunday, December 2, 12

... or this vintage credenza ...

Sunday, December 2, 12

... and rhinestone-studded underwear made of beef jerky ...

Sunday, December 2, 12

Beef jerky underwear is reasonably popular apparently. we’re on track to sell between $800MM and
9...

OCTOBER 2012
1.5 billion page views
55 m...

We love experiments.

Sunday, December 2, 12

At Etsy, we love experiments and A/B testing. And that’s the main thing I ...

Tons of active A/B tests and rampups.

Sunday, December 2, 12

Here’s a screenshot of an internal view of the various t...

Sunday, December 2, 12

We’ve invested plenty of time and effort into tooling to support this work. This is a
screenshot o...

Sunday, December 2, 12

We’ve built tools that protect us from some gnarly statistics. This wizard does the math for
you a...

Continuous Experimentation
Small, measurable changes.
Keeps...

Seung yun Yoo
Seoul, South Korea

...

Etsy Sales: Two Scenarios
Good product release

...

Another reason we measure:

Sunday, December 2, 12

Another reason we measure:
Experimental results are surprising!

Sunday, December 2, 12

“When I am comparison shopping, I open items in new
tabs. We should do that on Etsy.”

...

Sunday, December 2, 12

So we eventually stopped arguing about this and just tried it. We ran an A/B test that opened up i...

The Horrible Sound of Epic Failure

credit: EmbroideryEverywhere.etsy.com
Sunday, December 2, 12

When we tried that, ...

Surprise!

Sunday, December 2, 12

Surprise! We don’t argue about that anymore.

One big thing we’ve learned
from experiments:

Sunday, December 2, 12

We’ve been at thi...

Design and product process
must change to accommodate
experimentat...

Removing the Search
Inﬁnite Scroll
...

Removing the Search
Inﬁnite Scroll
Dro...

Inﬁnite Scroll
So hot right now.

Sunday, December 2, 12

First I’ll go through o...

Woah

Sunday, December 2, 12

If anyone doesn’t know what I mean by inﬁnite scroll: I mean that we changed search resul...

Seeing more items faster is
presumed to be a better experience.

Sunday, December 2, 12

The r...

Inﬁnite Scroll: Release Plan
(Implied)

1. Build inﬁnite scroll....

Inﬁnite Scroll: Results

Sunday, December 2, 12

so the results,

Inﬁnite Scroll: Results

Spoiler: they were not expected.

Sunday, December 2, 12

not to s...

Inﬁnite Scroll: Results

Median item impressions:

Inﬁ...

Inﬁnite Scroll: Results

Visitors seeing inﬁnite scroll clicked
fe...

Inﬁnite Scroll: Results

Visitors seeing inﬁnite scroll saved
fe...

Inﬁnite Scroll: Results

Visitors seeing inﬁnite scroll purchased
...

Initial reaction:
“something’s broken.”

Sunday, December 2, 12

The ﬁrst thing that occurred ...

Gradual, horrible realization:
“we changed many things at once.”

Sunday, December 2, 12

Eventually we ca...

Premise-validating Experiments
Or: “things we should have
done in the...

Are more items in search results better?

Sunday, December 2, 12

First of all, is it true that more items is better?

Sunday, December 2, 12

We ran a test where we just varied the number of results in normal search results.

Are more items in search results better?

Barely, maybe: more people get to an item
...

Are faster results better?

Sunday, December 2, 12

The other major premise was that faster search results would stop p...

Sunday, December 2, 12

We ran a test where we slowed down search artiﬁcially, by adding sleeps().

Are faster results better?

Meh

Sunday, December 2, 12

Absolutely not...

credit: LunaLetterpress.etsy.com
Sunday, December 2, 12

In the end the expected beneﬁts to inﬁnite scroll just didn’t see...

Inﬁnite Scroll: Release Plan
(Implied)

1. Build inﬁnite scroll...

A Slightly Better Inﬁnite
Scroll Release Plan

1. Validate premise: more items i...

Throwing out work sucks.

Sunday, December 2, 12

Throwing out work feels really horrible. Most of the time this is a r...

Inﬁnite scroll: not stupid.

Sunday, December 2, 12

My point is not that inﬁnite scroll is stupid. It may be great on ...

Removing the Search Dropdown
A much better experience
for me, persona...

2007

Sunday, December 2, 12

Pretty early on, we added this dropdown to the header, mainly to pick between handmade it...

2012

Sunday, December 2, 12

But as these things always do, it got way out of hand. It looked like this ﬁve years late...

Kill the Dropdown: Project Plan

1. Redesign marketplace facets.
2. Default to “all items.”
...

Kill the Dropdown: Project Plan

1. Redesign marketplace facets.
2. Default to “all items.” ...

Kill the Dropdown: Project Plan

1. Redesign marketplace facets.
2. Default to “all items.”
...

Kill the Dropdown: Project Plan

1. Redesign marketplace facets.
2. Default to “all items.”
...

Sunday, December 2, 12

So, the ﬁrst thing we had to address was the fact that the dropdown was used to cut the marketplac...

HYPOTHESIS:

Most users of the site don’t
know anything about th...

Sunday, December 2, 12

First we introduced this faceting on the left side of search results, and made it more obvious. Th...

Sunday, December 2, 12

But still, relatively few people noticed that. So we also built faceting into our autosuggest. We ...

Sales of Vintage Items:

+3.7%

Sunday, December 2, 12

After we did th...

VERIFIED HYPOTHESIS:

Casual users of the site don’t
know anyth...

Context-sensitive!

Sunday, December 2, 12

Another horrible behavior of the search dropdown was that it was context-se...

HYPOTHESIS:

Casual users of the site don’t
realize this.
...

Sunday, December 2, 12

To contend with this we introduced a secondary search box on shop pages so that people could do a
...

Sunday, December 2, 12

We also tried adding this search bar to the item page. But few people used it and those who did
pe...

Sunday, December 2, 12

So we took that part out.
If we had done the whole project all at once, we probably would not have...

Sunday, December 2, 12

Another thing the search dropdown could be used for was searching for shops. Nobody used it.

Sunday, December 2, 12

So we added shops suggestions to item results and made sure more people could ﬁnd shops

...plus ﬁve or ten other things
on the same scale.

Sunday, December 2, 12

So you more...

Data was involved at every step.

Sunday, December 2, 12

Inﬁnite Scroll

Design Develop Measure ಠ_ಠ

D...

Some Advice

Sunday, December 2, 12

I want to leave you with some parting advice.

Experiment with minimal
versions of your idea.

Sunday, December 2, 12

Experiment with a...

Plan on being wrong.

Sunday, December 2, 12

Plan on being wrong. If you measure, you’ll encounter many counterintuiti...

Prefer incremental redesigns.

Sunday, December 2, 12

This will not always work.
Occasionally, you may
need to make big...

...but it usually does.

Sunday, December 2, 12

But if you’re throwing this card down all the time you’re probably doi...

Thank you.
Dan McKinley
dan@etsy.com

Sunday, December 2, 12

than...

Partager Slideshare

LinkedIn
Facebook
Twitter

Intégrer

Taille (px)

Afficher les Slideshows associés à la fin

Code abrégé pour WordPress

Lien

Partager
E-mail

Enchanted Objects: The next wave of... by David Rose 26583 views
Building ZingMe News Feed System by Chau Thanh 4066 views
Choose Boring Technology by Dan McKinley 36063 views
Sơ lược kiến trúc hệ thống Zing Me by zingopen 4905 views
Design a scalable social network: P... by Chau Thanh 1716 views
Inside Zalo: Developing a mobile me... by Zalo_app 12559 views

Afficher sur Slideshare

1 sur 79 Publicité

Afficher sur Slideshare

1 sur 79 Publicité