SlideShare une entreprise Scribd logo
1  sur  79
Télécharger pour lire hors ligne
Design for Continuous
                    Experimentation
                                Dan McKinley
                            Principal Engineer, Etsy
                             November 30th, 2012




Sunday, December 2, 12

Hi my name’s Dan McKinley
www.   .com




Sunday, December 2, 12

and I’m here from etsy.com
The world’s handmade
        and vintage marketplace.


Sunday, December 2, 12

Etsy is the world’s handmade and vintage marketplace.
Sunday, December 2, 12

Etsy’s a place where you can buy all kinds of things, including handmade crafts like this
sampler
Sunday, December 2, 12

... or this vintage credenza ...
Sunday, December 2, 12

... and rhinestone-studded underwear made of beef jerky ...
Sunday, December 2, 12

Beef jerky underwear is reasonably popular apparently. we’re on track to sell between $800MM and
900MM in goods this year. This makes us about as big as Hot Topic.
OCTOBER 2012
                                         1.5 billion page views
                                         55 million unique visitors
                                         USD $83 million in transactions
                                         4.2 million items sold



                                                        http://www.etsy.com/blog/news/?s=weather+report

Sunday, December 2, 12

We had about 1.5B page views in October which makes us a reasonably large website.
We love experiments.



Sunday, December 2, 12

At Etsy, we love experiments and A/B testing. And that’s the main thing I want to talk about today.
Tons of active A/B tests and rampups.




Sunday, December 2, 12

Here’s a screenshot of an internal view of the various tests and config rampups running on
just one of our pages. As you can see, there are a whole lot of them.
Sunday, December 2, 12

We’ve invested plenty of time and effort into tooling to support this work. This is a
screenshot of our A/B analyzer, which automatically generates a dashboard with important
business metrics for every configured test.
Sunday, December 2, 12

We’ve built tools that protect us from some gnarly statistics. This wizard does the math for
you and lets you know how long an experiment will need to run in order to have a significant
result.
Continuous Experimentation
                             Small, measurable changes.
                                  Keeps us honest.
                          Prevents us from breaking things.




Sunday, December 2, 12

I’m going to call what we do “continuous experimentation,” for the lack of a better term. We try to make
small changes as much as possible, and we measure those changes so that we stay honest and don’t
break the site.
Seung yun Yoo
                                                                Seoul, South Korea

                                                                knifeinthewater.etsy.com




                                          http://www.etsy.com/blog/en/2012/featured-shop-knife-in-the-water/
Sunday, December 2, 12

So what do I mean by “breaking the site?” Well, behind every Etsy shop is a person that
depends on it, and counts on us not to push changes that hurt their business. So we would
be remiss not to measure our changes.
Etsy Sales: Two Scenarios
                                                 Good product release

                                  Awful product release




Sunday, December 2, 12

The second reason we measure product releases is so that we stay honest. Much of Etsy’s
sales are seller-driven, so our graphs currently tend to go up no matter what. Obviously that
can’t continue forever. But we have to use A/B testing to tell if we’ve made things worse or
better.
Another reason we measure:




Sunday, December 2, 12
Another reason we measure:
                         Experimental results are surprising!




Sunday, December 2, 12
“When I am comparison shopping, I open items in new
             tabs. We should do that on Etsy.”

                                                                         - Typical know-it-all
                                                                              Etsy employee




Sunday, December 2, 12

Let me give you an example. A few years ago there was controversy internally at Etsy over whether or
not items should open up in new tabs. Some Etsy employees do this themselves when they’re digging
through search results, and they wish that it happened by default. They thought that the average user
would be happier if this were the case.
Sunday, December 2, 12

So we eventually stopped arguing about this and just tried it. We ran an A/B test that opened up items
in new tabs.
The Horrible Sound of Epic Failure




 credit: EmbroideryEverywhere.etsy.com
Sunday, December 2, 12

When we tried that, 70% more people gave up and left the site after getting a new tab. Maybe some
Etsy employees know how to use tabs in a browser, but my grandmother doesn’t. We’ve replicated this
result more than once.
Surprise!




Sunday, December 2, 12

Surprise! We don’t argue about that anymore.
One big thing we’ve learned
                               from experiments:




Sunday, December 2, 12

We’ve been at this for a while and one of the main things we’ve learned from this, which is the main
thing I want to talk about today,
Design and product process
                         must change to accommodate
                               experimentation.




Sunday, December 2, 12

is that process has to change to accommodate data and experimentation. If you follow a waterfall
process and try to bolt A/B testing onto it, you will fail
Removing the Search
                         Infinite Scroll
                                                                Dropdown




Sunday, December 2, 12

to illustrate this I want to go through two projects that we’ve done
Removing the Search
                         Infinite Scroll
                                                           Dropdown
                    Monolithic release.                       Multi-stage release.
                      Effort up front.                               Iterative.
                Changes many things at once.                  One thing at a time.
                    A/B test as a hurdle.                A/B testing integral to process.
                      Assumptions.                                Hypotheses.




Sunday, December 2, 12

These were two projects done largely by the same team. Infinite scroll was poorly managed, and a
release removing a dropdown in our site header was well managed.
Infinite Scroll
                                      So hot right now.




Sunday, December 2, 12

First I’ll go through our deployment of infinite scroll in search results.
Woah




Sunday, December 2, 12

If anyone doesn’t know what I mean by infinite scroll: I mean that we changed search results so that as
you scroll down, more items load in, indefinitely.
Seeing more items faster is
                         presumed to be a better experience.




Sunday, December 2, 12

The reason we did this was because we thought that it obvious that more items, faster was a better
experience. There’s a lot of web lore out there to that effect, based mostly on some findings Google’s
made in their own search.
Infinite Scroll: Release Plan
                                            (Implied)

              1. Build infinite scroll.
              2. Fix some bugs.
              3. A/B to measure obvious big improvement.
              4. Rent warehouse.
              5. Hold release party in warehouse.




Sunday, December 2, 12

So when we decided to do this we just went for it. We designed and built the feature, and then we
figured we’d release it and it’d be great.
Infinite Scroll: Results




Sunday, December 2, 12

so the results,
Infinite Scroll: Results



                          Spoiler: they were not expected.




Sunday, December 2, 12

not to spoil the surprise, were not what we were expecting.
Infinite Scroll: Results

                             Median item impressions:


                                      Infinite scroll: 40

                                      Control group: 80




Sunday, December 2, 12

People who had infinite scroll saw fewer items in search results than people in the control group, not
more.
Infinite Scroll: Results




                         Visitors seeing infinite scroll clicked
                            fewer results than the control.
Sunday, December 2, 12

they clicked on fewer items.
Infinite Scroll: Results




                         Visitors seeing infinite scroll saved
                              fewer items as favorites.
Sunday, December 2, 12

they saved fewer items as favorites.
Infinite Scroll: Results




                         Visitors seeing infinite scroll purchased
                                 fewer items from search*
Sunday, December 2, 12

They bought fewer items from search.
Now they didn’t buy fewer items overall, they just stopped using search to find those items. Which is
kind of interesting.
It was clear we’d made search worse.
Initial reaction:
                         “something’s broken.”




Sunday, December 2, 12

The first thing that occurred to us is that there must have been bugs in the product that we missed. So
we spent a month trying to figure out if that was the case. We sliced results by browser and geographic
location. We sent a guy to a public library to try using an ancient computer. We did find some bugs, but
none of them changed the overall results.
Gradual, horrible realization:
             “we changed many things at once.”




Sunday, December 2, 12

Eventually we came to terms with the fact that infinite scroll had made the product worse, and we had
changed too many things in the process to have any clue which was the culprit.
Premise-validating Experiments
                          Or: “things we should have
                           done in the first place.”




Sunday, December 2, 12

So, we were in a situation where we weren’t sure if we should continue working on this or not. Even if
we had issues in IE or something, the behavior of people using Chrome wasn’t way better, it was also
worse. How do we know if it’s a good idea to finish this or not?
So we went back and tried to verify that the premises that made us do this were right.
Are more items in search results better?




Sunday, December 2, 12

First of all, is it true that more items is better?
Sunday, December 2, 12

We ran a test where we just varied the number of results in normal search results.
Are more items in search results better?


                         Barely, maybe: more people get to an item
                            page as the result count increases.

                            Absolutely no change in purchases.




Sunday, December 2, 12

And the answer was yes, maybe a little bit, but only barely. There was a very slight improvement in the
number of people that ever got to a item page. But the effect is very slight, and purchases aren’t
sensitive to this. There’s no increase in purchases when we increase the number of search results.
Are faster results better?




Sunday, December 2, 12

The other major premise was that faster search results would stop people from getting bored, and
they’d buy more as a result.
Sunday, December 2, 12

We ran a test where we slowed down search artificially, by adding sleeps().
Are faster results better?


                                               Meh




Sunday, December 2, 12

Absolutely nothing happened. Which isn’t to say that performance is pointless, but people buying items
don’t seem to be sensitive to performance at all.
credit: LunaLetterpress.etsy.com
Sunday, December 2, 12

In the end the expected benefits to infinite scroll just didn’t seem to be there. Our premises were
wrong. So we took infinite scroll out back and we shot it.
Infinite Scroll: Release Plan
                                             (Implied)

              1. Build infinite scroll.                            Lots of work

              2. Fix some bugs.
              3. A/B to measure obvious big improvement.
              4. Rent warehouse.
              5. Hold release party in warehouse.
                                             Didn’t happen

Sunday, December 2, 12

So if we go back to our “product plan,” we see a couple of major things wrong with it. We did a lot of
work, and it was pointless.
A Slightly Better Infinite
                            Scroll Release Plan

              1. Validate premise: more items is better (easy)
              2. Validate premise: faster is better (easy)
              3. Either:
                   A. Abort! (easy)
                   B. Build infinite scroll (hard).



Sunday, December 2, 12

A better way to have done this would have been to validate those premises ahead of time and then
make the call. But we didn’t do that.
Throwing out work sucks.




Sunday, December 2, 12

Throwing out work feels really horrible. Most of the time this is a really difficult choice to make, and
without a lot of honesty and discipline, most teams aren’t going to do it. We are not very rational
creatures in the face of sunk costs.
Infinite scroll: not stupid.




Sunday, December 2, 12

My point is not that infinite scroll is stupid. It may be great on your website. But we should have done a
better job of understanding the people using our website.
Removing the Search Dropdown
                         A much better experience
                            for me, personally.




Sunday, December 2, 12

So that was a bad release. I want to change gears now and go through a good one.
2007




Sunday, December 2, 12

Pretty early on, we added this dropdown to the header, mainly to pick between handmade items and
vintage items. It wasn’t intended to be permanent.
2012




Sunday, December 2, 12

But as these things always do, it got way out of hand. It looked like this five years later.
Kill the Dropdown: Project Plan

              1. Redesign marketplace facets.
              2. Default to “all items.”
              3. Rich autosuggest.
              4. Suggest shops in item results.
              5. Add favorites filter to search results.
              6. Search bars on item and shop pages.
              7. Kill the dropdown.


Sunday, December 2, 12

So we wanted to remove this thing. Chastened by the infinite scroll release, we did our best to plan this
out in smaller steps.
Kill the Dropdown: Project Plan

              1. Redesign marketplace facets.
              2. Default to “all items.”                     Short.
              3. Rich autosuggest.                        Measurable.
              4. Suggest shops in item results.            Isolated.

              5. Add favorites filter to search results.
              6. Search bars on item and shop pages.
              7. Kill the dropdown.


Sunday, December 2, 12

Each of these steps is small and isolated.
Kill the Dropdown: Project Plan

              1. Redesign marketplace facets.
              2. Default to “all items.”
                                                                             Opportunity
              3. Rich autosuggest.                                            to change
              4. Suggest shops in item results.                                 plans.
              5. Add favorites filter to search results.
              6. Search bars on item and shop pages.
              7. Kill the dropdown.


Sunday, December 2, 12

Each step is an opportunity to get real feedback and change directions if we have to.
Kill the Dropdown: Project Plan

              1. Redesign marketplace facets.
              2. Default to “all items.”
              3. Rich autosuggest.
              4. Suggest shops in item results.
              5. Add favorites filter to search results.
              6. Search bars on item and shop pages.
              7. Kill the dropdown.                        Ambitious design goal,
                                                           never out of sight.

Sunday, December 2, 12

And all of the individual releases were small, but the overall design goal was still ambitious.
Sunday, December 2, 12

So, the first thing we had to address was the fact that the dropdown was used to cut the marketplace
by different item types.
HYPOTHESIS:

                             Most users of the site don’t
                             know anything about this.




Sunday, December 2, 12

We were working from a hypothesis that most people using Etsy don’t even notice this. But again, we
had to verify this.
Sunday, December 2, 12

First we introduced this faceting on the left side of search results, and made it more obvious. This
relatively simple and it was an improvement over the old design that nobody used.
Sunday, December 2, 12

But still, relatively few people noticed that. So we also built faceting into our autosuggest. We made it
possible to drill down into categories as you typed.
Sales of Vintage Items:


                                               +3.7%




Sunday, December 2, 12

After we did this, sales of vintage items without the dropdown in place increased almost 4%. So we
increased the ability of buyers on Etsy to find vintage goods, we didn’t decrease it. Which is a great
thing to be able to tell our community.
VERIFIED HYPOTHESIS:

                             Casual users of the site don’t
                              know anything about this.




Sunday, December 2, 12

So we were right. Most people using the site in fact did not know how to use the dropdown for this.
Context-sensitive!




Sunday, December 2, 12

Another horrible behavior of the search dropdown was that it was context-sensitive. So if you were on a
shop page it defaulted to searching within the shop. And in some other situations it would search for
people.
HYPOTHESIS:

                            Casual users of the site don’t
                                    realize this.




Sunday, December 2, 12

So again, we figured that this was too complicated and nobody realized what was happening.
Sunday, December 2, 12

To contend with this we introduced a secondary search box on shop pages so that people could do a
search scoped to just the shop. This worked a lot better.
Sunday, December 2, 12

We also tried adding this search bar to the item page. But few people used it and those who did
performed very poorly.
Sunday, December 2, 12

So we took that part out.
If we had done the whole project all at once, we probably would not have noticed that this detail
sucked.
Sunday, December 2, 12

Another thing the search dropdown could be used for was searching for shops. Nobody used it.
Sunday, December 2, 12

So we added shops suggestions to item results and made sure more people could find shops
...plus five or ten other things
                                on the same scale.




Sunday, December 2, 12

So you more or less get the idea here. We had a big goal, which we could have been unmanageable
as a single release. We did it as ten or fifteen small releases.
Data was involved at every step.




Sunday, December 2, 12
Infinite Scroll


                         Design          Develop                      Measure              ಠ_ಠ


       Dropdown Redesign


                                  ✓           ✓                  ✗                ✓ ...


              Design Develop Measure



Sunday, December 2, 12

Contrasting the two release plans, infinite scroll was a big bet that didn’t work out.
The dropdown redesign was a series of small bets: some worked and some didn’t, but we didn’t have to throw
out everything when things didn’t work
Some Advice




Sunday, December 2, 12

I want to leave you with some parting advice.
Experiment with minimal
                              versions of your idea.




Sunday, December 2, 12

Experiment with a minimal version first. With infinite scroll, we should have verified the premises.
Plan on being wrong.




Sunday, December 2, 12

Plan on being wrong. If you measure, you’ll encounter many counterintuitive results.
Prefer incremental redesigns.




Sunday, December 2, 12
This will not always work.
                             Occasionally, you may
                            need to make big bets on
                                   redesigns.




Sunday, December 2, 12

This is not always going to work: you may still have to make big bets on big redesigns sometimes.
...but it usually does.




Sunday, December 2, 12

But if you’re throwing this card down all the time you’re probably doing it wrong
Thank you.
                           Dan McKinley
                          dan@etsy.com




Sunday, December 2, 12

thanks

Contenu connexe

Tendances

Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Be A Great Product Leader (Dropbox / AirBnB 2013)
Be A Great Product Leader (Dropbox / AirBnB 2013)Be A Great Product Leader (Dropbox / AirBnB 2013)
Be A Great Product Leader (Dropbox / AirBnB 2013)Adam Nash
 
Lean Analytics @ MicroConf
Lean Analytics @ MicroConfLean Analytics @ MicroConf
Lean Analytics @ MicroConfLean Analytics
 
성장을 좋아하는 사람이, 성장하고 싶은 사람에게
성장을 좋아하는 사람이, 성장하고 싶은 사람에게성장을 좋아하는 사람이, 성장하고 싶은 사람에게
성장을 좋아하는 사람이, 성장하고 싶은 사람에게Seongyun Byeon
 
B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020Gokul Rangarajan
 
Little Big Data #1. 바닥부터 시작하는 데이터 인프라
Little Big Data #1. 바닥부터 시작하는 데이터 인프라Little Big Data #1. 바닥부터 시작하는 데이터 인프라
Little Big Data #1. 바닥부터 시작하는 데이터 인프라Seongyun Byeon
 
Lean Analytics workshop (from Lean Startup Conf)
Lean Analytics workshop (from Lean Startup Conf)Lean Analytics workshop (from Lean Startup Conf)
Lean Analytics workshop (from Lean Startup Conf)Lean Analytics
 
[PAP] 실무자를 위한 인과추론 활용 : Best Practices
[PAP] 실무자를 위한 인과추론 활용 : Best Practices[PAP] 실무자를 위한 인과추론 활용 : Best Practices
[PAP] 실무자를 위한 인과추론 활용 : Best PracticesBokyung Choi
 
Deview2020 유저가 좋은 작품(웹툰)을 만났을때
Deview2020 유저가 좋은 작품(웹툰)을 만났을때 Deview2020 유저가 좋은 작품(웹툰)을 만났을때
Deview2020 유저가 좋은 작품(웹툰)을 만났을때 choi kyumin
 
Test & Learn: How to Find Your Product's North Star Metric
Test & Learn: How to Find Your Product's North Star Metric Test & Learn: How to Find Your Product's North Star Metric
Test & Learn: How to Find Your Product's North Star Metric Optimizely
 
5 Growth Hacking Metrics
5 Growth Hacking Metrics5 Growth Hacking Metrics
5 Growth Hacking MetricsSyafrizal Adi
 
Cohort Analysis at Scale
Cohort Analysis at ScaleCohort Analysis at Scale
Cohort Analysis at ScaleBlake Irvine
 
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...Dan Olsen
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongDan Chuparkoff
 
Google analytics 4 - présentation détaillée
Google analytics 4 - présentation détailléeGoogle analytics 4 - présentation détaillée
Google analytics 4 - présentation détailléeBruno Guyot
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...Sudeep Das, Ph.D.
 
Navigating GA4 Google Analytics
Navigating GA4 Google AnalyticsNavigating GA4 Google Analytics
Navigating GA4 Google AnalyticsLaura Lee Dooley
 
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?Juhong Park
 
Lean Analytics: A short summary
Lean Analytics: A short summaryLean Analytics: A short summary
Lean Analytics: A short summaryJan König
 

Tendances (20)

Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Be A Great Product Leader (Dropbox / AirBnB 2013)
Be A Great Product Leader (Dropbox / AirBnB 2013)Be A Great Product Leader (Dropbox / AirBnB 2013)
Be A Great Product Leader (Dropbox / AirBnB 2013)
 
Lean Analytics @ MicroConf
Lean Analytics @ MicroConfLean Analytics @ MicroConf
Lean Analytics @ MicroConf
 
성장을 좋아하는 사람이, 성장하고 싶은 사람에게
성장을 좋아하는 사람이, 성장하고 싶은 사람에게성장을 좋아하는 사람이, 성장하고 싶은 사람에게
성장을 좋아하는 사람이, 성장하고 싶은 사람에게
 
B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020
 
Little Big Data #1. 바닥부터 시작하는 데이터 인프라
Little Big Data #1. 바닥부터 시작하는 데이터 인프라Little Big Data #1. 바닥부터 시작하는 데이터 인프라
Little Big Data #1. 바닥부터 시작하는 데이터 인프라
 
Lean Analytics workshop (from Lean Startup Conf)
Lean Analytics workshop (from Lean Startup Conf)Lean Analytics workshop (from Lean Startup Conf)
Lean Analytics workshop (from Lean Startup Conf)
 
[PAP] 실무자를 위한 인과추론 활용 : Best Practices
[PAP] 실무자를 위한 인과추론 활용 : Best Practices[PAP] 실무자를 위한 인과추론 활용 : Best Practices
[PAP] 실무자를 위한 인과추론 활용 : Best Practices
 
Deview2020 유저가 좋은 작품(웹툰)을 만났을때
Deview2020 유저가 좋은 작품(웹툰)을 만났을때 Deview2020 유저가 좋은 작품(웹툰)을 만났을때
Deview2020 유저가 좋은 작품(웹툰)을 만났을때
 
Test & Learn: How to Find Your Product's North Star Metric
Test & Learn: How to Find Your Product's North Star Metric Test & Learn: How to Find Your Product's North Star Metric
Test & Learn: How to Find Your Product's North Star Metric
 
5 Growth Hacking Metrics
5 Growth Hacking Metrics5 Growth Hacking Metrics
5 Growth Hacking Metrics
 
Cohort Analysis at Scale
Cohort Analysis at ScaleCohort Analysis at Scale
Cohort Analysis at Scale
 
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
 
Google Analytics 4: A Quick Start Guide
 Google Analytics 4: A Quick Start Guide Google Analytics 4: A Quick Start Guide
Google Analytics 4: A Quick Start Guide
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
 
Google analytics 4 - présentation détaillée
Google analytics 4 - présentation détailléeGoogle analytics 4 - présentation détaillée
Google analytics 4 - présentation détaillée
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Navigating GA4 Google Analytics
Navigating GA4 Google AnalyticsNavigating GA4 Google Analytics
Navigating GA4 Google Analytics
 
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
 
Lean Analytics: A short summary
Lean Analytics: A short summaryLean Analytics: A short summary
Lean Analytics: A short summary
 

En vedette

Enchanted Objects: The next wave of the web
Enchanted Objects: The next wave of the webEnchanted Objects: The next wave of the web
Enchanted Objects: The next wave of the webDavid Rose
 
Building ZingMe News Feed System
Building ZingMe News Feed SystemBuilding ZingMe News Feed System
Building ZingMe News Feed SystemChau Thanh
 
Choose Boring Technology
Choose Boring TechnologyChoose Boring Technology
Choose Boring TechnologyDan McKinley
 
Sơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing MeSơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing Mezingopen
 
Design a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutionsDesign a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutionsChau Thanh
 
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...Zalo_app
 
Etsy Activity Feeds Architecture
Etsy Activity Feeds ArchitectureEtsy Activity Feeds Architecture
Etsy Activity Feeds ArchitectureDan McKinley
 
Essentials of Product Management
Essentials of Product ManagementEssentials of Product Management
Essentials of Product ManagementJanna Bastow
 
The Art of Product Management
The Art of Product ManagementThe Art of Product Management
The Art of Product ManagementSachin Rekhi
 
A Product Manager's Job
A Product Manager's JobA Product Manager's Job
A Product Manager's Jobjoshelman
 

En vedette (10)

Enchanted Objects: The next wave of the web
Enchanted Objects: The next wave of the webEnchanted Objects: The next wave of the web
Enchanted Objects: The next wave of the web
 
Building ZingMe News Feed System
Building ZingMe News Feed SystemBuilding ZingMe News Feed System
Building ZingMe News Feed System
 
Choose Boring Technology
Choose Boring TechnologyChoose Boring Technology
Choose Boring Technology
 
Sơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing MeSơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing Me
 
Design a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutionsDesign a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutions
 
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
 
Etsy Activity Feeds Architecture
Etsy Activity Feeds ArchitectureEtsy Activity Feeds Architecture
Etsy Activity Feeds Architecture
 
Essentials of Product Management
Essentials of Product ManagementEssentials of Product Management
Essentials of Product Management
 
The Art of Product Management
The Art of Product ManagementThe Art of Product Management
The Art of Product Management
 
A Product Manager's Job
A Product Manager's JobA Product Manager's Job
A Product Manager's Job
 

Similaire à Design for Continuous Experimentation

Big Data, Big Changes: Data-Driven Product Development at Etsy
Big Data, Big Changes: Data-Driven Product Development at EtsyBig Data, Big Changes: Data-Driven Product Development at Etsy
Big Data, Big Changes: Data-Driven Product Development at EtsyJason Davis
 
Data driven-products-now
Data driven-products-nowData driven-products-now
Data driven-products-nowDan McKinley
 
Shift from Packaged Software to Hosted Services
Shift from Packaged Software to Hosted ServicesShift from Packaged Software to Hosted Services
Shift from Packaged Software to Hosted ServicesSVPMA
 
Pet Startups or Create Your Own Pet Related Tech Startup!
Pet Startups or Create Your Own Pet Related Tech Startup!Pet Startups or Create Your Own Pet Related Tech Startup!
Pet Startups or Create Your Own Pet Related Tech Startup!BarkWorld Expo
 
Game Product Discovery: Validation & Iteration
Game Product Discovery: Validation & IterationGame Product Discovery: Validation & Iteration
Game Product Discovery: Validation & IterationMartyn Jones
 
Validation and Iteration in Games - Martyn Jones
Validation and Iteration in Games - Martyn JonesValidation and Iteration in Games - Martyn Jones
Validation and Iteration in Games - Martyn JonesMind the Product
 
iOS Prototyping with Xcode Storyboards
iOS Prototyping with Xcode StoryboardsiOS Prototyping with Xcode Storyboards
iOS Prototyping with Xcode StoryboardsKyle Oba
 
Turning Data Into Meaningful Insights
Turning Data Into Meaningful InsightsTurning Data Into Meaningful Insights
Turning Data Into Meaningful InsightsMark Joseph L. Tan
 
99 inception-deck
99 inception-deck99 inception-deck
99 inception-deckdrewz lin
 
Creating Powerful Proposals - Groupmind, Inc.
Creating Powerful Proposals - Groupmind, Inc.Creating Powerful Proposals - Groupmind, Inc.
Creating Powerful Proposals - Groupmind, Inc.John Sexton
 
How to Get There - An Ideathon Toolkit
How to Get There - An Ideathon ToolkitHow to Get There - An Ideathon Toolkit
How to Get There - An Ideathon ToolkitDaniel Teng
 
Class 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designClass 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designDon Stanley
 
NCDevCon2012_designing the mobile experience
NCDevCon2012_designing the mobile experienceNCDevCon2012_designing the mobile experience
NCDevCon2012_designing the mobile experienceDee Sadler
 
The Web Designers Toolkit
The Web Designers ToolkitThe Web Designers Toolkit
The Web Designers ToolkitR/GA
 
Beyond Milkshakes - Using JTBD for Innovation & Product Design
Beyond Milkshakes - Using JTBD for Innovation & Product DesignBeyond Milkshakes - Using JTBD for Innovation & Product Design
Beyond Milkshakes - Using JTBD for Innovation & Product DesignRedgate-Design
 

Similaire à Design for Continuous Experimentation (20)

Big Data, Big Changes: Data-Driven Product Development at Etsy
Big Data, Big Changes: Data-Driven Product Development at EtsyBig Data, Big Changes: Data-Driven Product Development at Etsy
Big Data, Big Changes: Data-Driven Product Development at Etsy
 
Data driven-products-now
Data driven-products-nowData driven-products-now
Data driven-products-now
 
Shift from Packaged Software to Hosted Services
Shift from Packaged Software to Hosted ServicesShift from Packaged Software to Hosted Services
Shift from Packaged Software to Hosted Services
 
Pet Startups or Create Your Own Pet Related Tech Startup!
Pet Startups or Create Your Own Pet Related Tech Startup!Pet Startups or Create Your Own Pet Related Tech Startup!
Pet Startups or Create Your Own Pet Related Tech Startup!
 
Game Product Discovery: Validation & Iteration
Game Product Discovery: Validation & IterationGame Product Discovery: Validation & Iteration
Game Product Discovery: Validation & Iteration
 
Validation and Iteration in Games - Martyn Jones
Validation and Iteration in Games - Martyn JonesValidation and Iteration in Games - Martyn Jones
Validation and Iteration in Games - Martyn Jones
 
iOS Prototyping with Xcode Storyboards
iOS Prototyping with Xcode StoryboardsiOS Prototyping with Xcode Storyboards
iOS Prototyping with Xcode Storyboards
 
Turning Data Into Meaningful Insights
Turning Data Into Meaningful InsightsTurning Data Into Meaningful Insights
Turning Data Into Meaningful Insights
 
13 0212 toccon - carpenter altmetrics 2
13 0212 toccon - carpenter altmetrics 213 0212 toccon - carpenter altmetrics 2
13 0212 toccon - carpenter altmetrics 2
 
99 inception-deck
99 inception-deck99 inception-deck
99 inception-deck
 
Continous delivery
Continous deliveryContinous delivery
Continous delivery
 
Creating Powerful Proposals - Groupmind, Inc.
Creating Powerful Proposals - Groupmind, Inc.Creating Powerful Proposals - Groupmind, Inc.
Creating Powerful Proposals - Groupmind, Inc.
 
The 4Cs in the 21st C
The 4Cs in the 21st CThe 4Cs in the 21st C
The 4Cs in the 21st C
 
How to Get There - An Ideathon Toolkit
How to Get There - An Ideathon ToolkitHow to Get There - An Ideathon Toolkit
How to Get There - An Ideathon Toolkit
 
Class 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website designClass 2: Setting the foundation for a successful website design
Class 2: Setting the foundation for a successful website design
 
NCDevCon2012_designing the mobile experience
NCDevCon2012_designing the mobile experienceNCDevCon2012_designing the mobile experience
NCDevCon2012_designing the mobile experience
 
The Web Designers Toolkit
The Web Designers ToolkitThe Web Designers Toolkit
The Web Designers Toolkit
 
Dan mc kinley_final
Dan mc kinley_finalDan mc kinley_final
Dan mc kinley_final
 
Scalding at Etsy
Scalding at EtsyScalding at Etsy
Scalding at Etsy
 
Beyond Milkshakes - Using JTBD for Innovation & Product Design
Beyond Milkshakes - Using JTBD for Innovation & Product DesignBeyond Milkshakes - Using JTBD for Innovation & Product Design
Beyond Milkshakes - Using JTBD for Innovation & Product Design
 

Dernier

Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 

Dernier (20)

Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 

Design for Continuous Experimentation

  • 1. Design for Continuous Experimentation Dan McKinley Principal Engineer, Etsy November 30th, 2012 Sunday, December 2, 12 Hi my name’s Dan McKinley
  • 2. www. .com Sunday, December 2, 12 and I’m here from etsy.com
  • 3. The world’s handmade and vintage marketplace. Sunday, December 2, 12 Etsy is the world’s handmade and vintage marketplace.
  • 4. Sunday, December 2, 12 Etsy’s a place where you can buy all kinds of things, including handmade crafts like this sampler
  • 5. Sunday, December 2, 12 ... or this vintage credenza ...
  • 6. Sunday, December 2, 12 ... and rhinestone-studded underwear made of beef jerky ...
  • 7. Sunday, December 2, 12 Beef jerky underwear is reasonably popular apparently. we’re on track to sell between $800MM and 900MM in goods this year. This makes us about as big as Hot Topic.
  • 8. OCTOBER 2012 1.5 billion page views 55 million unique visitors USD $83 million in transactions 4.2 million items sold http://www.etsy.com/blog/news/?s=weather+report Sunday, December 2, 12 We had about 1.5B page views in October which makes us a reasonably large website.
  • 9. We love experiments. Sunday, December 2, 12 At Etsy, we love experiments and A/B testing. And that’s the main thing I want to talk about today.
  • 10. Tons of active A/B tests and rampups. Sunday, December 2, 12 Here’s a screenshot of an internal view of the various tests and config rampups running on just one of our pages. As you can see, there are a whole lot of them.
  • 11. Sunday, December 2, 12 We’ve invested plenty of time and effort into tooling to support this work. This is a screenshot of our A/B analyzer, which automatically generates a dashboard with important business metrics for every configured test.
  • 12. Sunday, December 2, 12 We’ve built tools that protect us from some gnarly statistics. This wizard does the math for you and lets you know how long an experiment will need to run in order to have a significant result.
  • 13. Continuous Experimentation Small, measurable changes. Keeps us honest. Prevents us from breaking things. Sunday, December 2, 12 I’m going to call what we do “continuous experimentation,” for the lack of a better term. We try to make small changes as much as possible, and we measure those changes so that we stay honest and don’t break the site.
  • 14. Seung yun Yoo Seoul, South Korea knifeinthewater.etsy.com http://www.etsy.com/blog/en/2012/featured-shop-knife-in-the-water/ Sunday, December 2, 12 So what do I mean by “breaking the site?” Well, behind every Etsy shop is a person that depends on it, and counts on us not to push changes that hurt their business. So we would be remiss not to measure our changes.
  • 15. Etsy Sales: Two Scenarios Good product release Awful product release Sunday, December 2, 12 The second reason we measure product releases is so that we stay honest. Much of Etsy’s sales are seller-driven, so our graphs currently tend to go up no matter what. Obviously that can’t continue forever. But we have to use A/B testing to tell if we’ve made things worse or better.
  • 16. Another reason we measure: Sunday, December 2, 12
  • 17. Another reason we measure: Experimental results are surprising! Sunday, December 2, 12
  • 18. “When I am comparison shopping, I open items in new tabs. We should do that on Etsy.” - Typical know-it-all Etsy employee Sunday, December 2, 12 Let me give you an example. A few years ago there was controversy internally at Etsy over whether or not items should open up in new tabs. Some Etsy employees do this themselves when they’re digging through search results, and they wish that it happened by default. They thought that the average user would be happier if this were the case.
  • 19. Sunday, December 2, 12 So we eventually stopped arguing about this and just tried it. We ran an A/B test that opened up items in new tabs.
  • 20. The Horrible Sound of Epic Failure credit: EmbroideryEverywhere.etsy.com Sunday, December 2, 12 When we tried that, 70% more people gave up and left the site after getting a new tab. Maybe some Etsy employees know how to use tabs in a browser, but my grandmother doesn’t. We’ve replicated this result more than once.
  • 21. Surprise! Sunday, December 2, 12 Surprise! We don’t argue about that anymore.
  • 22. One big thing we’ve learned from experiments: Sunday, December 2, 12 We’ve been at this for a while and one of the main things we’ve learned from this, which is the main thing I want to talk about today,
  • 23. Design and product process must change to accommodate experimentation. Sunday, December 2, 12 is that process has to change to accommodate data and experimentation. If you follow a waterfall process and try to bolt A/B testing onto it, you will fail
  • 24. Removing the Search Infinite Scroll Dropdown Sunday, December 2, 12 to illustrate this I want to go through two projects that we’ve done
  • 25. Removing the Search Infinite Scroll Dropdown Monolithic release. Multi-stage release. Effort up front. Iterative. Changes many things at once. One thing at a time. A/B test as a hurdle. A/B testing integral to process. Assumptions. Hypotheses. Sunday, December 2, 12 These were two projects done largely by the same team. Infinite scroll was poorly managed, and a release removing a dropdown in our site header was well managed.
  • 26. Infinite Scroll So hot right now. Sunday, December 2, 12 First I’ll go through our deployment of infinite scroll in search results.
  • 27. Woah Sunday, December 2, 12 If anyone doesn’t know what I mean by infinite scroll: I mean that we changed search results so that as you scroll down, more items load in, indefinitely.
  • 28. Seeing more items faster is presumed to be a better experience. Sunday, December 2, 12 The reason we did this was because we thought that it obvious that more items, faster was a better experience. There’s a lot of web lore out there to that effect, based mostly on some findings Google’s made in their own search.
  • 29. Infinite Scroll: Release Plan (Implied) 1. Build infinite scroll. 2. Fix some bugs. 3. A/B to measure obvious big improvement. 4. Rent warehouse. 5. Hold release party in warehouse. Sunday, December 2, 12 So when we decided to do this we just went for it. We designed and built the feature, and then we figured we’d release it and it’d be great.
  • 30. Infinite Scroll: Results Sunday, December 2, 12 so the results,
  • 31. Infinite Scroll: Results Spoiler: they were not expected. Sunday, December 2, 12 not to spoil the surprise, were not what we were expecting.
  • 32. Infinite Scroll: Results Median item impressions: Infinite scroll: 40 Control group: 80 Sunday, December 2, 12 People who had infinite scroll saw fewer items in search results than people in the control group, not more.
  • 33. Infinite Scroll: Results Visitors seeing infinite scroll clicked fewer results than the control. Sunday, December 2, 12 they clicked on fewer items.
  • 34. Infinite Scroll: Results Visitors seeing infinite scroll saved fewer items as favorites. Sunday, December 2, 12 they saved fewer items as favorites.
  • 35. Infinite Scroll: Results Visitors seeing infinite scroll purchased fewer items from search* Sunday, December 2, 12 They bought fewer items from search. Now they didn’t buy fewer items overall, they just stopped using search to find those items. Which is kind of interesting. It was clear we’d made search worse.
  • 36. Initial reaction: “something’s broken.” Sunday, December 2, 12 The first thing that occurred to us is that there must have been bugs in the product that we missed. So we spent a month trying to figure out if that was the case. We sliced results by browser and geographic location. We sent a guy to a public library to try using an ancient computer. We did find some bugs, but none of them changed the overall results.
  • 37. Gradual, horrible realization: “we changed many things at once.” Sunday, December 2, 12 Eventually we came to terms with the fact that infinite scroll had made the product worse, and we had changed too many things in the process to have any clue which was the culprit.
  • 38. Premise-validating Experiments Or: “things we should have done in the first place.” Sunday, December 2, 12 So, we were in a situation where we weren’t sure if we should continue working on this or not. Even if we had issues in IE or something, the behavior of people using Chrome wasn’t way better, it was also worse. How do we know if it’s a good idea to finish this or not? So we went back and tried to verify that the premises that made us do this were right.
  • 39. Are more items in search results better? Sunday, December 2, 12 First of all, is it true that more items is better?
  • 40. Sunday, December 2, 12 We ran a test where we just varied the number of results in normal search results.
  • 41. Are more items in search results better? Barely, maybe: more people get to an item page as the result count increases. Absolutely no change in purchases. Sunday, December 2, 12 And the answer was yes, maybe a little bit, but only barely. There was a very slight improvement in the number of people that ever got to a item page. But the effect is very slight, and purchases aren’t sensitive to this. There’s no increase in purchases when we increase the number of search results.
  • 42. Are faster results better? Sunday, December 2, 12 The other major premise was that faster search results would stop people from getting bored, and they’d buy more as a result.
  • 43. Sunday, December 2, 12 We ran a test where we slowed down search artificially, by adding sleeps().
  • 44. Are faster results better? Meh Sunday, December 2, 12 Absolutely nothing happened. Which isn’t to say that performance is pointless, but people buying items don’t seem to be sensitive to performance at all.
  • 45. credit: LunaLetterpress.etsy.com Sunday, December 2, 12 In the end the expected benefits to infinite scroll just didn’t seem to be there. Our premises were wrong. So we took infinite scroll out back and we shot it.
  • 46. Infinite Scroll: Release Plan (Implied) 1. Build infinite scroll. Lots of work 2. Fix some bugs. 3. A/B to measure obvious big improvement. 4. Rent warehouse. 5. Hold release party in warehouse. Didn’t happen Sunday, December 2, 12 So if we go back to our “product plan,” we see a couple of major things wrong with it. We did a lot of work, and it was pointless.
  • 47. A Slightly Better Infinite Scroll Release Plan 1. Validate premise: more items is better (easy) 2. Validate premise: faster is better (easy) 3. Either: A. Abort! (easy) B. Build infinite scroll (hard). Sunday, December 2, 12 A better way to have done this would have been to validate those premises ahead of time and then make the call. But we didn’t do that.
  • 48. Throwing out work sucks. Sunday, December 2, 12 Throwing out work feels really horrible. Most of the time this is a really difficult choice to make, and without a lot of honesty and discipline, most teams aren’t going to do it. We are not very rational creatures in the face of sunk costs.
  • 49. Infinite scroll: not stupid. Sunday, December 2, 12 My point is not that infinite scroll is stupid. It may be great on your website. But we should have done a better job of understanding the people using our website.
  • 50. Removing the Search Dropdown A much better experience for me, personally. Sunday, December 2, 12 So that was a bad release. I want to change gears now and go through a good one.
  • 51. 2007 Sunday, December 2, 12 Pretty early on, we added this dropdown to the header, mainly to pick between handmade items and vintage items. It wasn’t intended to be permanent.
  • 52. 2012 Sunday, December 2, 12 But as these things always do, it got way out of hand. It looked like this five years later.
  • 53. Kill the Dropdown: Project Plan 1. Redesign marketplace facets. 2. Default to “all items.” 3. Rich autosuggest. 4. Suggest shops in item results. 5. Add favorites filter to search results. 6. Search bars on item and shop pages. 7. Kill the dropdown. Sunday, December 2, 12 So we wanted to remove this thing. Chastened by the infinite scroll release, we did our best to plan this out in smaller steps.
  • 54. Kill the Dropdown: Project Plan 1. Redesign marketplace facets. 2. Default to “all items.” Short. 3. Rich autosuggest. Measurable. 4. Suggest shops in item results. Isolated. 5. Add favorites filter to search results. 6. Search bars on item and shop pages. 7. Kill the dropdown. Sunday, December 2, 12 Each of these steps is small and isolated.
  • 55. Kill the Dropdown: Project Plan 1. Redesign marketplace facets. 2. Default to “all items.” Opportunity 3. Rich autosuggest. to change 4. Suggest shops in item results. plans. 5. Add favorites filter to search results. 6. Search bars on item and shop pages. 7. Kill the dropdown. Sunday, December 2, 12 Each step is an opportunity to get real feedback and change directions if we have to.
  • 56. Kill the Dropdown: Project Plan 1. Redesign marketplace facets. 2. Default to “all items.” 3. Rich autosuggest. 4. Suggest shops in item results. 5. Add favorites filter to search results. 6. Search bars on item and shop pages. 7. Kill the dropdown. Ambitious design goal, never out of sight. Sunday, December 2, 12 And all of the individual releases were small, but the overall design goal was still ambitious.
  • 57. Sunday, December 2, 12 So, the first thing we had to address was the fact that the dropdown was used to cut the marketplace by different item types.
  • 58. HYPOTHESIS: Most users of the site don’t know anything about this. Sunday, December 2, 12 We were working from a hypothesis that most people using Etsy don’t even notice this. But again, we had to verify this.
  • 59. Sunday, December 2, 12 First we introduced this faceting on the left side of search results, and made it more obvious. This relatively simple and it was an improvement over the old design that nobody used.
  • 60. Sunday, December 2, 12 But still, relatively few people noticed that. So we also built faceting into our autosuggest. We made it possible to drill down into categories as you typed.
  • 61. Sales of Vintage Items: +3.7% Sunday, December 2, 12 After we did this, sales of vintage items without the dropdown in place increased almost 4%. So we increased the ability of buyers on Etsy to find vintage goods, we didn’t decrease it. Which is a great thing to be able to tell our community.
  • 62. VERIFIED HYPOTHESIS: Casual users of the site don’t know anything about this. Sunday, December 2, 12 So we were right. Most people using the site in fact did not know how to use the dropdown for this.
  • 63. Context-sensitive! Sunday, December 2, 12 Another horrible behavior of the search dropdown was that it was context-sensitive. So if you were on a shop page it defaulted to searching within the shop. And in some other situations it would search for people.
  • 64. HYPOTHESIS: Casual users of the site don’t realize this. Sunday, December 2, 12 So again, we figured that this was too complicated and nobody realized what was happening.
  • 65. Sunday, December 2, 12 To contend with this we introduced a secondary search box on shop pages so that people could do a search scoped to just the shop. This worked a lot better.
  • 66. Sunday, December 2, 12 We also tried adding this search bar to the item page. But few people used it and those who did performed very poorly.
  • 67. Sunday, December 2, 12 So we took that part out. If we had done the whole project all at once, we probably would not have noticed that this detail sucked.
  • 68. Sunday, December 2, 12 Another thing the search dropdown could be used for was searching for shops. Nobody used it.
  • 69. Sunday, December 2, 12 So we added shops suggestions to item results and made sure more people could find shops
  • 70. ...plus five or ten other things on the same scale. Sunday, December 2, 12 So you more or less get the idea here. We had a big goal, which we could have been unmanageable as a single release. We did it as ten or fifteen small releases.
  • 71. Data was involved at every step. Sunday, December 2, 12
  • 72. Infinite Scroll Design Develop Measure ಠ_ಠ Dropdown Redesign ✓ ✓ ✗ ✓ ... Design Develop Measure Sunday, December 2, 12 Contrasting the two release plans, infinite scroll was a big bet that didn’t work out. The dropdown redesign was a series of small bets: some worked and some didn’t, but we didn’t have to throw out everything when things didn’t work
  • 73. Some Advice Sunday, December 2, 12 I want to leave you with some parting advice.
  • 74. Experiment with minimal versions of your idea. Sunday, December 2, 12 Experiment with a minimal version first. With infinite scroll, we should have verified the premises.
  • 75. Plan on being wrong. Sunday, December 2, 12 Plan on being wrong. If you measure, you’ll encounter many counterintuitive results.
  • 77. This will not always work. Occasionally, you may need to make big bets on redesigns. Sunday, December 2, 12 This is not always going to work: you may still have to make big bets on big redesigns sometimes.
  • 78. ...but it usually does. Sunday, December 2, 12 But if you’re throwing this card down all the time you’re probably doing it wrong
  • 79. Thank you. Dan McKinley dan@etsy.com Sunday, December 2, 12 thanks