Data-Driven Design Improves Software Through A/B Testing and Funnel Analysis
•Télécharger en tant que ODP, PDF•
9 j'aime•2,316 vues
The document discusses using data and statistics to improve software and websites. It covers instrumentation to track user behavior, funnel analysis to identify where users drop off, and A/B testing to compare design alternatives. The goal is to supplement design with scientific process to decrease costs, uncertainty and increase satisfaction, success and profits by learning from user data.
![Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Data-Driven Design Improves Software Through A/B Testing and Funnel Analysis
Similaire à Data-Driven Design Improves Software Through A/B Testing and Funnel Analysis (20)
Dernier
Dernier (20)
Data-Driven Design Improves Software Through A/B Testing and Funnel Analysis
- 1. Data-driven Design Making Software & Websites Better Through Use of Statistics Patrick McKenzie April 15th, 2010 Creative Commons Licensed CC-BY. Please feel free to share.
- 4. What Happened D:ropboxy Dropboxhotosakura Season 2010MG_0348.JPG
- 7. Art into Science 芸術から技術へ Supplement the vision and craft of designers with the logic, regimentation, and process of science. デザイナーの芸術を技術の倫理、数学、体験方式で拡張する。
- 11. CrazyEgg
- 23. Before
- 24. After
- 26. http://farm1.static.flickr.com/21/33346213_e5d5842024.jpg Pretend You Own A Bar Salty Peanut by EuroMagic. Half of Customers Get... Half of Customers Get...
- 28. Actual A/B Test Half of Users See... Half of Users See...
- 31. Landing Pages experianvasm experianvbsm Source: whichtestwon.com
- 34. Signup Forms Vast contact dealer form
- 36. You can quickly create bingo cards using our included wizards! Old Version
- 37. C:sersatrickicturesew-version.png Spot The New Feature
- 41. Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/ Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/ Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/ Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/ Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/ Photo Credits
Notes de l'éditeur
- I delivered this presentation on April 15 th , 2010 to the Design Matters group at the Apple Store in Osaka. Please feel free to re-use any part of it. Pictures are mostly from other people and licensed CC-BY as well: see last page for attributions. Non-attributed pictures are my own -- you can consider them CC-BY as well.
- Interesting note: Garr Reynolds, the guy behind Presentation Zen, suggested this slide get excised later (“Outline slides are boring, cost attention, and distract from the narrative.”). Oh, what will I ever do without structure…
- Back when I was in college, outsourcing was in full swing and I thought I would be unlikely to be competitive with hundreds of thousands of Indian/Chinese engineers if I only knew how to program. So I decided to pick up a language, and stumbled into Japanese. I was going to get a nice, safe job at Microsoft or some similar international company after graduation, but didn’t feel my Japanese was up to snuff, so…
- I came to Ogaki (a city of 150,000 in Gifu Prefecture) in 2004 to be a technical translator for the prefectural technology incubator. I worked there for three years, doing a bit of translation, a bit of interpreting, and a bit of R&D when I had neither to keep me busy. I grew to love Ogaki so much that when my contract ended I wanted to stay a little longer, so I became…
- … a Japanese salaryman , at a large company in Nagoya. (No, not THAT large company in Nagoya. One of the other ones.) I learned an awful lot about being a professional engineer, as opposed to the hobbyist with coding skills that I was on graduating university and working at the very low-stress position at the technology incubator. On the flip side, being a salaryman is sort of not conducive with work/life balance, so I was ready for a change after 2.5 years. While this was happening…
- Brief introduction of my business. Bingo Card Creator: creates bingo cards for teachers, mostly for use in lessons. Started 7/2006 (sideline business), went full-time on 4/1/2010. Desktop application (Java) + web version. ~3,000 customers, 500,000 cards created in last year, millions upon millions downloaded.
- Making interface/interaction design into a science, not just an art. Art is lovely, but: * Failure is too common * Uncertainty is too high * Beliefs are too unquestioned * Competent practioners are too rare Using scientific approaches means * Repeatable, scalable processes * Mathematically quantifiable certainty * Constant change with users * Any idiot (i.e. me) can do it
- Business needs addressed by this presentation * Cost cutting: rather than spending lots on a one-shot, do many cheap experiments, develop what works. * Uncertainty: Acknowledge that we don't know what solution looks like, work to discover it, rather than praying we guess right. * Satisfaction: Adapt to measured preferences of users. Give them what they want, not what they ask for. * Success: Measurable improvements in task success. * Profits: All of the above means you make $$$.
- High-level description of software instrumentation. Software instrumentation means tools & techniques for: * Making your software capture usage data * Getting the data out again * Analyzing it * Acting on what you learn
- Quick overview of several types of instrumentation.
- CrazyEgg: heat mapping software. The more people click on an interface element, the ”hotter” it appears. Lets you quickly see what users are actually focused on on your pages/applications. Technical notes: works via Javascript. Business notes: dirt cheap ($9, $19, etc monthly)
- Actual example of a problem solved by CrazyEgg This is a landing page for an adwords campaign. Goal is to have customer click on Big Ol' Button to sign up for service and hopefully purchase it. Note left hand column is not active , but customers are clicking on it. Frustrated customers leave page without signing up, costing money! Note how it is very obvious what the problem is – one look at this shows it. Nice feature to have in instrumentation – the less work you have to do to get useful results, the more likely you are to actually use it.
- Redone version of same page Note left side now has links, and they're being clicked on.
- Paving the trails: observe what people do, then, direct your resources into making what they do better. Has great relevance for creating/extending features, creating content, etc . This story went over VERY well at the presentation: There was once a park in America created by a famous architect. When he delivered his final set of specs, the owners realized that he hadn’t included any roads in the park. They asked him how he was such a famous architect and could forget something more basic. ”Ahh, but I have not forgotten the roads – I merely do not know where to put them yet.” So they asked how he could be a famous architect and not know where to put the roads. ”Oh, I have some ideas. But then I would be laying roads just for me. So I’m going to wait a while, and watch as people walk within the park. The paths they past most often will have the grass stamped down, and trails will form, worn by each pair of passing feet. Then I will tell you where to put your roads. Pave the trails, because a road is where people want to walk. 歩くところこそが道である。 ” In software, if you want to know where you should spend your resources: pave the trails !
- Standard overly complicated options dialog What do customers actually perceive as valuable here, and how can we cut it such that their experience improves?
- Sample of 10,000 customers. Red numbers indicate percentage who tweaked the particular setting. As we can see, customers care about some far more than others. Note that consistently ~8% of customers tweak EVERYTHING. Power users. Identify power users automatically, give them the full version of this screen. Let normal users see only the stuff they're likely to perceive value from. Increases success tremendously (see next section on Funnels).
- Explanation of funnel analysis
- Software to use for Funnel Analysis I really, really like Mixpanel . See http://www.bingocardcreator.com/articles/tracking-with-mixpanel.htm
- Steps folks go through in the core interaction loop in Bingo Card Creator.
- Collecting data on user success at every stage in core interaction loop.
- Find obvious candidates for improvement of core interaction loop.
- Figure out ways to make core interaction loop better at hotspots identified for improvement. These days I’d A/B test all of these suckers, but I did not have a convenient way of doing so at the time.
- The complicated version of the Customize form.
- The simplified version of the Customize form – shown by default now to any user who heuristics identify as a “newbie”. (Heuristics are another topic I could do a presentation on. Ask me some time!)
- Pretend you own a bar in a hip hop happening town like Osaka. Your business model is, obviously, getting patrons to order drinks. There are a lot of things you could do to improve how many drinks you sell: maybe spend more money on advertising the bar, or hiring comely waitresses to stand outside and hawk your bar to passersby, or changing the menu, or even moving locations. But these all cost money and you’re not guaranteed that any will work. And who is to say that if sales go up, it is because of the thing you changed, as opposed to seasonal factors or blind chance? So, because you’re a smart bartender, you start small: you figure “hey, maybe salty peanuts will make people drink more beer”. Peanuts are not more expensive than the edamame (that green soybean) you’re giving folks as an appetizer right now. So you have your staff flip a coin for every party that comes in. Heads, they get edamame , tails, they get salty peanuts. This gets marked discretely on their check. Then, at the end of the week, you sum up how many drinks are ordered by both groups, and apply a little math magic. Since your customers are exactly identical with the exception of their appetizer – because your coin does not play favorites – you know that if your math magic says “Wow, peanuts really work!” then peanuts really work. And if on average peanuts cause every person to order 5 drinks instead of 4, then just switching to peanuts increases your sales 25% without hiring more staff or moving to a new location. Welcome to A/B testing.
- This was around Halloween. Depressingly, the Halloween-themed signup page got thumped.
- Landing pages are easy to test: constant incoming traffic, can create alternatives virtually without impact to rest of site, customers are extraordinarily sensitive to small changes because they don’t yet have an investment in a relationship with you. Test on right won.
- Boring signup form. Signup forms are another great place to test, because a) everybody goes through them (changes have immediate, massive business impact) and they’re so easy to screw up.
- Prettier signup form. Won, by a bit.
- The madlibs test. Not a great A/B test design – too much changes at once, so it is hard to isolate cause of improvement – but the right form squashed the left one. Note when I tried a similar test madlibs lost by quite a bit. Lesson: run your own tests.
- A/B testing can help you tell which 80% of your engineer hours are being wasted prior to burning all of them. If the simplest possible version of the feature gets NO uptake – nobody even clicks on the screen to open it -- then you can be fairly confident that pouring more resources past the click is a waste.
- Bingo Card Creator 2.51.
- Bingo Card Creator 3.0. Did you see the change I labored for a month on? Nope? Good, you’re in good company with about 59,978 of the first 60,000 people to use this version. (The new features are under the Internet menu. Huge failure in giving customers what they wanted and in communicating the existence of a feature they wanted – ask me some other time.)