At giffgaff, we do everything differently, including lean UX; yes, we even experiment on the experimental method! This case study recounts our journey to hypothesis-driven development, and how we adopted and adapted it as we re-booted the whole way we do business.
22. No matter how
smart a person is,
No matter how
smart a person is,
no matter how
elegant their
hypothesis,
No matter how
smart a person is,
no matter how
elegant their
hypothesis, if it
does not agree
with
experimentation
No matter how
smart a person is,
no matter how
elegant their
hypothesis, if it
does not agree
with
experimentation, it
is wrong
23. What is the most important
thing we need to learn
next?
This is me – guitarist and cyclist. Here’s me presenting at the 02 Accelerate conference in full lycra (everyone else was in a suit)
Giffgaff are the UK’s leading SIM provider – no contracts (“Free to stay free to go”)
giffgaff is based around a community model – run by our members, e.g. we have no help centre, the members answer all queries, Members make most of our sales, etc.
We used to do Scrum and 2 weekly release, but that didn’t really work (we failed 60% of the releases…), so…
…we had a big freeze and downed tools to move to Continuous Delivery (CD). 2 weekly releases only gives us 26 opportunities a year to learn things, CD – actually twice daily releases - allows us to learn about our members every few hours…
Developers picked their own teams (every month-ish) based on what they knew, were interested in, or wanted to skill up in, and we built up our CD pipeline, and a set of automated tests that gave us the confidence in automated releases.
We had lots of mobbing e.g. round a big telly writing automated tests as here….
We eventually realized we needed to get stable tests so all mobbed on the last few remaining pain points
Page Load Timeout was a dark time, where we nearly failed…
But, we madr it, and now have fully automated releases – if the tests are green, the code goes automatically into production
Given we now had a shiny new CD capability, we needed a new lean way of building products
We set up a Lean UX workshop with Jeff Gothelf – author of Lean UX - attended by the whole company…
As part of this, we set up loads of member interviews, on video and/or phone to rapidly validate ideas and assumptions…
The workshop culminated in a Science Fair, where every team presented what they’d done. Some teams went from initial idea to production release within the 5 days – top stuff. Even more impressive, some teams worked out that they *shouldn’t* build some things which is, in fact more interesting. Rather than spend 6 months developing a feature, if we can prove nobody wants it in a few hours, that is a big win!
A video of a wonderful “Wizard of Oz” experiment – pressing a cardboard button stuck on a window that says “Send more Info” leads to a text on your phone. Cool, huh?
We decide to go for a data-/hypothesis-driven approach. We’d built some pretty big features that weren’t successful, so wanted to make sure we were building the right thing quicker and cheaper
The starting point on Lean UX is solving user problems, not business problems. As an example, we didn’t get from barber shop shaving to electric razors by askng “how can we increase shaving revenue in our barber shops?”, we did it by identifying user problems:
Customers don’t have time to go to the barber everyday – so, we sell them their own razors
Customers didn’t have the skills to use a cut throat razor without cutting themselves – so, we sold them safety razors
Customers didn’t have time to lather up every day. But, they had dead time on the train – so, we sold them electric razors
...and so on
We decide to go for a data-/hypothesis-driven approach. We’d built some pretty big features that weren’t successful, so wanted to make sure we were building the right thing quicker and cheaper
It all boils down to these 2 questions…
If we have made an important assumption, we need to validate if it’s true. As an example we were convinced users wanted email login. We were able to disprove this in a couple of hours, rather then spending 3 months building it to only then find out nobody wants it…
The most valuable thing to experiment on (biggest questions to be answered) are high-risk, high-value. If we can prove or disprove these, we are getting the greatest learning…
The second question. The answer is rarely “let’s build a feature, release it, and see what happens”
Low fidelity evidence for your question can be easily obtained – just ask people in the street, for instance. If the first 100 people dislike your idea, you’ve validated it’s probably wrong quite cheaply. The better the quality of the evidence, the more expensive it is.
Whatever you do, tho’, get out of the pink area – just talking and planning in the office gives you no new learnings…
The whole team needs to be involved in discovery and delivery so that everyone has all the learnings on which to base their decisions – design, coding, etc. – devs can speak to customers (seriously!) and why can’t PO’s be involved in testing or even pair program?
The quickest way to get something out the door is co-creation; PO, UX and devs sit together and build it between them – no wire frames, no pixel perfect specs, etc.
So this is the Lean UC landscape, and how things like “Just Do It”, e.g. compliance and regulatory stuff – fits in.
Culture is a big part of Lean. Here we see developer, PO and UX designer sat together to co-create a feature…
We used Workplace by Facebook as a collaboration tool as it kind of blurs the social and work aspects; this led to lots of sharing of articles, etc. and lots of collaboration across teams…
We implemented an andon cord so we can all down tools and fix things that could affect our speed round the cycle, e.g. the tests are red, or the delivery pipeline is broken. Anybody can “pull the cord” without comeback; proper psychological safety
Collaboration across the whole team is key in this approach, so team-building is critical. Here we see people doing a treasure hunt we organised for all the teams. They had to take photos that represented random words we gave them, So…
Top right – “Seal”
Bottom Right – “Far Away”
Left – Also “Far Away” – this is so far away from what he normally wears
We set up Universigaff to bring in inspirational speakers to help us change people’s mindset to work in this new way. Including Bruce Daisley from Facebook….
So, we did a 3 step journey…
Re-boot our delivery capability; move to (twice) daily delivery
Re-boot the way we build products; Lean UX and data-driven product discovery
Re-boot the culture and mindset; instil psychological safety and the experimental mindset – “it’s OK to fail (if you learn)”
Low fidelity evidence for your question can be easily obtained – just ask people in the street, for instance. If the first 100 people dislike your idea, you’ve validated it’s probably wrong quite cheaply. The better the quality of the evidence, the more expensive it is.
Whatever you do, tho’, get out of the pink area – just talking and planning in the office gives you no new learnings…
So after a few months, we reviewed this. The number of experiments should be the opposite of the truth curve, i.e. more experiments at the cheaper side of the graph. As you can see this wasn’t happening – proportionally too many (expensive) A/B tests. If we check the results of those tests, we can see that only 11% were successful – there must be cheaper ways of finding these things out…
We discovered there was a continuum between “Just Do It” tasks where there is very little scope for experimentation (Right To Be Forgotten (RTBF) has only one key result – “did we delete everybody’s data and avoided a fine?”, whereas email login was ripe for experimentation; we actually discovered we *shouldn’t* build it after experimenting around it, even though we were sure it was wanted…
The New Community Platform was, we thought, a JDI as we were simply re-platforming it. However, as we worked through, we realized we didn’t need to re-build all features, and we could experiment and validate assumptions. Learning: Not all JDI is JDI!
Be careful of “Data-Driven Delay” syndrome – doing lots of good discovery then just building the feature you were going to build in the first place, in the same way you were going to build it! This is not giving you the experimental approach benefit it’s just slowing you down…
How do we know if we’re successful? Meeting our OKRs, obviously, but if everything w do is a success, how do we know we are building the best thing we could, rather than sticking to safe bets? Would we be more successful if we delivered, in the same timeframe
10 features that each gave a 1% uplift, or
10 features where 9 failed, and the 10th gave us 25% uplift
We should be set up for failure, not success. #controversial
