In her talk Anna shares learnings of frequently testing with users in agile teams. Also she will show that testing with users can be simple, effective & prove its value. Anna is an interaction designer & UX researcher with 9 years of experience. She likes to challenge the ‘WHY‘ of user actions. Two years ago she started the UX Lab at IceMobile to give a voice to the user. Today the UX Lab validates all concepts & products with users. She created 'Pulse UX' a simple method of frequent user testing that lets agile teams develop better products faster. Pulse UX is now part of every sprint at IceMobile.

1. Bringing the user into the
user story
Pulse UX by Anna Witteman
April 2015

2. Anna Witteman
UX Researcher
Hello :)

3. Hello, my name is Anna.
I work as a UX Researcher at IceMobile.
I used to work as an interaction designer
for over 7 years, but the last 2 years
I have fully dedicated my work to
user research, here at IceMobile
where I am head of our UX Lab.
Notes

4. Today
What I would like to talk about

5. Today I would like to share our learnings
of frequently testing with users in agile teams.
Really bringing the user into every user story.
Notes

6. Bringing the user into the user story
helped us deliver better products
faster.

7. Bringing the user into the user story has
not only made our products better,
it also creates them faster,
and strengthens our team at the same time.
Notes

8. Requirements
• Requirement 1
• Requirement 2
• Requirement 3
• Requirement N

9. Once upon a time…..
before we used to work agile we had long
lists of requirements, which were then turned
into a product.
Often it took a long time to build
and to get the product to the user.
Notes

10. ‘As a user I want to
..................................
so that I can
................................’

11. With agile the way to the user became much
shorter. Also ‘user stories’ made sure that the
focus was on the user from the start.
Still from the user story to the product the user
does not have a defined role, between the
user story and the release of the product the user
is a fictional character in the process.
This always seemed weird to me.
Notes

12. • deliver a better product, faster
• strengthen our team
• with hardly extra costs
Giving the user a place in the process, helped us

13. We found that giving the user a fixed place
into the development process does not have
to be complex
and has led to creating better products faster,
while also strengthening our team
and with hardly any extra costs.
Notes

14. Context
IceMobile

15. • Amsterdam based company
• > 110 employees
• > 25 nationalities
• Either a foodie, code king, design nerd, beer brewer or unicorn
IceMobile Amsterdam

16. Well we are here at IceMobile,
Amsterdam based company with over 110
employees from over 25 nationalities.
Nerds, unicorns, brewers - a colorful crowd.
Notes

17. the full mobile agency

18. We have been making apps for around 10 years.
We use to work like the full mobile agency creating
all kind of very different apps for big brands.
Appie for Albert Heijn, ABN Banking app
and The Voice of Holland for Talpa and
Vodafone an entertainment app.
Notes

19. the mobile agency for food retail

20. In 2012 our company merged with BrandLoyalty
a company focussed on loyalty programs for
supermarkets.
We changed our strategy to become the mobile
agency for food retail.
We now make apps for retail brands world wide.
Currently we work on apps for Denmark, China,
Taiwan and the Netherlands.
Notes

21. Products
Specialized in Food Retail
Focus on (end) Users
One-offs
All round
Focus on clients

22. For us this did not only lead to a change in the
type of apps we are making. It also means
creating a product that we can customize
to a certain brand instead of one-offs.
It also means switching from being all round to
experts of food retail. And where previously it was
mainly the brand (or our client) who would
determined when the app was ready. It is now
more our users that determine the quality of the
application.
Notes

23. UX Lab
the voice of our user

24. For me this was a clear sign that we had to learn
much more about our users, and that they should
have a clear role throughout all the development
process.
To give voice to our users, to that I started the UX
Lab.
UX Lab is not just a room in a building, it is a mindset
to give voice to the user.

26. Today we give voice to the user in many ways.
We have created ways that user research
fits our way of working, that is fits our company
Here you can see a few, clockwise…
with Pulse - bi-weekly usability testing,
diary studies using Whatsapp,
our Deep Dive tool that allows recordings in real-life
and experience mapping; to maps the different
experiences our users have with our services.

27. Problem
Why don’t you frequently test
with users?

28. But this did not happen over night..
we tried some testing with users, but it was not
working well.
I questioned:
‘Why is it that everybody knows that frequently
testing with users is important, and still so very few
companies & projects do this?’
I knew that this was not something that was only
a problem at IceMobile.
Notes

29. “ It doesn’t
fit in our
sprint”“The results
are hard to
measure”
“It feels
like extra
work”
“We were
just up
to speed.”
“It is too
soon to test
with users”
“We don’t have
time to make
a prototype.”
“Knowing the
user is not
part of my job.”
“How do I know
what I will get
out of it?”
“Where do
we get
users from?”
“How do we know
which test results
we should use?”
“We do not have
resources to do
user test(s)”

30. I started to ask around to people…
Why are you not testing frequently with users?
And what I got were very very different answers,
that touch different levels in an organization.
Notes

31. Vision
Interest
Plan
Means
Skills
Vision: Does decision-makers really have a good UX as a priority?
Or is more a buzz word?
Interest:
Vision
Interest
Plan
Means
Skills
: What is the motivation for user testing?
Improve the product or prove you are right?Interest:
How and when are we doing user testing?
Is it really a plan or more an improvisation?
What do we need to be able to start user testing?
Users? Prototype? Software? etc.
Skills:Can we conduct a user test (effective)?
Do we know how?

32. I realized that often people were having difficulties
on multiple levels, which makes it something that
is complex to tackle.
So this is a checklist we developed to recognize
where the biggest hurdles are to start test with
users.
Notes

33. Transfer information effective
The team should learn most of the user test, not just the UX researcher
Fit the user into the process
Do testing so that it is not ‘extra work’
Make results visible
Value should be clear and measurable
We wanted to find a way to:

34. Every project or company is different.
For us, these were the biggest problems we
needed to tackle:
- The person that was learning most in a user test
was the researcher and not the team.
- The testing did not fit into the sprint of the team,
so it always felt like a side project.
- The results of the tests were not visible enough.
Notes

35. our Solution
How to create a place where
great UX will happen.

36. Which practice proved its value?
Individuals and interactions
Working software
Customer collaboration
Responding to change
over
over
over
over
processes and tools
comprehensive documentation
contract negotiation
following a plan
Agile Manifesto

37. Every project or company is different. For us, to
solve this puzzle, we decided to look at
something that works well in our company…
the agile manifesto
and we looked at some of these principles to see if
we could apply these to user testing to make it work
better for us.
Notes

38. Copyright © A. Witteman & R. van den Oever, All Rights Reserved.
Setup
Process
Proof

39. This has led to ‘Pulse UX’ a way to test with users
in agile team. We call it Pulse because it frequently
checks in with the heartbeat of the product.
To see how our product is doing.
Pulse is not a perfect method. Like I mentioned,
every project has its own challenges. Pulse is just a
way to make frequent user testing work for us.
I would like to go over these 3 topics to show
you how it works.
Notes

40. Copyright © A. Witteman & R. van den Oever, All Rights Reserved.
Setup
Process
Proof

41. So first our Setup….

42. Product = Prototype
• Write a scenario
• Create a prototype
• Recruit users
• Pilot test
• Conduct Tests
• Edit videos
• Analyze video
• Write report
• Present results
User test = Report

43. Here you see the steps that I would take when setting up & conducting a test
with users. This was not effective enough since it took around two to three
weeks to do one test.
Most of the time got into building a prototype, analyzing the data and writing
a report.. the ux researcher would learn most of the test..
and when I presented it the team and PO questioned the results. Not because
they did not trust me or my research skills, but because it is very hard to
judge the results when you did not see what happens.
So this had to be more effective. In the agile manifesto they say ‘eliminate
waste’ so…
What if the test itself is the documentation, so I did not need to write a report
anymore? What if the product - the work in progress - is our prototype?
Suddenly the test is much more compact and much more effective.

45. And this is how that would look for us.
We would create 2 rooms, one for the researcher and
the user and one where the PO, the team, and the
client and other stakeholders would be.
It is no longer the researcher that notes the results.
The team, PO and client/stakeholders are responsible
for identifying what should be better and changed.
They are in charge of the outcome.

47. And here you see a snippet of a recording of one of
the sessions, with a not fully finished product.
The team can see what happens in the application,
but also see the interaction of the user,
and where she is tapping.

48. Our setup is a combination of simple toolsWhat kind of fancy testing tool is this?
less than €120,-

49. You might ask yourself what is this testing tool?
Well it actually is a combination of these simple tools.

50. Conclusion for the Setup:
• Optimized the user test:
• Give the team responsibility
• No paperwork & no overhead
• Maintain a mutual goal
• Make the team smarter over time
• Team gets motivated

51. Concluding for the Setup: We optimized the test, we made
the team and the PO responsible for the results.
This works great because they get a mutual goal.
Also the teams gets smarter over time, they learn what
matters most to users. Most important the team gets very
motivated.
It is not nice to read in a report that a feature is not working
well, but to have to see it week in, week out, it is unbearable…
so the team is highly motivated to improve the product. We
have had multiple occasions where a team stayed over time,
to fix something.

52. Copyright © A. Witteman & R. van den Oever, All Rights Reserved.
Setup
Process
Proof
Setup
Process
Proof

53. To make it work we knew that the test did not only had
to be more effective, but they also had to have a fixed
place into our process, so that it would not stay
‘something extra’.
This is how we fitted Pulse into our process.

54. Source: Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems," Proceedings of ACM INTERCHI'93 Conference (1993), pp. 206-213.
How many users should do a usability test?
0 3 6 9 12 15
Number of Test Users
100%
75%
50%
25%
0%
Percentageofusabilityproblemsfound

55. This is a graph by Nielsen & Landauer that is often
showed when people explain qualitative research, to
show that unlike with quantitive research you do not
need lots of participants for your qualitative research.
But this graph mostly explains something else,
it shows that qualitative research, testing with users is
an iterative process. It shows that it is way more
effective to frequently do small tests, then to do one or
few bigger tests.

56. iteration is the heartbeat
of agile & user testing
use it!
Copyright © A. Witteman & R. van den Oever, All Rights Reserved.

57. It says that iteration is the heartbeat of testing with
users, just like it is the heartbeat of agile.
We should use it.

58. Our renewed agile way of working
Product
Backlog
Sprint
Backlog
Potentially
Shippable
Product
2-3
weeks
24
hours
Daily Scrum
Meeting
Design
Development
Testing
User Testing

59. So what we did is this. Here you see how we used to
work…
What we did is we simply added a cycle of user
testing. Now every sprint, every two weeks, we have
a user test, and thus all sprints have users in it.

60. Iterative Testing
•Prioritize: test important part(s) most extensive
•Frequent: reliable results, also test improvements
•Empathize: learn what matters to users
•Optimize: learn to test
Allow responding to change
'I have not failed.
I've just found 10000
ways that won't work.’
Sir Thomas Edison

61. By doing our user testing iterative:
We test the most important parts best,
because that is where you start just like with agile.
We do not only test to identify problems, we also have
a chance to test our improvements.
We learn to know our users and understand what
matters to them.
And, because we do it so regularly, we do not only
improve our product but also learn how to test better.

62. • User test in every first week of the sprint
• Demo in last week of the sprint
• User test with 4 users per sprint
• 2 hours in total (4* half hour)
• Scenario about sprint (& previous sprints)
• No need to react immediately to findings
Pulse testing: Bring the user into the Sprint

63. This is how it works:
- One week the demo, the other week the user test
- We test with 4 users per sprint.
- We test in 2 hours per sprint, 30 min per user.
- A scenario of what is in sprint or the previous sprint.
- We do not respond to findings immediately.

64. Setup
Process
Proof
Setup
Process
Proof

65. So now we know when we test, and how we test..
but we also keep track of the findings.
And made results visible & proof that this way of
working leads to better products and
is faster then our previous way of working.

66. How to keep track of the findings?
# user that
did this task
# user that had
this problem
time stamps
of this issue
on video
description
of issue

67. Here is how we keep track of the findings.
We ask the PO & team to write down what they see
including the timestamp.
We collect all issues in a doc. including the number of
users that ever had this issue before, and
their names and the timestamps. So that if needed we can
easily make a compilation video of all people that have
experienced this before.
The most important issues will surface, and the 10 most
found issues, are then presented to the team & the client,
to keep focus.

68. Follow up and integrate in existing process:
1.Most important issues will surface over time
2.The most important issues will become user stories itself
3.These user stories are put in the product backlog

69. Here is how we follow up on this..
the most important issues will surface, and
become user stories themselves,
which can then be placed on the product backlog.

70. Example sprint contents
0
15
30
45
60
1 2 3 4 C 5 6 7 8 C
New Features Bugs Rework resulting from UT

71. To prove the effect of frequent user testing we kept track
of the contents of our sprints.
And we found that rework of user stories is now much
better divided over sprints, the earlier something is
improved the better and the less it costs.
But we also found that the ‘first-time-right’ of user stories
have gone up, because team members understand their
users much better.

72. Concluding.
What it comes down to.

73. So concluding….
Testing with users in agile can be simple & fast.
Notes

74. • Look at common practice within your company or project
• Eliminate waste
• Merge iterations
• Visualize progress
Take aways:

75. The take aways from all of this is that by looking closely at
the common practice in our company:
We have been able to eliminate waste and make user
testing much more effective. With just a couple of hours a
sprint we can put our users in every sprint.
We have merged the iterations of user testing and agile,
so that user testing fits our process.
And we have made results visible by showing the impact
of user testing on the speed.
Notes

76. Thank you!
Feedback? Questions?
Share experience with user testing?
I would love to talk some more about this.
anna@icemobile.com
linkedin.com/annawitteman

77. IceMobileAmsterdam @icemobile@icemobile