By Sonia Koesterer The world is imperfect. Every “happy path” intersects with dozens of crappy paths caused by typos, technical errors, and data that goes missing, is mis-assigned, adulterated, or is otherwise compromised/ stolen by evil data pirates. While you can’t prevent all data fails, you can avoid catastrophic failures, design graceful recoveries, and even turn the weakest points of your service into a strategic advantage. In short, you can create great services despite bad data. The impact of data failure can be a humorous accident, minor inconvenience, or completely detrimental. For example, each year, the U.S. government falsely declares over 12,000 people dead due mostly to typos. In sheer percentage this is a rarity of a corner case of an edge case… but for those 12,000 individuals who suddenly lose their social security benefits, health insurance, bank accounts, and can’t easily prove they are alive, it’s catastrophic. So design for the the edge-case! Understand the weakest points of your service, learn from them, and turn your failures into great experiences.

1. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 1 1 0
0 1 1 1 0 0
1 0 1 0 0 1
0 1 0 0 1 1
0 0 1 0 1 0
1 0 0 1 0 1
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 0 1 1
0 0 1 0 1 0
1 0 0 1 0 1
1
0
1
1
1
0
1
1
0
1
0
1
1
1
0
1
1
0
NOT DEAD YET…
Designing Great Experiences
with Bad Data
@ SONIA KOESTERER

2. PART ONE
Why are we here?

3. PERSONAL BACKGROUND
Why does a designer
want to talk to you
about data?

4. SHOW OF HANDS
How many of you
design happy paths?

5. HAPPY PATH (NOUN)
A default positive scenario,
which demonstrates the
sequence of activities executed
if everything goes as expected.

6. I MADE A SWITCH
Consulting Healthcare

8. QUICK OVERVIEW
What exactly does
Collective Health do,
anyway?

9. Traditional healthcare is
disconnected.

10. We connect all the pieces.

11. Live Concierge ServiceWeb Applications Mobile Apps
And help you make sense of
your health benefits.

12. MY AWAKENING
HEALTHCARE
DATA IS F CKED*

13. THE PROBLEM
HEALTHCARE $ =
1/5 FINANCIAL $

14. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

15. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 0 0 1 1 0 1 0 1
0 1 1 0 1 0 0 0 1 1 1 0
1 1 0 1 0 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

16. PART TWO
What does data
failure mean for
design?

17. DATA FAILURE
Erodes trust & can
be dangerous.

18. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

19. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 1 0 0 1
0 1 0 0 1 0 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

20. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

21. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

22. 1
0
0
1
1
0
1
0
1
1
10
0
1 0 1 0 0 1 0
1
1
1
0
0 1 0 0 1 0 1 0 1
1
1

23. When you say
edge case, you're
just defining the
boundaries of what
you care about.
BETH DEAN

24. We need a process for
understanding & fixing
failures in design.
Solution
Problem Space
FAIL
FAIL
FAIL
FAIL

25. Why not borrow concepts
from Manufacturing,
Software Engineering,
Systems Engineering &
Safety Engineering?
Solution
Problem Space
FAIL
FAIL FAIL
FAIL

26. Overflow

27. PART THREE
How do we get
there?

28. REMEMBER
Data doesn’t
know it has
a problem.
1
1
1
0
0
0

29. EXAMPLE ONE

30. Imagine FBI agents
turn up at your home
and forcefully
declare it to be the
site of illicit activities…

31. FRAUDF R AU D
Imagine FBI agents
turn up at your home
and forcefully
declare it to be the
site of illicit activities…

32. FRAUDF R AU D
A quiet Kansas
home wound up
with 600-million
IP addresses and
a world of trouble.
KASHMIR HILL, FUSION.NET

33. 1 0 0
1
0
1
1 0 1 0 1
1
0 1 1
00
1
1
0
1 1 1
0
0 1
1
0
1 1 0 1 0 1
0 1 1 0
1
0
0
1 0
1
1
0
0
1 1 1
01
1
0 1 0
0
1
1
0 1 01 1
1
0 0 1
0
1 0
0
1 0 1 0 1
1
1
0
0
1 0
1
1
0
0
1
0
0
1 0
1
0
0
1 0 1
0
1 1 0
0 1
0
1 1
0
0
1 1 1
0
0
1 0 1
0
0 1
0
10 0
1
1
0 0
1
0 1 0
1 0
0
1 0 1
1
1
0 1 0 0 1 1 0 1 0 1
1
1 0 0 1
0 1
0
0
1
0 1 0
1
1
1
0
0
1 0 1 1
0 0 1 0 0 1
0
1 0 0 1 01 0 1 1 0
0 1 0
0
1 1
0 0 1 0 1 0
1
0
0 1 0 1
1
0
1
1
1
0
1
10
1
0
1
1
1
0
1
1
0

34. 1 0 0
1
0
1
1 0 1 0 1
1
0 1 1
00
1
1
0
1 1 1
0
0 1
1
0
1 1 0 1 0 1
0 1 1 0
1
0
0
1 0
1
1
0
0
1 1 1
01
1
0 1 0
0
1
1
0 1 01 1
1
0 0 1
0
1 0
0
1 0 1 0 1
1
1
0
0
1 0
1
1
0
0
1
0
0
1 0
1
0
0
1 0 1
0
1 1 0
0 1
0
1 1
0
0
1 1 1
0
0
1 0 1
0
0 1
0
10 0
1
1
0 0
1
0 1 0
1 0
0
1 0 1
1
1
0 1 0 0 1 1 0 1 0 1
1
1 0 0 1
0 1
0
0
1
0 1 0
1
1
1
0
0
1 0 1 1
0 0 1 0 0 1
0
1 0 0 1 01 0 1 1 0
0 1 0
0
1 1
0 0 1 0 1 0
1
0
0 1 0 1
1
0
1
1
1
0
1
10
1
0
1
1
1
0
1
1
0
PROBLEM ONE
Data is messy.
Sloppy data, uneven data,
varying levels of precision

35. WHAT IF WE COULD
Learn from our
mistakes?

36. Experiment at a
small scale before
launching at scale.

37. LEAK BEFORE FAILURE

38. *Not a healthy motto for life outside of work.
LEAK BEFORE FAILURE
BE LEAN, BUT
FIRST GET FAT.

39. EXAMPLE TWO

40. Imagine going to the
hospital to have your
tonsils removed, but
coming back with a
leg operation.

41. Imagine going to the
hospital to have your
tonsils removed, but
coming back with a
leg operation.

42. Hospitals and
medical practices
mix up 8% of
patient identities.
JOHN MCQUAID, STATNEWS.COM

43. 1 1 0 1 1 0 1 0 0 1 1 0 0
1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 1 1 0 0 1 1 1 0
1 1 0 0 1 1 0 1 0 1 1 1 1
0 1 0 0 1 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0
0 1 0 1 1 0
1 0 0
1 0 1 0 0 1
0 1 0 0 1
0 0 1 0 1 0
1 0 0 1 1
1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 1 1 0 0 1 0 1
0 0 1 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 0 1 1
0 0 1 0 1 0
1 0 0
0
1
1
1
0
1
1
1
0
1
1
0
1
1

44. 1 1 0 1 1 0 1 0 0 1 1 0 0
1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1
0 1 1 0 1 0 0 1 1 1 0 0 1 1 1 0
1 1 0 0 1 1 0 1 0 1 1 1 1
0 1 0 0 1 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0
0 1 0 1 1 0
1 0 0
1 0 1 0 0 1
0 1 0 0 1
0 0 1 0 1 0
1 0 0 1 1
1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 1 1 0 0 1 0 1
0 0 1 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 0 1 1
0 0 1 0 1 0
1 0 0
0
1
1
1
0
1
1
1
0
1
1
0
1
1
PROBLEM TWO
Data is incomplete.
Blank fields, disappearing feeds,
and uncollected information

45. WHAT IF WE COULD
Just say no?

46. Design forcing
functions to get the
data you need.

47. POKA–YOKE
KNOW YOUR
ANTI-USE CASE.

48. EXAMPLE THREE

49. Imagine you come
home to find that your
house is being treated
as a playground…

50. Imagine you come
home to find that your
house is being treated
as a playground…

51. Man’s home
mistakenly set
as a Pokémon
GO gym.
MARCUS LESHOCK, WGNTV
BOON SHERIDAN @BOONERANG

52. 1 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 1 1 1 0 1 1 0 1 0 1
0 1 1 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 1 1 0
0 1 1 1 0
1 0 1 0 0 1
0 1 0 0 1 1
0 0 1 0 1 0
1 0 1 0 1
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 1 0 1 1
0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
1 0 0 1 1
0 0 1 0 1 0
1 0 0 1 0 1
1
0
1
1
1
0
1
1
0
0 0
0
0
0
0
0
0
0
0
1
0
1
1
1
0
1
1
0
0
1
1 1
0
1
1
1
1
1
1
1

53. 0 1 0 1 0 1 1 0 1 1 0 0
1 1 0 1 1 0 1 1 0 1 0 1
0 1 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 1 1 0 0 1
0 1 0 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 1 1
0 1 1 1
1 0 1 0 0 1
0 1 0 0 1 1
0 0 1 0
1 0 1
1 1 0 1 0 0 1 1 0 1 0 1 0 0
0 0 1 0 1 0 1 1 1 1 0 1 1
1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
1 0 0 1 1
0 0 1 0 1 0
1 0 0 1 0
0
1
1
1
0
1
0
0 0
0
0
0
0
0
0
0
0
1
0
1
1
1
0
1
0
0
1
1 1
0
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
1
1
1
1
1
1
0
0 0
0
0
0
0
0
0 0
1
1 1
0
1
1
1
1
1 1
0
0
0
0
0
0
0
0
0
0
1
1
0
1
1
1
1
1
1
1
PROBLEM THREE
Data is fickle.
Temporary data, old data,
information no longer valid

54. WHAT IF WE COULD
Measure twice,
and cut once?

55. Design the means
to capture data from
redundant sources.

56. REDUNDANCY
GET AHEAD
OF CHANGES.

57. EXAMPLE FOUR

58. Imagine going to the
doctor for a cough,
and being told they
can’t help you because
you’re dead…

59. Imagine going to the
doctor for a cough,
and being told they
can’t help you because
you’re dead…
deceased

60. deceasedOver 12,000
Americans are
erroneously
declared dead
each year.
MARISSA FESSENDEN, SMITHSONIAN.COM

61. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 5 1 1 0 0 1 1 0 1 1 0 C A T
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 2 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 1 1 2
0 1 1 1 0 0
1 0 1 0 0 1
0 1 0 0 9 1
0 0 1 0 1 0
1 0 0 1 0 1
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 8 0 1 0 1 0 1 1 1 0 2 1 0 1 1
0 0 1 0 0 W T F 0 0 1 0 1 0 1 1 0
C A T 0 1 1
0 0 1 0 1 2
1 0 0 1 0 1
3
0
1
1
1
0
1
1
0
4
0
1
1
1
0
1
1
8

62. 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0
1 1 0 5 1 1 0 0 1 1 0 1 1 0 C A T
0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0
1 1 0 1 0 0 1 1 0 1 0 2 1 1 0 0 1
0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1
0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0
0 1 0 1 1 2
0 1 1 1 0 0
1 0 1 0 0 1
0 1 0 0 9 1
0 0 1 0 1 0
1 0 0 1 0 1
1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1
0 1 8 0 1 0 1 0 1 1 1 0 2 1 0 1 1
0 0 1 0 0 W T F 0 0 1 0 1 0 1 1 0
C A T 0 1 1
0 0 1 0 1 2
1 0 0 1 0 1
3
0
1
1
1
0
1
1
0
4
0
1
1
1
0
1
1
8
PROBLEM FOUR
Data is imperfect.
Typos, duplications, bad
formatting, oh my!

63. WHAT IF WE COULD
Trust but verify?

64. Design feedback
loops to prevent
harmful mistakes.

65. SUCCESSIVE CHECKS
DO THE
RIGHT THING.

66. IN SUMMARY
What are the big ideas
you can take home?

67. 1. First be fat.
Pilot new offerings at a small scale to learn where the
weak points are and gracefully recover.
2. Know your anti-use case.
Define what you don’t want your service to enable, and
create forcing functions to prevent disastrous mistakes.
3. Get ahead of changes.
Future-proof your service by continuously gathering
updated data from multiple sources.
4. Do the right thing.
Verify the validity of key transitions where major status,
account, or behavior changes occur for your users.

68. INTERESTING
THOUGHTS
FROM SMART
PEOPLE
1. Fat Thinking and
Economies of Variety
Venkatesh Rao
2. Go Home, Data, You’re
Drunk
Danielle Malik
3. Goodhart’s Law and Why
Measurement is Hard
David Manheim

69. If you torture the
data long enough,
it will confess.
RONALD H. COASE
Thank you.
@SoniaKoesterer