“Data scientist” has been called the sexiest job of the 21st century but it’s well on its way to becoming the most dangerous: the state of the art is a collection of siloed, ad hoc techniques developed to solve important and interesting challenges without a robust, principled approach. To paraphrase Michael I. Jordan, we are building dangerous, planetary-scale inference-and-decision making systems without a sufficiently stable engineering discipline, just as people built bridges and buildings, many of which fell, before there was civil engineering. In this talk, we’ll delve into how we got where we are today, what types of large-scale systems we’re building in medicine, information technology, finance, transport and society at large, and paths forward. Talk given at Data Council, NYC.

1. 11/12/2019
data scientist, or the most
dangerous job of the 21st century
hugo bowne-anderson
datacamp
@hugobowne

2. ➔ Where we are
➔ How we got here
➔ Paths forward
We need to talk about
Charlie Chaplin, Modern Times

3. ➔ Where we are
➔ How we got here
➔ Paths forward
Charlie Chaplin, Modern Times
We need to talk about

4. Illustrations you can use, just copy/pasteA new discipline of engineering

5. Illustrations you can use, just copy/pasteA new discipline of engineering

6. Illustrations you can use, just copy/pasteWhat we have: recycled intelligence
➔ “The success stories in language &
vision processing came from
scrupulously mimicking human
intelligence. No new intelligence
has been created; existing human
intelligence is merely recycled.” --
David Donoho

7. Illustrations you can use, just copy/pasteA new discipline of engineering

8. Illustrations you can use, just copy/pasteStructural failure: buckling

9. Illustrations you can use, just copy/pasteStructural failure in civil engineering
1. Structure is not strong enough to
support the load
2. Fatigue or corrosion
3. Manufacturing error or defective
materials
4. Lack of consideration of unexpected
problems

10. Illustrations you can use, just copy/pasteStructural failure in data science
1. Poor design (e.g. does not generalize as
expected, not all users considered)
2. Lack of maintenance (e.g. model drift)
3. Failure due to bad parts (e.g. training
data, assumptions, objective function)
4. Lack of consideration of unexpected
problems (e.g. bad actors such as
Cambridge Analytica)

11. Engineering medicine at scale
➔ White spots around heart possible
markers for Down Syndrome (risk 1 in
20)
➔ Amniocentisis could tell more (fatal 1
out of 300)
➔ White spots were white noise due to
increased resolution of modern
ultrasound

12. Engineering medicine at a planetary scale
➔ Measure variable/outcome
➔ (Statistical) analysis
➔ (Attempt to) generalize
➔ Decision making

13. Engineering finance at a planetary scale

14. Engineering finance at a planetary scale

15. Engineering finance at a planetary scale
➔ Targeted ads based on e-credit score

16. Engineering finance at a planetary scale

17. Engineering finance at a planetary scale

18. Engineering finance at a planetary scale

19. Technological redlining

20. Technological redlining: facial recognition

21. Engineering society at scale: WMDs
Important, secretive, and harmful algorithms that impact
➔ Financial decisions
➔ Livelihood decisions
➔ Decisions around liberty
➔ Decisions around information

22. Engineering society at scale: WMDs
➔ The Value Added Model for teachers
➔ A widespread hiring model for minimum wage
positions that had, embedded in it, a personality test
that essentially performed a mental health
assessment, which is illegal under the Americans
with Disability Act
➔ Recidivism risk algorithms in the criminal justice
system

23. Engineering society at scale: the digital poorhouse
➔ An attempt at automating the welfare eligibility
system in Indiana;
➔ A predictive model that uses data in social services
databases to determine whether or not children
are at risk of abuse and/or neglect in Pennsylvania;
➔ An algorithm used in Los Angeles to determine
who will get housing, out of the city’s tens of
thousands of homeless people

24. Engineering society at scale: the digital poorhouse
Those at highest risk
➔ had all types of personal data collected with little
regard for security or privacy with minimal upside;
➔ were classified as “risky investments” and
“problematic parents” by the relevant predictive
models;
➔ Were targets of vast networks of surveillance.

25. Engineering information at a planetary scale

26. Engineering information at a planetary scale

27. Engineering information at a planetary scale

28. Engineering information at a planetary scale

29. Engineering information at a planetary scale

30. Illustrations you can use, just copy/paste
Andrew Ng, The AI Transformation Playbook
The Virtuous Cycle of AI

31. Illustrations you can use, just copy/pasteThe Not-So Virtuous Cycle of AI

32. Illustrations you can use, just copy/pasteThe Not-So Virtuous Cycle of AI

33. ➔ Where we are
➔ How we got here
➔ Paths forward
Charlie Chaplin, Modern Times
We need to talk about

34. Illustrations you can use, just copy/pasteBuilders don’t represent users
Safiya Noble, Algorithms of Oppression

35. Illustrations you can use, just copy/pasteParadigm shift: traditional software to software 2.0
Andrej Karpathy, Software 2.0

36. Illustrations you can use, just copy/pasteParadigm shift: traditional software to software 2.0

37. Illustrations you can use, just copy/pasteBiased data & broken systems scaled
Survival of the Best Fit, an educational game about hiring bias in AI.

38. Illustrations you can use, just copy/pasteBiased data & broken systems scaled

39. Illustrations you can use, just copy/pasteOptimization, objective functions & proxies

40. Illustrations you can use, just copy/pasteOptimization, objective functions & proxies

41. Illustrations you can use, just copy/pasteOptimization, objective functions & proxies

42. Illustrations you can use, just copy/pasteOptimization, objective functions & proxies

43. Illustrations you can use, just copy/pasteOptimization, objective functions & proxies
Rachel Thomas, The problem with metrics is a big problem for AI

44. Illustrations you can use, just copy/pasteThe Efficiency Movement
Charlie Chaplin, Modern Times

45. Illustrations you can use, just copy/pasteThe Efficiency Movement

46. Illustrations you can use, just copy/pasteThe Efficiency Movement

47. Illustrations you can use, just copy/pasteShareholder value maximization & neoliberalism

48. Illustrations you can use, just copy/pasteVC-backed growth model

49. Illustrations you can use, just copy/pasteVC-backed growth model

50. Illustrations you can use, just copy/pasteCDA Section 230

51. Illustrations you can use, just copy/pasteCDA Section 230

52. Illustrations you can use, just copy/pasteModern webapp tech stack & “creepware”

53. ➔ Where we are
➔ How we got here
➔ Paths forward
Charlie Chaplin, Modern Times
We need to talk about

54. Illustrations you can use, just copy/pasteA Double Movement

55. Illustrations you can use, just copy/pasteChase the power
chris wiggins, matt jones, data: past, present and future

56. Illustrations you can use, just copy/pasteBeyond the “pipeline model”

57. Illustrations you can use, just copy/pasteEmployee walkouts

58. Illustrations you can use, just copy/pasteEmployee walkouts

59. Illustrations you can use, just copy/pasteEmployee walkouts

60. Illustrations you can use, just copy/pasteEmployee walkouts

61. Illustrations you can use, just copy/pasteCommunity-based activism

62. Illustrations you can use, just copy/pasteCommunity-based activism

63. Illustrations you can use, just copy/pasteLegislative change

64. Illustrations you can use, just copy/pasteSide note: ICP vs facial recog

65. Illustrations you can use, just copy/pasteFuture music: algorithmic audits & IRB analogues for tech

66. Illustrations you can use, just copy/pasteMore organizations to watch & work with

67. Illustrations you can use, just copy/pasteMore organizations to watch & work with

68. Illustrations you can use, just copy/pasteWhat you can do today
➔ Contribute to any of these projects
➔ Read, talk & educate widely
➔ Pressure your employers to consider
the impact of your work on all
stakeholders
➔ Pressure to move past the “pipeline
model”
➔ Define principles and design for them
in a principled, robust manner
➔ Continually ask “who am I building
these bridges for? Who is using them?”

69. Thank you!
Hugo Bowne-Anderson
@hugobowne