"What Scares Me About AI" By Rachel Thomas, Co-founder of fast.ai & Professor at the University of San Francisco

1. What Scares Me
About AI
Rachel Thomas
fast.ai & USF Data Institute
@math_rachel

2. What Scares Me About AI
• Algorithms are often implemented without ways to address
mistakes.
• AI makes it easier to not feel responsible.
• AI encodes & magnifies bias.
• Optimizing metrics above all else leads to negative outcomes.
• There is no accountability for big tech companies.
@math_rachel

4. @math_rachel

5. Algorithms are used differently than human
decision makers:
• Algorithms are more likely to be implemented with no appeals
process in place.
• Algorithms are often used at scale.
• Algorithmic systems are cheap.
• People are more likely to assume algorithms are objective or error-
free (even if they’re given the option of a human override)
@math_rachel
The privileged are processed by people; the poor are processed by
algorithms. (Cathy O’Neil, Weapons of Math Destruction)

8. @math_rachel

9. “I’m just an engineer.”

10. Bureaucracy has often been used to shift/evade responsibility (who do
you hold responsible in a complex system?)
Today’s algorithmic systems are extending bureaucracy.

12. Joy Buolamwini & Timnit Gebru

13. @math_rachel

16. Runaway Feedback Loops
“YouTube may be one of the most
powerful radicalizing instruments of
the 21st century.”
- Zeynep Tufekci, New York Times

17. Guillaume Chaslot

19. Choosing NOT to just maximize a metric

21. "That’s not 20/20 hindsight. The
scale of this problem was
significant and it was already
apparent."

23. Unenforced USA regulations
• Age Discrimination in Employment
Act (1967)
• Fair Housing Act (1968)
@math_rachel

24. Early cars:
• sharp metal knobs on dashboard that
could lodge in people’s skulls in crash
• non-collapsible steering columns would
frequently impale drivers
• belief that cars were dangerous
because of the people driving them

25. What Scares Me About AI
• Algorithms are implemented without ways to address mistakes.
• AI makes it easier to not feel responsible.
• AI encodes & magnifies bias.
• Optimizing metrics above all else leads to negative outcomes.
• There is no accountability for big tech companies.
@math_rachel

26. How We Can Do Better
• Make sure there is a meaningful, human appeals process.
Plan for how to catch and address mistakes in advance.
• Take responsibility, even when our work is just one part of
the system.
• Be on the lookout for bias. Create datasheets for data sets.
• Choose not to just optimize metrics.
• Push for standards and regulations for the tech industry.
@math_rachel

27. Resources
• Zeynep Tufekci: “How social media took us from Tahrir Square to
Donald Trump”
• Renee DiResta: “Up next: a better recommendation system”
• Timnit Gebru: “Datasheets for Datasets”
• Latanya Sweeney: “Saving Humanity”
• Arvind Narayanan: “21 Definitions of Fairness”
• Kate Crawford: “Politics of AI”
• danah boyd: “How an algorithmic world can be undermined”
• Joy Buolamwini: gendershades.org

31. course.fast.ai

32. Questions?
Rachel Thomas
@math_rachel
data science blog: fast.ai
https://medium.com/@racheltho