2. Automated Decision Making: Pros
Handles large volumes of data (Google search, airline
reservations, online markets, ..)
Avoids certain kinds of bias
Parole judges being more lenient after a meal
Making hiring decisions based on the name of the person
Subjectivity in evaluations of papers, music, teaching, etc.
Human judgment in NYC stop and frisk policy
4.4 M were stopped between 2004-2012
88% of them led to no further action
83% of the people stopped were Black or Hispanic – only
about half in the population are.
2
3. Complex and Opaque Decisions
Hard to understand and make sense of
Values, biases and potential discrimination built in
The code is opaque and often trade secret
Facebook’s newsfeed algorithm, recidivism algorithms, genetic testing
3
4. Gatekeeping Function
Decide what gets attention, what is
published, and what is censored
Google’s search results of geopolitical
queries might depend on location, e.g.,
different maps of Pakistan or India.
Learning algorithms that make hiring
decisions.
Pattern: Low commute time favors
low turnover
Policy: Don’t hire from far off places
with bad public transportation
Impact: People from poor and far off
neighborhoods may not be hired
4
5. Subjective Decision Making
Algorithms to understand and translate language, drive cars, pilot
planes, and diagnose diseases.
No right answer, but judgment and values.
Detecting and removing terrorist content on the social networks.
The definition of important words such as `terrorist’ and ‘extreme
content’ are controversial
The scale makes it difficult for manual intervention.
Algorithmic decisions may not be as good as people
5
6. Machine Learning
Programs might be using protected attributes such as race and
gender to make predictions
Even if the protected attributes are not used, they could be using
other “proxy” attributes which will have the same effect, e.g., zip
code.
Recommendations based on earlier actions might create
bubbles, eg. Detecting trends on Twitter.
Example: Predictive policing
Predicting the neighborhoods most likely to be involved in
future crime based on crime statistics
Rational but may be indistinguishable from racial profiling
More police in the neighborhood lead to more arrests.
Could lead to positive feedback loops and become a self-
fulfilling prophecy.
6
7. Data Privacy
Who owns your browser data?
Can your insurance company get access to your grocery list or
peek into your fridge?
Can hospitals get access to consumer data to predict who is
going to get sick?
Can your employer access your grades?
7
8. Transparency and Notification
If the algorithm is opaque, there is no understanding or trust in
the program, e.g., medical decisions, hiring decisions
Google’s search algorithm judged not demonstrably anti-
competitive in the US
European Commission has successfully pursued an anti-
trust investigation
Many points of trust: algorithm, input, learning data, control
surfaces, assumptions and models the algorithm uses, etc.
Complete transparency makes it vulnerable to hacking. Does
not guarantee scrutiny.
Consumers might demand the right to be notified when using
their information or demand excluding their personal information
8
9. Algorithmic Accountability
How search engines censor violent/sexual search terms
What influences Facebook’s newsfeed program or Google’s
advertisements
Need causal explanations that link our digital experience with
data they are based upon
9
10. Government Regulation
Destabilizing effect of high-speed trading systems led to
demands of transparency of these algorithms and ability to
modify them
Should search algorithms be forced to follow some “search
neutrality rules”?
Requires public officials to have access to the program and
modify it in the interest of public.
There is no one right answer to the queries Google handles,
which makes it difficult.
10
11. Case Study: Recidivism Assessment
COMPAS is a program to assess the recidivism of the prisoners –
their propensity to commit a crime in 3 years after the release.
Propublica analyzed data of 10,000 prisoners in a Florida county
There is one such table for Blacks and another for Whites. Θ is
chosen for each group separately.
False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
Positive Predictive Value 𝑃𝑃𝑉 =
𝑇𝑃
(𝐹𝑃+𝑇𝑃)
Propublica: FPR(Blacks) = 2 FPR(Whites)
NorthPointe: PPV(Blacks) = PPV(Whites)
11
Recidivism Score ≤ θ Score > θ
False TN FP
True FN TP
12. Conflicting Demands on Fairness
12
Red = False positives, FP; Blue= True positives TP
Assumptions:
Prevalence or rate of recidivism is higher for one group (say blacks)
Positive Predictive Value 𝑃𝑃𝑉 =
𝑇𝑃
(𝐹𝑃+𝑇𝑃)
= same for both
False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
= higher for blacks.
White Black
Recidivism
=True
Prediction = Positive
13. Fairness of Recidivism Scores
Recidivism LowScore HighScore
False TN FP
True FN TP
13
False Positive Rate 𝐹𝑃𝑅 =
𝐹𝑃
(𝐹𝑃+𝑇𝑁)
False Negative Rate 𝐹𝑁𝑅 =
𝐹𝑁
(𝐹𝑁+𝑇𝑃)
Prevalence p =
(𝐹𝑁+𝑇𝑃)
(𝐹𝑁+𝐹𝑃+𝑇𝑁+𝑇𝑃)
𝐹𝑃𝑅 =
𝑝
1 − 𝑝
1 − 𝑃𝑃𝑉
𝑃𝑃𝑉
(1 − FNR)
Conclusion: If the prevalence p is different for two classes and
PPVs are the same then FNR or FPR or both must be different.
The differences in FPR and FNR lead to disparate impacts –
more penalty for Blacks in both recidivism groups than Whites.
14. Summary
It is mathematically impossible to achieve both equal PPV and
equal FPR across different groups.
The differences in FPR and FNR persist in subgroups of
defendants.
However, evidence suggests that data-driven risk assessment
tools (in medicine) are more accurate than human judgment.
Human driven decisions are themselves prone to exhibiting
racial bias, eg, paroles, sentencing, stop and frisk, arrests, etc.
14
15. Case Study: Online Market Places
How do we ensure that the sellers are honest about the quality of their
goods?
Study: In early 2000’s eBay merchants misrepresented the quality
of their sports trading cards
Problem largely solved by the feedback and reputation systems
New development: demand for more information
Study (2012): Subjects rated trustworthiness of potential borrowers
from photographs of them.
People who looked trustworthy are more likely to get loans
They are also more likely to repay their loans.
More information leads to more freedom
People can now choose whom to do business with based on looks
A growing body of evidence suggests this leads to discrimination
15
16. Discrimination in Online Markets
Air-BnB Study: 20 profiles sent to 6400 hosts
The profiles are identical except 10 of them have names common to
white people and the rest to blacks
Result: Requests for black-sounding names were 16% less
successful
Discrimination was pervasive. Most of the people who rejected
never hosted a black guest.
Other areas of discrimination: credit, labor markets, housing.
Discrimination also occurs in algorithmic decisions.
Searches for black sounding names on Google were more likely to
bring up ads about arrest records.
Why?
Learning from the past search data.
16
17. Principles and Recommendations
Don’t Ignore potential discrimination
Collect good data including race and gender stats
Do regular reports and occasional audits
Public disclosure of discrimination-related data
Keep an experimental mindset to evaluate different design options
Airbnb withholding host pictures from its ads
17
18. Design Decisions
Control the information, its timing and salience
When can you see the picture of Uber driver?
Increase automation and charge for control
Make instant book the default on AirBnB and charge a fee if the
host wants to approve the guest first
Prioritize discrimination issues
Remind the host about anti-discrimination policies at the time of
the transaction
Make algorithms discrimination-aware
Set explicit objectives: want my black and white customers to
be rejected at the same rate
18
19. Virtual Screens
In mid 60’s less than 10% of the big 5 orchestras were women
Moved away from face-to-face to behind-the-screen auditions
Success rate of female musicians increased by 160%
The online market allows virtual screens between buyers and sellers,
between employers and employees.
19
20. Case Study: Gerrymandering
20
Background
In the US, states are divided into
congressional districts every 10 years
Each state is divided into precincts of
equal population
The precincts are clustered into
congressional districts
Whoever wins the majority of precincts
in the district wins that district
Gerrymandering (named after Elbridge
Gerry) refers to manipulation of districts to
influence the outcome of an election.
Packing: Pack most of the voters in the
opposing side into a small number of
districts
Cracking: Split the voters of the
opposing side into several districts
where they are minority The original political cartoon on
Gerrymandered map of Essex
County Massachusetts, 1812
21. Impact of gerrymandering
Racial gerrymandering that intentionally reduces minority
representation was ruled illegal in 1960.
In 1980, voting rights act was amended to make states redraw
maps if they had racially discriminatory impact.
Partisan gerrymandering has not been ruled illegal
When republicans drew the maps (17 states) they won about 53 percent of
the vote and 72 percent of the seats.
When democrats drew the maps (6 states), they won about 56 percent of
the vote and 71 percent of the seats.
Proportional representation: Each party receives roughly the
same percent of votes as it wins the percent of the seats
Wasted votes: Votes cast to the losing side or above the
minimum the winner needed to win.
Efficiency gap: The difference in the wasted votes / total wasted.
It is intended to measure partisan bias.
21
22. Wisconsin’s redistricting in 2011
22
Wisconsin’s Republican-led redistricting was struck
down by a 3 judge panel. It was heard by the supreme
court on October 3. A decision is pending.
The arguments of the plaintiffs:
Big efficiency gap indicates bias especially if it is
persistent. Wisconsin’s gap is the biggest ever.
It violates voters’ right to equal treatment
It discriminates against their views (first
amendment argument)
Arguments of the defendants:
Efficiency gaps arise naturally, e.g., when
democrats pack into cities
Courts should stay out of it. States can appoint
independent commissions if they are concerned
Justice Kennedy’s vote is probably going to be
decisive.
23. Discussion
Suppose you are heading an independent commission to
recommend a fair redistricting approach.
How do you define fair redistricting? Why?
How would you go about implementing your recommendation?
What role do computer algorithms play?
23
