Final project for the "DL in NLP" course by ipavlov. Based on Kaggle competition https://www.kaggle.com/c/gendered-pronoun-resolution. Paper & code https://github.com/Yorko/gender-unbiased_BERT-based_pronoun_resolution

1. Gender-unbiased BERT-based
Pronoun Resolution
Yury Kashnitsky
@yorko in ODS Slack
@festline in Telegram
Final project for the “DL in NLP” course

2. Gender-unbiased BERT-based
Pronoun Resolution
Pronoun resolution. Example:
John entered the room and saw [A] Julia. [Pronoun] She was talking to
[B] Mary Hendriks and looked so extremely gorgeous that John was
stunned and couldn’t say a word.
HuggingFace demo
2

3. Gender-unbiased BERT-based
Pronoun Resolution
Pronoun resolution. Easy?
“Roxanne, a poet who now lives in France. Isabel believes that she is there
to help Roxanne during her pregnancy with her toddler infant, but later
realizes that her father and step-mother sent her there so that Roxanne
would help the shiftless Isabel gain some direction in life. Shortly after
[Pronoun] she arrives, [A] Roxanne confides in [B] Isabel that her French
husband, Claude-Henri has left her.”
HuggingFace demo
3

4. Gender-unbiased BERT-based
Pronoun Resolution
4

5. Gender-unbiased BERT-based
Pronoun Resolution
5
- test: 2000 rows
- development: 2000 rows
- validation: 454 rows
Gender-balanced
M – F1 score for Masculine B – Bias (F/M)
F – F1 score for Feminine O – Overall F1 score

6. Gender-unbiased BERT-based
Pronoun Resolution
6
Competition metric

7. Gender-unbiased BERT-based
Pronoun Resolution
7
Our contributions:
1. Beated Google AI GAP results
with a large margin
2. Showed that our BERT-based
model is gender-unbiased

8. Gender-unbiased BERT-based
Pronoun Resolution
8
Solution steps:
1. Extracting BERT-embeddings for
named entities A, B, and pronouns
2. Fine-tuning BERT classifier
3. Hand-crafted features
4. Neural network architectures
5. Correcting mislabeled instances

9. Gender-unbiased BERT-based
Pronoun Resolution
9
BERT-embeddings + MLP
Then masking and -4,-5,-6 layers
1. Extracting BERT-embeddings for
named entities A, B, and pronouns
Fine-tuning BERT classifier
Hand-crafted features
Neural network architectures
Correcting mislabeled instances

10. Gender-unbiased BERT-based
Pronoun Resolution
10
The key idea: average predictions of many
models. At least, to minimize log-loss
Extracting BERT-embeddings for
named entities A, B, and pronouns
2. Fine-tuning BERT classifier
Hand-crafted features
Neural network architectures
Correcting mislabeled instances

11. Gender-unbiased BERT-based
Pronoun Resolution
11
Extracting BERT-embeddings for
named entities A, B, and pronouns
Fine-tuning BERT classifier
3. Hand-crafted features
Neural network architectures
Correcting mislabeled instances

12. Gender-unbiased BERT-based
Pronoun Resolution
12
Several MLPs + Siamese networks
Separate for Cased and Uncased BERT models
Extracting BERT-embeddings for
named entities A, B, and pronouns
Fine-tuning BERT classifier
Hand-crafted features
4. Neural network architectures
Correcting mislabeled instances

13. Gender-unbiased BERT-based
Pronoun Resolution
13
Not a huge impact on log-loss
But influenced gender bias removal a lot
Extracting BERT-embeddings for
named entities A, B, and pronouns
Fine-tuning BERT classifier
Hand-crafted features
Neural network architectures
5. Correcting mislabeled instances

14. Gender-unbiased BERT-based
Pronoun Resolution
14

15. Gender-unbiased BERT-based
Pronoun Resolution
15
Main results:
1. High F1 scores (92% vs. Google AI’s 74%)
2. Almost no gender bias

16. Gender-unbiased BERT-based
Pronoun Resolution
16
Accepted to ACL 2019 workshop
on gender bias in Machine Learning

17. Participate in Research competitions!
Yury Kashnitsky
@yorko in ODS Slack
@festline in Telegram