This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019. For the document retrieval task, we adapt the Duet model to ingest a "multiple field" view of documents—we refer to the new architecture as Duet with Multiple Fields (DuetMF). A second submission combines the DuetMF model with other neural and traditional relevance estimators in a learning-to-rank framework and achieves improved performance over the DuetMF baseline. For the passage retrieval task, we submit a single run based on an ensemble of eight Duet models.

1. Duet @ TREC 2019
Deep Learning Track
B haskar Mitra , Microsof t & Univer sity College London, Canada
bmitra@microsof t.com @ UnderdogGeek
N ick C raswell, Microsof t, USA
nickcr@microsof t.com @nick_craswell

2. Motivation for participation in TREC 2019 Deep
Learning track
Enrich the document pool to improve
reusability of TREC DL dataset
Benchmark Duet on a large public dataset
Try Duet + Neural Ranking model with
Multiple Fields (NRMF) [Zamani et al., 2018]
Source: original Duet paper [Mitra et al., 2017]

3. What is the Duet
model?

4. Duet on TREC CAR

5. Adapting Duet for
MS MARCO

6. What is NRMF?

7. Duet with Multiple Fields (DuetMF)

8. Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet

9. Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet—separate
parameter set corresponding to each field

10. Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet—separate
parameter set corresponding to each field
Aggregate match vectors from each Duet
sub-model to estimate overall relevance of
document to query

11. Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet—separate
parameter set corresponding to each field
Aggregate match vectors from each Duet
sub-model to estimate overall relevance of
document to query
Structured dropout across fields and Duet
sub-models

12. Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet—separate
parameter set corresponding to each field
Aggregate match vectors from each Duet
sub-model to estimate overall relevance of
document to query
Structured dropout across fields and Duet
sub-models
Train using RankNet loss over <q, dpos, dneg>

13. Unsupervised pretraining
Randomly sample two documents dpos and dneg from the collection
Randomly pick either the title or the URL of dpos, and treat it as a
pseudo-query qpseudo
Mask corresponding field for both dpos and dneg
Compute RankNet loss over <qpseudo, dpos, dneg>

14. Summary of runs
We submitted three runs:
1. A DuetMF model for the document reranking task
2. A Learning-to-Rank model for the document retrieval task
• Candidate generation using query likelihood (QL)
• Reranking features: DuetMF, Dual Embedding Space Model (DESM), Sequential
Dependence Model (SDM), Pseudo-Relevance Feedback (PRF), Best Match (BM25), and
features based on query length and domain quality
3. An ensemble of eight Duet models for the passage reranking task
• Code: https://github.com/bmitra-msft/NDRM/blob/master/notebooks/Duet.ipynb

15. Results
ms_ensemble NDCG@10=0.578 vs. best trad run NDCG@10=0.561
ms_duet_passage NDCG@10=0.614 vs. best trad run NDCG@10=0.556

16. Ideas for TREC 2020
Deep Learning track
• Pretraining Duet on large document
collections (e.g., Wikipedia + books
corpus)
• Duet with BERT/transformer based
distributed sub-model
• Retrieval, not reranking: using query-term
independence assumption [Mitra et al.,
2019] for fullrank setting

17. Questions?