This document discusses opportunities for collaboration between researchers working in systematic reviews and electronic discovery (e-discovery). It notes similarities in the challenges both fields face, including the need for high recall with bounded costs and reliance on multi-stage review pipelines. The document proposes that technologies developed for semi-automated citation screening and crowdsourcing could help address current limitations. It concludes by encouraging information retrieval researchers to investigate open problems in systematic reviews as opportunities to advance technologies beyond other tasks and help bring together interested parties through forums like the TREC Total Recall track.

1. Systematic Review is e-Discovery
in Doctor’s Clothing
Joint work with
Matt Lease
ir.ischool.utexas.edu
slideshare.net/mattlease
@mattlease
ml@utexas.edu
Gordon V. Cormack (U. Waterloo) An Thanh Nguyen (U. Texas)
Thomas A. Trikalinos (Brown U.) Byron C. Wallace (U. Texas)

2. “The place where people & technology meet”
~ Wobbrock et al., 2009
www.ischools.org
2

3. • System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
3Matt Lease <ml@utexas.edu>
Roadmap

4. • System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
4Matt Lease <ml@utexas.edu>
Roadmap

5. Evidence-Based Medicine n.
The conscientious, explicit and judicious
use of current best evidence in making
decisions about the care of
individual patients
5

6. Systematic reviews: from biomedical
articles to actionable evidence
6

7. PubMed
?
2 search database
1 formulate question,
protocol & query
4 extract data
treatment
outcome
ba
c d
3 screen retrieved citations
Studies
AIMS1988
ASSET1988
Aber1976
Amery1969
Anderson1983
Bassand1986
Bett1973
Bossaert1987
Brunelli1988
Buchalter1987
Croydon1987
Dewar1963
Durand1987
ECSG−11979
ECSG−21988
EWP1971
Fletcher1959
GISSI1986
Gormsen1973
Guerci1987
Heikinheim1971
ISAM1986
ISISPilot1987
ISIS−21988
Ikram1986
Julian1987
Khaja1983
Leiboff1984
Maublant1988
Meinertz1988
NHFAustra1988
Olson1986
Raizner1985
Rentrop1984
Sainsous1986
Schreiber1986
Simoons1985
TICO1988
Topol1987
WWICSK1983
WWIVSK1988
White1987
Overall (I^2=19% , P=0.147)
0 0.01 0.02 0.04 0.08 0.190.270.38 0.76 1.91 3.82 7.65 18.26
OddsRatio(logscale)
5 synthesize extracted data 7
Formulate RQ &
Boolean Query
Boolean Search
Document Collection
All Tasks but #2 done
manually by MDs

8. On average, 75 articles describing results from
clinical trials are published every day.
Bastian, PLoS Med, 2010
The median length to complete a single review: 1110
person-hours.
Allen & Olkin, JAMA, 1998
8

12. 12
Technologies for semi-automated
citation screening are relatively mature
and slowly gaining acceptance

13. Research on citation screening
• Methods for handling imbalance with asymmetric costs [ICDM
2011; ICDM 2012; KAIS 2013]
• Active learning strategies [KDD 2010; SDM 2011; KDD 2013;]
– Nguyen, Wallace, and Lease. Combining Crowd and Expert
Labels using Decision Theoretic Active Learning. HCOMP 2015.
• Test Collection: github.com/bwallace/crowd-sourced-ebm
• Dually supervised methods [ICML 2011; KDD 2010]
13

14. • System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
14Matt Lease <ml@utexas.edu>
Roadmap

15. PubMed
?
2 search database
1 formulate question,
protocol & query
4 extract data
treatment
outcome
ba
c d
3 screen retrieved citations
Studies
AIMS1988
ASSET1988
Aber1976
Amery1969
Anderson1983
Bassand1986
Bett1973
Bossaert1987
Brunelli1988
Buchalter1987
Croydon1987
Dewar1963
Durand1987
ECSG−11979
ECSG−21988
EWP1971
Fletcher1959
GISSI1986
Gormsen1973
Guerci1987
Heikinheim1971
ISAM1986
ISISPilot1987
ISIS−21988
Ikram1986
Julian1987
Khaja1983
Leiboff1984
Maublant1988
Meinertz1988
NHFAustra1988
Olson1986
Raizner1985
Rentrop1984
Sainsous1986
Schreiber1986
Simoons1985
TICO1988
Topol1987
WWICSK1983
WWIVSK1988
White1987
Overall (I^2=19% , P=0.147)
0 0.01 0.02 0.04 0.08 0.190.270.38 0.76 1.91 3.82 7.65 18.26
OddsRatio(logscale)
5 synthesize extracted data 15
Request for
Production (RFP):
Boolean Query
Review Documents for
“Responsiveness” Parties use documents
Review Responsive
Documents for Privilege
Boolean Search
Document Collection
Electronically Stored Information (ESI)
e.g., Enron email archive

16. Manual Review does not Scale
16
Paul, George L., and Jason R. Baron.
Information inflation: Can the legal
system adapt? Rich. JL & Tech. 13 (2007).

17. IR Research in e-Discovery
• NIST TREC Track: 2006-2011
• Oard & Webber, FnTIR Book, 2013
• A variety of published work at SIGIR++
– e.g., Cormack & Grossman, SIGIR 2016
17

18. • System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
18Matt Lease <ml@utexas.edu>
Roadmap

19. Commonalities
• Need high-recall with bounded cost
• Follow 3-Stage Pipeline Today
– Boolean query
– Screening (traditionally manual by experts)
– Final review & use
• Pipeline approach useful but limits improvement
– overall framing & unrecoverable errors
• Limiting reliance on experts
– Traditionally assumed to be infallible 19

20. Can we crowdsource screening?
Michael Mortenson, Byron C. Wallace, Gaelen Adam, Tom Trikalinos and Tim Kraska.
Crowdsourcing Citation Screening for Systematic Reviews. (Under review).
20

21. 21

22. Total Recall: Applications
22
E-Discovery

23. Total Recall: Strategies
23

24. Conclusion
• Systematic Review & e-Discovery have much in common,
but SR has received relatively little attention in IR
– Open problems & current assumptions give IR researchers
fertile opportunities for research beyond other IR tasks
– Public test collections available for both
• github.com/bwallace/crowd-sourced-ebm
• Aaron Cohen’s: http://skynet.ohsu.edu/~cohenaa/systematic-drug-
class-review-data.html
– Reading list: https://github.com/bwallace/automating-ebm-
resources/wiki/Papers
• TREC Total Recall Track (trec-total-recall.org) offers a
great forum for bringing together those interested
24

25. Thank You!
ir.ischool.utexas.eduSlides: www.slideshare.net/mattlease
25