YHEC presented on Improving search efficiency for economic evaluations in major databases using semantic technology at the HTAi conference in Dublin in 2010

1. Improving search efficiency
for economic evaluations in
major databases using
semantic technology
Julie Glanville, Carol Lefebvre, Pamela
Negosanti, Bill Porter
jmg1@york.ac.uk
Oct 2010

2. Overview
 Why are we interested in economic evaluations?
 Can economic evaluations be identified efficiently at
present?
 This research project
 Methods
 Results
 Discussion
 Next steps

3. Why are we interested in
economic evaluations?
 Systematic reviews and technology assessments frequently
consider cost-effectiveness as well as effectiveness outcomes
 This information is published in economic evaluations
 Cost-effectiveness analyses
 Cost-utility analyses
 Cost-benefit analyses
 Issues in identifying reports of economic evaluations
 Poor reporting
 abstracts may contain terms which signal an economic evaluation but not an
explicit term
 Economics is often mentioned in passing in abstracts
 Increases number of irrelevant records retrieved

4. Can economic evaluations be
identified efficiently?
 In healthcare databases
 Yes and No
 Specific economic evaluation databases are available (NHS
EED and HEED)
 BUT may need to carry out top up/supplementary searches
in large bibliographic databases
 Beyond healthcare
 Seem to be no economic evaluation databases
 Need to search large bibliographic databases such as ERIC
and Criminal Justice Abstracts

5. What about search filters?

6. Can search filters help?
 In healthcare databases
 Many search filters
 search filters to find economic evaluations in EMBASE and
MEDLINE achieve high sensitivity (100%) (1)
 BUT they have poor precision (less than 4%): very high proportion
of irrelevant studies are retrieved (1)
 Beyond health
 Few filters available
 Issues of precision likely to be similar to health
(1)Glanville J, Kaunelis D,Mensinkai S.How well do search filtersperform in identifying
economic evaluations in MEDLINE and EMBASE. Int J Tech Assess Hlth Care
2009;25:522-529

7. This research project
 How can we improve efficiency of retrieval of
economic evaluations in large bibliographic
databases?
 Traditional Boolean approaches don’t seem to be
helping
 Indexing isn’t very helpful at present
 Can semantic analysis software help?
 Collaboration with Expert System to explore potential
for identifying economic evaluations using their
Cogito software

8. Semantic Net

9. Semantic analysis
Analysis hat assigns a meaning, a sense, to a syntactic
structure and consequently to a linguistic unit, according
to the knowledge contained in the semantic network.

10. Methods
 Gold standard set of 1950 economic evaluation records
(published 2000, 2003, 2006)
 identified from NHS EED and then downloaded from MEDLINE.
 Comparator set of 4136 matching MEDLINE records for the 3
years (2000, 2003, 2006)
 not economic evaluations
 But identified using the NHS EED filter
 Loaded into Cogito
 Divided randomly into test sets and validation sets
 Used in-built semantic analysis and also created new rules to
categorise economic evaluations to categorise records as
economic evaluations or non-economic evaluations

11. Testing and validation

12. Results
Test set
(Gold Standard
records=975)
(Comparator records =
2068)
Validation set
(Gold Standard
records=975)
(Comparator records =
2068)
Number of gold standard (GS)
records retrieved 975 975
Number of comparator records
retrieved 203 385
Sensitivity
(number GS retrieved/number of
GS records) 100% 100%
Precision
(number of GS retrieved/number
of records retrieved) 82.77% 71.69%

13. Results, 2
Precision
(combined Test and
Validation sets)
Sensitivity
(combined Test and
Validation sets)
Using Cogito in-built
semantic rules (no filter) 77.23% 100%
Using filter with records
scoring 50
78%
90%
Using filter with records
scoring 100 80% 85%
Using filter with records
scoring 200 81% 83%

14. Discussion
 Cogito performs as well as Boolean searching in terms of
sensitivity
 Cogito has a much improved precision score compared to
performance of Boolean filters
 Over 70% (Cogito) compared to under 10% (Glanville et al)
 Cogito performs well ‘out of the box’
 Although early training efforts did not improve precision, further
exploration might yield improved results

15. Next steps
 Identifying funding to carry out further exploration
 Exploring economic evaluation identification optimisation further
 Exploring the effects of importing results from a range of databases into
Cogito
 Exploring whether semantic analysis has potential to achieve
improvements in retrieval of other hard to find research where filters do
not perform well
 diagnostic test accuracy studies and quality of life research
 Exploring the potential of semantic analysis for analysing records
by study design obtained from a range of databases in healthcare,
social care, education and criminal justice contexts
 in-built rules are database independent.

16. For further information
Julie Glanville, York Health Economics
Consortium
jmg1@york.ac.uk
Bill Porter at Expert System
http://www.expertsystem.net/
bporter@expertsystem.net