1. Ritu Khare1, Jiao Li2, Zhiyong Lu1
1National Center for Biotechnology Information (NCBI), U. S. National Library of Medicine, NIH
2Institute of Medical Information, ChineseAcademy of Medical Sciences
Toward Creating a Gold Standard of Drug Indications
from FDA Drug Labels

2. Presentation Order
2
1. Motivation
2. Materials and Methods
3. Results
4. Discussion

3. Drug Disease Treatment Relationships
 Which drug(s) are approved for
treating which diseases(s)
 Most frequently sought
information among clinicians
(Ely et al. 2000)
 Among top 10 most frequent
multi-concept queries on
PubMed (Dogan et al. 2009)
 Applications
 Google Knowledge Graph
(quick referencing)
 Training biomedical systems
(Lu et al. 2013, Li and Lu 2012)
 Controlling errors in EMRs
(Khare et al. 2013)
3
Disease 1
Disease 2
Disease 3
Disease
Drug Indications
(e.g. What are the indicated uses of
Fluoxetine capsule)
Disease Treatments
(e.g. What are the prescribed drugs
for hypertension)

4. Gold Standard
Properties
1. Factual
2. Structured and Normalized
3. Specific to a dose-form
 e.g. ear drop, oral tablet, topical gel,…
 Ketorolac injection and Ketorolac ophthalmic solution have different indications
Existing resources
- DrugBank (University of Alberta), MedicineNet (WebMD), DailyMed (National Library of Medicine
/ FDA)
- Factual, Specific (Not Structured)
- NDF-RT (U.S. Dept ofVeteran Affairs), Freebase (Google)
- Factual, Structured (Not Specific)
4
Disease 1
Disease 2
Disease 3
Drug 1
Drug 2
Drug 3
RXCUI
RXCUI
RXCUI
UMLS CUI
UMLS CUI
UMLS CUI

5. DailyMed: The Drug Indication Data Store
 Drug Database of the National Library of Medicine (NLM)
 Most recent drug labels (or packet inserts) submitted to FDA by various
pharmaceutical companies.
5
Factual
Structured
Dose Form Specific


X

6. Identify Indications from Drug Labels
The Challenges
6
Drug Indication Excerpts in DailyMed
d1 Dutasteride capsules are indicated for the treatment of symptomatic benign
prostatic hyperplasia.
Dutasteride is not approved for the prevention of prostate cancer.
d2 Ranitidine is indicated in the treatment of GERD. Concomitant antacids
should be given for pain relief to patients with GERD.
d3 In patients with coronary heart disease, but with multiple risk factors for
coronary heart disease such as retinopathy, albuminuria, smoking, or
hypertension.
contraindication
other drug’s
risk factors

7. Related Studies with DailyMed Indications
1. Neveol and Lu (2010)
 SemRep (tool for identifying relationships)
 1,263 ingredients
 73% accuracy
2. Wei et al. (2013)
 Use SIDER2 (based on DailyMed)
 1,554 ingredients
 67% accuracy
3. Fung et al. (2013)
 MetaMap (biomedical concept recognizer)
 2,105 Drugs (ingredient + dose form)
 77% accuracy (on 295 drugs)
7
- Biomedical Text
Mining Tools (65-
80% accuracy)
- Expert Annotation

8. Presentation Order
8
1. Motivation
2. Materials and Methods
3. Results
4. Discussion

9. DailyMed: Dataset
 Downloaded:August 24 2012 version
 Multiple drug labels for same drug by different
manufacturers.
 Clustered drug labels using RxNorm identifiers
 Determined a representative drug label for each
cluster
 Frequently sought drugs
 303 ingredients are most frequently sought (80%
access) on PubMed Health (query logs 2010-2011)
 Top Drugs: Clonazepam(Klonopin),Acetaminophen
(Tylenol),Azithromycin (Zithromax), …
9
18,353 human prescription drug labels
2,497 unique drug labels
(based on RxNorm identifiers)
504 frequent drug labels
100 drug labels
(randomly selected)

10. Mining Indications from Drug Labels
1. Automatically identify the candidate indications from drug
labels
2. Display the drug labels with pre-computed candidate
indications, i.e., preannotations (Neveol et al. 2011) on an
annotation interface
3. Two expert annotators accept/reject the preannotations
 Educational background: medical and library sciences
 Training: Biomedical Literature Indexing
10

11. Method:
Identify Disease Mention from Drug Labels
 UMLS-based disease lexicon
 Seed concepts: UMLS CUIs
 Vocabulary: MeSH, SNOMED-CT
 Semantic Types: 12 types belonging to “Disorder” semantic group
 Terms:
 Removed: acronyms, abbreviations, fully specified names, and stop words.
 Included: all English language non-suppressed synonyms, and their normalized
strings (NLM’s normalization tool NORM)
 Extracting disease mentions from Drug Labels
 Tokenized, lengths 1 -6.
 All tokens and their normalized versions matched with lexicon terms
 Overlapping mentions, e.g.“arthritis” and “rheumatoid arthritis,” resolved
by choosing the more specific (longer) match
11

12. Annotation Interface
12

13. Annotation Workflow: Two Rounds
 Round 1
 Pre-annotations = All Disease Mentions
 A1 and A2 independently perform annotations
 Round 2
 Pre-annotations (color-coded) = (i) exclusive judgments (ii) pre-annotations from round-1 not
selected by either
 A1 and A2 independently improve previous annotations.
13

14. Annotation Workflow
Sets and Guidelines
Sets Preliminary Guidelines
 100 Drug Labels
 50 drug labels at a time
 Set-1 (avg. 126 words/drug label)
 Set 2 (avg. 249 words/ drug label)
 Annotation Order
 Set-1 round-1
 Error analysis/Update Guideline
 Set-1 round-2
 Error analysis/Update Guideline
 Set-2 round-1
 Error analysis/Update Guideline
 Set-2 round-2
 Error analysis/Update Guideline
14

15. Preliminary Annotation Guidelines Examples
 What to Annotate
 Select all types of indications (treatment, relief, prevent,…)
 What NOT to Annotate
 Do not select medical procedures
15

16. Evaluation
Ground Truth Evaluation
 Ground truth for the 100-
drug label dataset.
 Three study investigators
 Reviewed drug labels
 Derived the indicated usages and
the UMLS concepts.
 Consulted NDF-RT and PubMed
Health
 Total 461 ground truth
indications
1. Pre-annotation Performance
 Precision, Recall
2. Annotator Performance
 Common judgments (Both
annotators agree)
 Joint performance
 Recall, Precision, F1-measure
 Inter-annotator Agreement (Jaccard)
 (num_match/num_match+num_
nonmatch)
16

17. Presentation Order
17
1. Motivation
2. Materials and Methods
3. Results
4. Discussion

18. Pre-annotation Quality
850 Pre-annotations (UMLS-CUIs) for 100 drug labels
Precision Recall
 51.88%
 Remaining disease mentions:
 Contraindications
 This drug should not be used for treating
type I diabetes.
 Part of organization’s name
 The Advisory Council for the Elimination
ofTuberculosis, theAmericanThoracic
Society, …
 Characteristics of an indication
 A major depressive episode implies a
prominent and relatively persistent
depressed or dysphoric mood that usually
interferes with daily functioning
 Symptoms, Organism names, Risk factors of
…
 95.67%
 Missed cases
 natural language challenges
 Identifying “skin infections” from “skin and
soft tissue infections”
 limitations of the lexicon
 the concepts “tick fever” and “pylori
infection” were not included.
18

19. Judgment(Expert Annotation) Assessment
Number of Drug Labels and Duration
Joint Performance
 A nearly perfect joint precision
 Avg. 7.5% improvement in F1-measure
improved from round-1 to round-2.
 Inter-annotator agreement
 Set-1: 76.2%
 Set-2: 93.9%
19
Round-1
#Drug Labels
Round-2
# Drug Labels
Avg.Total
Time
/Annotator
Set-1 50 22 124 min
Set-2 50 28 173 min

20. Error Analysis
Set 1
Set 2
 Missed Indications
 Alprazolam is also indicated for the
treatment of panic disorder,with or
without agoraphobia
 Incorrect Judgments
 Selecting Symptoms
 Panic disorder is characterized by
following symptoms:palpitations,
pounding heart,or accelerated
heart rate …
 Selecting Indications of other Drugs
 Cimetidine hydrochloride injection is
indicated for the short term treatment of
active duodenal ulcer.Concomitant
antacids should be given as needed for
relief of pain.
 Missed Indications
 Drug labels were long upto 800
words
 Incorrect judgments
 Selecting species names
 Respiratory tract infections caused by
Streptococcus pneumoniae
 Selecting conditions (e.g. sedation)
caused by the drug.
20

21. Updated Guidelines
What NOT to Annotate What To Annotate
1. Contraindications
2. Indicated Usages of Another Drug
3. Disease mentions part of an
organization’s name
4. Explicitly specified symptoms
5. Species or organism names
6. Medical procedures
7. Characteristics of an indication
8. Risk factors
1. All indicated usages
2. All types of indications (treat,
prevent, manage, relief…)
3. Main and associated indications
4. Indication treated by a combination
of drugs
5. Efficacy established in clinical trials
21
Special Cases of Annotation
1. Causing Disease
2. Optional Indication
3. In patients with a disease

22. Updated Guidelines
Special Cases (Need Domain Knowledge)
1. Causing Indication
 Hydroxyzine Hydrochloride: Useful in the management of pruritus due to
allergic conditions such as chronic urticaria and atopic and contact
dermatoses
 Diclofenac Epolamine: Flector Patch is indicated for the topical treatment of
acute pain due to minor strains, sprains, and contusions
2. Optional Indication
 Fluoxetine Hydrochloride:Acute treatment of Panic Disorder, with or without agoraphobia, in
adult patients
 Alprazolam:Alprazolam is also indicated for the treatment of panic disorder, with or without
agoraphobia.
3. In patients /adults with a Disease
 Azithromycin :Azithromycin tablet is indicated for the prevention of disseminated Mycobacterium
avium complex (MAC) disease in persons with advanced HIV infection.
 KEPRA: KEPPRA XR™ is indicated as adjunctive therapy in the treatment of partial onset
seizures in patients ≥16 years of age with epilepsy.
22

23. Presentation Order
23
1. Motivation
2. Materials and Methods
3. Results
4. Discussion

24. Conclusions
 Semi-automatic method (NLP +Annotation by two experts)
 Toward factual, structured, specific gold standard
 A promising performance, joint judgments as gold
 Avg. 3 min/drug label by each annotator
 F1-measure = avg. 0.95
 First study involving annotation of drug indications
 Specific and detailed indication annotation guidelines.
 What toAnnotate,What Not to …, Special Cases
 Challenges
 About half disease mentions (pre-annotations) not indications
 Long textual drug labels
 Special Cases of Annotation
24

25. Limitations and Future Work
 Framework
 Pre-process drug labels for
improved presentation and
summarization
 Algorithm for preparing
pre-annotations needs
sophisticated text mining
techniques (e.g. MetaMap,
NegEx)
 Evaluation
 Different pair(s) of annotators
 Compare gold standard with
existing resources/studies
 Classification ability of annotated
corpus
 Current Status
 534 unique drug labels curated (~
7,688 drug labels)
 272 Frequently Sought
Ingredients
25

26. Acknowledgments
 Grant
 Intramural Research Program of the NIH, National Library of
Medicine
 Two Human Annotators
 Zanmei Li
 Yujing Ji
 BiomedicalText Mining Group at NCBI
 Robert Leaman
 Yuqing Mao
 Chih-HsuanWei
26