Proper application of machine learning to accurately evaluate the accuracy of biomarkers is challenging and error-prone for those without expertise in machine learning or programming. We offer an easy to use tool which implements the best practices and produces a comprehensive yet clinically-relevant report when comparing several biomarkers or different methods/studies. It is called neuropredict, which is open source and applicable to any domain whose biomarkers can be represented by numbers.

1. neuropredict: a proposal and a
tool towards standardized and
easy assessment of biomarkers
github.com/raamana
Pradeep Reddy Raamana, PhD
crossinvalidation.com

2. What are biomarkers?
2
[1]. Strimbu, K., & Tavel, J. A. (2010). What are Biomarkers? Current Opinion in HIV and AIDS, 5(6), 463–466.

3. What are biomarkers?
• “The term “biomarker”, a portmanteau of
“biological marker”, refers to a broad
subcategory of medical signs – that is,
objective indications of medical state
observed from outside the patient – which can
be measured accurately and reproducibly. ”1
2
[1]. Strimbu, K., & Tavel, J. A. (2010). What are Biomarkers? Current Opinion in HIV and AIDS, 5(6), 463–466.

4. What are biomarkers?
• “The term “biomarker”, a portmanteau of
“biological marker”, refers to a broad
subcategory of medical signs – that is,
objective indications of medical state
observed from outside the patient – which can
be measured accurately and reproducibly. ”1
• simplified: “set of numbers predicting label(s)”
2
[1]. Strimbu, K., & Tavel, J. A. (2010). What are Biomarkers? Current Opinion in HIV and AIDS, 5(6), 463–466.

5. What are biomarkers?
• “The term “biomarker”, a portmanteau of
“biological marker”, refers to a broad
subcategory of medical signs – that is,
objective indications of medical state
observed from outside the patient – which can
be measured accurately and reproducibly. ”1
• simplified: “set of numbers predicting label(s)”
• biomarkers are essential for computer-aided
diagnosis: 1) detection of disease and staging
their severity, and 2) monitoring response to
treatment.
2
[1]. Strimbu, K., & Tavel, J. A. (2010). What are Biomarkers? Current Opinion in HIV and AIDS, 5(6), 463–466.

6. Measuring biomarkers accuracy
is hard and error-prone!
3
• As proper application of ML requires
• training in linear algebra and statistics
• training in programming and
engineering
• It only gets harder in biomarker domain:
• blind application is not enough
• interpretability/limitations are important
• Too many black-boxes and knobs -->

7. Measuring biomarkers accuracy
is hard and error-prone!
3
• As proper application of ML requires
• training in linear algebra and statistics
• training in programming and
engineering
• It only gets harder in biomarker domain:
• blind application is not enough
• interpretability/limitations are important
• Too many black-boxes and knobs -->

8. Typical ML/biomarker workflow
4
Raw data Preproce
ssing
Feature
extraction
Cross-
validation
(CV)
Analysis
of CV
results
Visualize
and
compare

9. Typical ML/biomarker workflow
4
Raw data Preproce
ssing
Feature
extraction
Cross-
validation
(CV)
Analysis
of CV
results
Visualize
and
compare
Tools exist to do many of the small tasks individually,
but not as a whole!

10. Typical ML/biomarker workflow
4
Raw data Preproce
ssing
Feature
extraction
Cross-
validation
(CV)
Analysis
of CV
results
Visualize
and
compare
Tools exist to do many of the small tasks individually,
but not as a whole!
To those without machine learning or
programming experience, this is incredibly hard.

11. Typical ML/biomarker workflow
4
Raw data Preproce
ssing
Feature
extraction
Cross-
validation
(CV)
Analysis
of CV
results
Visualize
and
compare
neuropredict covers
these parts
Tools exist to do many of the small tasks individually,
but not as a whole!
To those without machine learning or
programming experience, this is incredibly hard.

12. neuropredict : easy and comprehensive predictive analysis

13. Accuracy distributions
neuropredict : easy and comprehensive predictive analysis

14. Confusion Matrices
Accuracy distributions
neuropredict : easy and comprehensive predictive analysis

15. Confusion Matrices
Accuracy distributions
Intuitive comparison of
misclassification rates
neuropredict : easy and comprehensive predictive analysis

16. Confusion Matrices
Feature Importance
Accuracy distributions
Intuitive comparison of
misclassification rates
neuropredict : easy and comprehensive predictive analysis

17. Billions of dollars and decades of research,
but not much insight into biomarkers!
6Woo, CW., et al.. (2017). Nature Neuroscience, 20(3), 365-377.

18. Standardized measurement
and reports are necessary!
• Research studies do not report all the
information necessary
• to assess biomarker performance
well, and
• to engage in statistical comparison
with previous studies/biomarkers
• Standardization of performance
measurement and reports is needed!
7

19. neuropredict is an attempt to
standardize and learn from each other!
8
This is NOT specific to neuroscience.
Ideas and tools are generic!

20. I have a plan
9
Consensus on
standards of
analysis
Consensus on
significance
tests!
Standardize
report format
Open
validation of
neuro-predict
Cloud repo
and web
portals
Release, test,
improve and
iterate!
but I need your support!

21. Come, join me!
let’s improve biomarker science.
one commit at a time!
10
github.com/raamana