How to best manage your data to make the most of it for your research - With ODAM Framework (Open Data for Access and Mining) Give an open access to your data and make them ready to be mined
1. Daniel Jacob – INRA - 2018
How to best manage your data
to make the most of it for your research
Make your data great now
Give an open access to your data
and make them ready to be mined
Open Data for Access and Mining
ODAM Framework
Daniel Jacob
INRA UMR 1332 BFP – Metabolism Group
Bordeaux Metabolomics Facility
Oct 2018
2. Daniel Jacob – INRA - 2018
DATA Studies
Project
During a research project
Know-how Knowledge
Input Output
3. Daniel Jacob – INRA - 2018
What do they become?
• Nothing ! They rest on a disk space (up to its death!)
Among the possible scenarios, two of them are extreme
• Creation of a comprehensive database managing all
data and metadata in its entirety, associated with a
visualization and querying interface.
Expected objectives
After the project is completed
DATA Studies
Project
4. Daniel Jacob – INRA - 2018
Expected objectives
Scientific Data Repositories
Enrichment
Expected links
DATA Studies
Project
Publishing policies
http://www.omicsdi.org/ …
5. Daniel Jacob – INRA - 2018
Raw Data
Processed
data
Analyzed
data
Published
data
Processed data is the raw
data processed in a way so
that they can highlight
some features, i.e. some
type of variables linked to
the focus of the study.
Raw data
Information
(often partial)
Open access
Specialized Data Repositories
Data flow Concerned data
Experiment
Data Tables
Know-how Knowledge
Annotation, Curation, Validation
Partly not automatically
reproducible because it
requires human expertise
6. Daniel Jacob – INRA - 2018
Open Data
Accessible data
including incomprehensible and
unexploitable documents by
automatons (programmatic way)
Open API
Queryable data
according to the imposed API scheme
Metadata
Application Program Interface
PUBLISH DATA "5 STARS" THE FAIR DATA PRINCIPLES
7. Daniel Jacob – INRA - 2018
Open Data
Accessible data
including incomprehensible and
unexploitable documents by
automatons (programmatic way)
Open API
Queryable data
according to the imposed API scheme
Metadata
Application Program Interface
PUBLISH DATA "5 STARS" THE FAIR DATA PRINCIPLES
Data capture Using data
The more simple as possible and in a normalized way
As far as possible, the most appropriate choice seems to keep the old way of using
the scientist's worksheets,
while allowing other more efficient approaches and this, throughout the data flow
Producers
are scientists
Consumers
are scientists
8. Daniel Jacob – INRA - 2018
Before the project begins
After the project is completed
Future
Data flow
DATA
A data management plan or DMP is a formal
document that outlines how data are to be handled
both during a research project, and after the project is
completed.
The goal of a data management plan is to consider the
many aspects of data management, metadata
generation, data preservation, and analysis before the
project begins
this ensures that data are well-managed in the
present, and prepared for preservation in the future.
Description by metadata (Ontology / Controlled Vocabulary)
Data capturing, data formatting, data linking
Implying data archaeology after several months / years
https://dmp.opidor.fr/
https://www6.inra.fr/datapartage/
Data management
Publishing policies
9. Daniel Jacob – INRA - 2018
During a research project
Know-how
Knowledge
Project
Data flow
DATA
Data / Metadata
Data Mining Modeling
Find out
“biomarkers”
Explain data
Data/Metadata Exploration
Data Visualization
Data mining / Modeling
Data exploration Descriptive statistics
First glimpse of the data that can show trends.
Allow the data to be well characterized, which is
necessary to then choose how to analyze them.
Repetition of multiple scenarios on
different subsets of data
Selection subsets of data
Implying lots of data manipulation
data capturing, data formatting, data linking,
data import / data export
Linking both metadata and data for data
mining
Data processing
10. Daniel Jacob – INRA - 2018
Before the project begins
After the project is completed
Future
During a research project
Know-how
KnowledgeProject
Data flow
2
1 2
1
Data management
Time is clearly explicit
Data processing
Time is often implicit
Reduce data manipulation
Data sharing & data availability
Facilitate the subsequent data mining
Facilitate the data dissemination
Make consistent the two axes :
Motivations
How ?
DATA
The "data life" must therefore be integrated
into the scientific research process
11. Daniel Jacob – INRA - 2018
seeding harvesting
samples
preparation
samples analysis
Sample
identifiers
Experiment
Data Tables
Experiment Design
Make both metadata and data
available for data mining
Several operators,
technics, data
types, SOPs, …
Each time we plan to share data coming from a common experimental
design, the classical challenges for fast using data by every partner are data
storage and data access
Several partners
Use-Case
“Metabolism”
Research question Project Experiment Experimental set-up
During a research project
Plant Metabolism
• Systems Biology
• Biomarkers
associated with plant
performance
12. Daniel Jacob – INRA - 2018
Whatever the kind of experiment, this assumes
a design of experiment (DoE) involving
individuals, samples or whatever things, as the
main objects of study (e.g. plants, tissues,
bacteria, …)
This also assumes the observation of dependent
variables resulting of effects of some controlled
experiment factors.
Moreover, the objects of study have usually an
identifier for each of them, and the variables
can be quantitative or qualitative.
Promote good practices
samples : Sample features
Data capture The experimental context: needs / wishesseeding harvesting
samples
preparation
Sample
identifiers
Experiment Design (DoE)
samples analysis
Use-Case “Metabolism”
identifier factors Quantitative Qualitative
Data
Promote non-proprietary
format like CSV or TSV
13. Daniel Jacob – INRA - 2018
Promote good practices
samples : Sample features
Data capture The experimental context: needs / wishesseeding harvesting
samples
preparation
Sample
identifiers
Experiment Design (DoE)
samples analysis
Use-Case “Metabolism”
Shortname Description Unit
SampleID Pool of several harvests Identifier
Treatment Treatment applied on plants Factor
DevStage fruit development stage Factor
FruitAge fruit age Days post-anthesis (dpa) Factor
FruitDiameter Fruit diameter mm Variable
FruitHeight Fruit height mm Variable
FruitFW Fruit Fresh Weight(g) g Variable
Rank Row of the invidual plant on the table Feature
Truss Position on the stem of the truss Feature
Description of the different
columns within data files
Metadata
Data
Promote non-proprietary
format like CSV or TSV
14. Daniel Jacob – INRA - 2018
Experiment
Data Tables
Data storage
drag & drop
No database schema, no programming code and no additional configuration on the server side.
Data capture Minimal effort (PUT)
Merely dropping data files in a data
storage (e.g. a local NAS or distant
storage space)
PUT
Data capture Using Data
The "core idea"
(See Good Practices)
Data center
mount
Data can be downloaded,
explored and mined
Data analysis / mining
Maximum efficiency (GET)
http://myhost.org/
GET
The more simple as possible and in a normalized way
15. Daniel Jacob – INRA - 2018
Experiment
Data Tables
Data storage
+2 metadata files
drag & drop
No database schema, no programming code and no additional configuration on the server side.
Data capture Minimal effort (PUT)
Merely dropping data files in a data
storage (e.g. a local NAS or distant
storage space)
Web API
Data center
mount
Data can be downloaded,
explored and mined
Data analysis / mining
Maximum efficiency (GET)
http://myhost.org/
GET
PUT
Data capture Using Data
EDTMS
Experiment Data Tables Management System
(EDTMS)
Implementation
F
A
INTEROPERABLE
R
16. Daniel Jacob – INRA - 2018
s_subsets.tsv This metadata file allows to associate a key concept to each data subset file EDTMS
Metadata files
In order to allow data to be explored and mined, we have to adjoin some minimal but relevant metadata:
It must exist a relationship between object types that we assume of “obtainedFrom" type.
To linked together two tables, it implies a common attribute, i.e. an identifier in most case.
Optional(*)
(*) in fact, rather deferred
17. Daniel Jacob – INRA - 2018
a_attributes.tsv This metadata file allows each attribute (variable) to be annotated with
some minimal but relevant metadata
factor
quantitative
qualitative
identifier
categories
EDTMS
Metadata files
In order to allow data to be explored and mined, we have to adjoin some minimal but relevant metadata:
Plants
Harvests
Samples
Compounds
…
…
Optional (*)
Good Practices
Description of the different columns within data files
(*) in fact, rather deferred
18. Daniel Jacob – INRA - 2018
s_subsets.tsv
a_attributes.tsv
…
…
EDTMS
Time
Data
Make consistent both data flow
Additional subsets can be added step by step, as soon as data are produced.
Metadata files some minimal but relevant metadata
19. Daniel Jacob – INRA - 2018
Using DataFRIM1 Fruit Integrative Modelling http://www.erasysbio.net/index.php?index=266
Dataset example
20. Daniel Jacob – INRA - 2018
Using DataFRIM1 Fruit Integrative Modelling
21. Daniel Jacob – INRA - 2018
Using DataFRIM1 Fruit Integrative Modelling
22. Daniel Jacob – INRA - 2018
Metadata files
In order to allow data to be explored and
mined, we have to adjoin some minimal
but relevant metadata
Using DataFRIM1 Fruit Integrative Modelling
23. Daniel Jacob – INRA - 2018
Using Data
http://myhost.org/
Application
Programming
Interface
F
A
INTEROPERABLE
R
EDTMS
GET http://myhost.org/getdata/<data format>/<dataset name>/< … >/< … >
REST Services: hierarchical tree of resource naming (URL)
Metadata files
In order to allow data to be explored and
mined, we have to adjoin some minimal
but relevant metadata
http://pmb-bordeaux.fr/odamsw/
Data Emancipation
24. Daniel Jacob – INRA - 2018
plants samples activome qNMR_metabo
Identifiers
https://pmb-bordeaux.fr/getdata/xml/frim1/(activome,qNMR_metabo)/treatment/Control
Get of data subsets by merging all the subsets with lower rank than the specified
subsets and following the pathway defined by the “obtainedFrom" links.
Avoids lots of data
manipulation
Facilitates linking both
metadata and data for
data mining (activome,qNMR_metabo) plants + samples
+ (aliquots+activome)
+ (pools+qNMR_metabo)
FRIM1
Application
Programming
Interface
EDTMS
http://pmb-bordeaux.fr/odamsw/
25. Daniel Jacob – INRA - 2018
Metadata files
In order to allow data to be explored and
mined, we have to adjoin some minimal
but relevant metadata
Using Data
F
A
INTEROPERABLE
R
EDTMS
Develop if needed, lightweight tools
- R scripts (Galaxy), lightweight GUI
(R shiny)
Tools
Data emancipation
regarding Tools
Data API Tools
Data
http://myhost.org/
Application
Programming
Interface
Data Emancipation
26. Daniel Jacob – INRA - 2018
https://pmb-bordeaux.fr/dataexplorer/?ds=frim1
FRIM1 Example online
R shiny
Visual data exploration
a first key step for deeper analyses
https://pmb-bordeaux.fr/dataexplorer/
Using Data
EDTMS
27. Daniel Jacob – INRA - 2018
FRIM1
Metadata files
In order to allow data to be explored and
mined, we have to adjoin some minimal
but relevant metadata
Using Data
EDTMS
32. Daniel Jacob – INRA - 2018
FRIM1
Export as …
Using Data
As far as possible,
keep the old way of
using the scientist's
worksheets …
33. Daniel Jacob – INRA - 2018
The R package
Rodam
Copy-Paste
The Comprehensive R
Archive Network
https://cran.r-project.org
FRIM1 Using Data
… while allowing
a way to be more
efficient ...
34. Daniel Jacob – INRA - 2018
The R package
Rodam
Selection subsets of data
Repetition of multiple scenarios on
different subsets of data
Data mining / Modeling https://cran.r-project.org/web/packages/Rodam/index.html
The Comprehensive R Archive Network Using Data
35. Daniel Jacob – INRA - 2018
R markdown
knitr
Reproducible Research … with R and RStudio
R code
https://rmarkdown.rstudio.com/authoring_quick_tour.html
The R package
Rodam
EDTMS
ODAM Framework
36. Daniel Jacob – INRA - 2018
Reproducible Research … with R and RStudio
“How you gather your data directly impacts how reproducible your research will be.
If all of your data gathering steps are tied together by your source code, then independent
researchers (and you) can more easily regather the data“
II. 6 - Gathering Data with R
II. 7 - Preparing Data for Analysis
“Once we have gathered the raw data that we want to include in our statistical analyses
we generally need to clean it up so that it can be merged into a single data file.”
https://englianhu.files.wordpress.com/2016/01/reproducible-research-with-r-and-studio-2nd-edition.pdf
This is exactly what the ODAM framework aims to answer in
a normalized way the easier and faster as possible
Chap II Data Gathering and Storage (70 pages out of 300)
Christopher Gandrud (2015)
https://github.com/christophergandrud/Rep-Res-Book
37. Daniel Jacob – INRA - 2018
https://pmb-bordeaux.fr/dataexplorer/?ds=frim1
doi:10.15454/95JUTK
https://data.inra.fr/
FINDABLE
A
I
R
Data Dissemination
R scripts (Rmd) If applicable
38. Daniel Jacob – INRA - 2018
Data as the subject of a paper
Data Paper
The Data Paper describes the data
It includes the associated descriptive elements (metadata),
and all the technical information (methods, formulas, software
applications ...) useful to the understanding of the obtaining of
data and their reuse by other scientists
https://www6.inra.fr/datapartage/Partager-Publier/Valoriser-ses-donnees/Publier-un-Data-Paper
This tool allows you to generate a draft data paper (scientific publication describing
a dataset) from the DOI of a dataset deposited in the data.inra.fr portal
A draft data paper
https://data.inra.fr/datapartage-datapapers-web/
39. Daniel Jacob – INRA - 2018
Make your data great now
All the actors in the data acquisition chain must be convinced that
the data repository can bring added value, all the more so as
producers will do so as soon as possible, i.e.:
• Reduce data manipulation
• Data sharing & data availability
• Facilitate the subsequent data mining
• Facilitate the reproducible research
• Facilitate the data dissemination
• Assistance with decision-making in the selection of samples, in
the choice of additional analyses,
• Assistance with annotation by cross-referencing, a priori
knowledge input,
• etc.
Make consistent the two axes
Data processing
Data management
This implies that (bio)computer scientists
• propose useful and/or innovative tools
• to motivate and convince researchers to submit
their data as soon as possible.
The data management system becomes
completely independent of data usage.
One dataset Several applications
&
One application Several datasets
The "data life" must therefore be integrated into
the scientific research process
40. Daniel Jacob – INRA - 2018
Need to take care of data
Take-home message
Thank you to your attention
Have a good fun !!
Open Data for Access and Mining
https://hub.docker.com/r/odam/getdata/
http://pmb-bordeaux.fr/dataexplorer/
https://github.com/INRA/ODAM
https://cran.r-project.org/package=Rodam
http://pmb-bordeaux.fr/odamsw/
Notes de l'éditeur
Présentation à la fois d’un travail mais aussi d’une réflexion autour de la gestion de la données
À savoir comment en tirer le meilleur parti à toute les étapes de leur cycle de vie.
Schématisation d’un projet du point des données concernant les Entrées / Sorties
Time is more often implicit
Time is clearly explicit
Time is more often implicit
Time is clearly explicit
As far as possible, keep the old way of using the scientist's worksheets,
https://englianhu.files.wordpress.com/2016/01/reproducible-research-with-r-and-studio-2nd-edition.pdf
Chap 6 - Gathering Data with R (p109, PDF p138)
How you gather your data directly impacts how reproducible your research will be. You should try your best to document every step of your data gathering process. Reproduction will be easier if your documentation (especially variable descriptions and source code) makes it easy for you and others to understand what you have done. If all of your data gathering steps are tied together by your source code, then independent researchers (and you) can more easily regather the data. Regathering data will be easiest if running your code allows you to get all the way back to the raw data files –the rawer the better.
Chap 7 - Preparing Data for Analysis (p129, PDF p158)
Once we have gathered the raw data that we want to include in our statistical analyses we generally need to clean it up so that it can be merged into a single data file.