SlideShare une entreprise Scribd logo
1  sur  36
FAIR principles
and metrics for evaluation
1
Michel Dumontier, Ph.D.
Distinguished Professor of Data Science
@micheldumontier::#DANSLOD:2017-05-01
Principles to enhance the value of all digital resources
and their metadata.
data, images, software, web services, repositories
@micheldumontier::#DANSLOD:2017-05-012
http://www.nature.com/articles/sdata201618
@micheldumontier::#DANSLOD:2017-05-013
Rapid Adoption of Principles
Developed and
endorsed by
researchers, publishers,
funding agencies,
industry partners.
As of May 2017,
100+ citations since
2016 publication
Included in G20
communique, EOSC,
H2020, NIH, and more…
@micheldumontier::#DANSLOD:2017-05-015
Hypothesis
Improving the FAIRness of digital
resources will increase their reuse.
@micheldumontier::#DANSLOD:2017-05-016
What is FAIRness?
FAIRness reflects the extent to which a digital
resource addresses the FAIR principles as per the
expectations defined by a community of
stakeholders.
@micheldumontier::#DANSLOD:2017-05-017
How do we assess compliance to the
FAIR principles?
• Principles identify what needs to be there, but
they don’t tell what is necessary and/or
sufficient
• They also don’t tell you how to achieve FAIR
• Going beyond the principles requires some
thought about what constitutes FAIRness and
how do we measure it.
@micheldumontier::#DANSLOD:2017-05-018
Fundamental Questions
• In what ways can we assess the FAIRness of a digital
resource?
• To what degree can we automate this assessment?
• Must we treat each type of digital resource differently?
• Who will use the metrics? The producers, the funders, or
the users?
• Can one resource be more FAIR than another?
• Will/should FAIRness assessments impact funding
decisions?
• Should only one organization define these metrics? Or can
anybody make their own metrics? What happens if a
digital resources scores well against one set of metrics, but
not another?
@micheldumontier::#DANSLOD:2017-05-019
Horizon 2020: Data Management Plan
Section 2. FAIR data
1. Making data findable, including provisions for
metadata (5 questions)
2. Making data openly accessible (10 questions)
3. Making data interoperable (4 questions)
4. Increase data re-use (through clarifying
licenses - 4 questions)
Additional sections:
1. Data summary (6 questions, 5 of which also
cover aspects of FAIRness)
2. Allocation of resources (4 questions)
3. Data security (2 questions)
4. Ethical aspects (2 questions)
5. Other issues (2 questions)
Total of 23 + 16 = 39 questions!!
@micheldumontier::#DANSLOD:2017-05-0110
https://goo.gl/Strjua
FAIRness of repositories
• IDCC17 Practice Paper “Are the FAIR Data
Principles fair?” by Alastair Dunning,
Madelein de Smael, Jasmin Böhmer
• web-interfaces, help-pages and metadata-
records of over 40 data repositories were
examined to score the individual data
repository against the FAIR principles
• ~2 months of work
@micheldumontier::#DANSLOD:2017-05-0111
Data: http://dx.doi.org/10.4121/uuid:5146dd06-98e4-426c-9ae5-dc8fa65c549f
Paper: https://zenodo.org/record/321423#.WNFNrTvytm8
37 repositories
@micheldumontier::#DANSLOD:2017-05-0112
Scoring the resources
@micheldumontier::#DANSLOD:2017-05-0113
@micheldumontier::#DANSLOD:2017-05-0114
@micheldumontier::#DANSLOD:2017-05-0115
Overall evaluation
@micheldumontier::#DANSLOD:2017-05-0116
Summary of Study
• Impressive first attempt at a assessment of
FAIRness across repositories
• Issues
– Lack of fully described mechanism by which
repository owners can provide the necessary
information.
– Fully manual effort, but AFAIK inter-annotator
agreement not established.
– Not easy to scale, can we automate it?
@micheldumontier::#DANSLOD:2017-05-0117
Measures for Digital Repositories
• Data Seal of Approval
– 6 core requirements
– 16 criteria
• DIN31644: Information and documentation -
Criteria for trustworthy digital archives
– 10 core requirements
– 34 criteria
• ISO16363: : Audit and certification of trustworthy
digital repositories
– 100+ criteria
@micheldumontier::#DANSLOD:2017-05-0118
DSA
The data can be found on the Internet
The data are accessible (clear rights
and licences)
The data are in a usable format
The data are reliable
The data are identified in a unique and
persistent way so that they can be
referred to
@micheldumontier::#DANSLOD:2017-05-0119
DSA 16 requirements
1. mission to provide access to and preserve data
2. licenses covering data access and use and monitors compliance.
3. continuity plan
4. ensures that data created/used in compliance with norms.
5. adequate funding and qualified staff through clear governance
6. mechanism(s) for expert guidance and feedback
7. guarantees the integrity and authenticity of the data
8. accepts data and metadata to ensure relevance and understandability
9. applies documented processes in archival
10. responsibility for preservation that is documented.
11. expertise to address data and metadata quality
12. archiving according to defined workflows.
13. enables discovery and citation.
14. enables reuse with appropriate metadata.
15. infrastructure
16. infrastructure
@micheldumontier::#DANSLOD:2017-05-0120
https://www.datasealofapproval.org
Data Seal of Approval
• self-assessment in the DSA online tool. The
online tool takes you through the
16 requirements and provides you with
support.
• Once you have completed your self-
assessment you can submit it for peer review.
@micheldumontier::#DANSLOD:2017-05-0121
• Score data on each FAIR dimension (e.g. from
1 to 5)
• Total score of FAIRness as an indicator of data
quality
• Scoring can only be partly automatic, not all
principles can be established objectively:
– scoring at ingest by data archivists of TDR
– after reuse by data users (community review)
@micheldumontier::#DANSLOD:2017-05-0122
Peter Doorn: https://dans.knaw.nl/nl/actueel/PresentationP.D..pdf
DANS FAIR metrics proposal
@micheldumontier::#DANSLOD:2017-05-0123
@micheldumontier::#DANSLOD:2017-05-0124
@micheldumontier::#DANSLOD:2017-05-0125
@micheldumontier::#DANSLOD:2017-05-0126
@micheldumontier::#DANSLOD:2017-05-0127
http://www.w3.org/TR/hcls-dataset/
http://hw-swel.github.io/Validata/
VALIDATA DEMO
@micheldumontier::#DANSLOD:2017-05-0128
RDF constraint validation tool
Configurable to any profile
Declarative reusable schema description
Shape Expression (ShEx) constraints
Open source javascript implementation
NIH Commons Framework Working Group on
FAIR Metrics
Aim: To identify and prototype methods to
assess the FAIRness of a digital resource.
– Identify and include initial stakeholders
– Develop and discuss potential metrics
– Explore ways in which to report and assess
metrics.
@micheldumontier::#DANSLOD:2017-05-0129
What is a metric?
• A metric is a standard of measurement.
• It must provide clear definition of what is being
measured, why one wants to measure it.
• It must describe the process by which you
obtain a valid measurement result, so that it
can be reproduced by others. It needs to
specify what a valid result is.
@micheldumontier::#DANSLOD:2017-05-0130
Example of a FAIRness Metric
F1 (meta)data are assigned a globally unique and persistent
identifier
Aspect: Identifier Persistence
Rationale: An identifier can be used to find, access, and reuse a
resource. As such, it must be available to users in the longest term
possible otherwise we will not be able to perform those functions with
the identifier in hand.
Relevant FAIR Principles: F,A,I,R
Metric: Availability of data management plan, which includes a section
dealing with continuity and contingencies related to the persistence of
identifiers. The value of the metric is true or false.
Procedure: Check and verify the URL in the resource metadata points to
a data management plan with continuity section. Document should
follow a community standard, or recommend a basic structure.
@micheldumontier::#DANSLOD:2017-05-0131
Current Thinking:
FAIRness Index
• A FAIRness Index is a collection of metrics that
are aligned to the FAIR principles and can be
consistently and transparently evaluated.
• A community, comprised of clearly defined
stakeholders (researchers, publishers, users,
etc), may define their own FAIRness Index
that expresses what makes a digital resource
ideally or maximally FAIR.
@micheldumontier::#DANSLOD:2017-05-0132
Stakeholders
People worried about
– Findability
– Accessibility
– Interoperability
– Reuse
– Provenance
– Licensing
– Citation
– Value
@micheldumontier::#DANSLOD:2017-05-0133
People who are
- Potential users
- Resource creators
- Academics
- Publishers
- Industry
- The public
- Funding agencies
Ways can we gather information to
assess FAIRness
A) Self assessment
B) Self-appointed FAIR Assessment Team
C) Automated assessment
D) Crowdsourcing
E) All of the above
@micheldumontier::#DANSLOD:2017-05-0134
• Is there structured metadata describing the resource?
– Check for embedded metadata as microdata or linked data
– Check for hyperlinked documents with standardized formats: HCLS dataset
description/DCAT schema.org annotations, etc
• Are entries identified with a persistent identifier?
– Is there a DOI with scholarly publications?
– Is there a permanent URL for each item (w/out query parameters)
– Is there a resource type specified, does it use a well known vocabulary such
as EDAM, identifiers.org, etc.
• Can the resource be found in a recognized repository?
– E.g. a database in Biosharing
– E.g. a tool in Elixir bio.tools
– E.g. gene expression data in GEO
• Can the resource be found with a web search engine?
– What rank does the resource appear at when using the identifier or title in a
web search?
@micheldumontier::#DANSLOD:2017-05-0135
Sample Findable Metrics
Sample FAIR Metrics
Accessible metrics
• Are the (meta)data accessible by permanent URL?
• Can you obtain the resource as a standardized language (e.g. HTML, XML, JSON, JSON-LD)?
• Are the data downloadable in bulk or in part with an application programming interface
(API)? Is the API documented using Swagger, smartAPI, or follow the Hydra protocol?
Interoperable metrics
• Are the (meta)data described with a community vocabulary?
• Are the data and metadata linked to other datasets, vocabularies and ontologies?
• Are the data and metadata expressed in universal languages (e.g. XML, JSON, JSON-LD,
RDF/XML)
Reusable metrics
• Is there a license specified? Is it a standardized license? Is it linked to in the resource
metadata?
• Is it clear how the work should be cited? See the FORCE11 Data Citation Implementation
Pilot and bioCADDIE Working Group 5.
• Is there any indication of reuse beyond its original context and original creators?
• Is there any indication of access through published statistics?
@micheldumontier::#DANSLOD:2017-05-0136
michel.dumontier@maastrichtuniversity.nl
Website: http://maastrichtuniversity.nl/ids
Presentations: http://slideshare.com/micheldumontier
37
Early stages of thinking about assessing the FAIRness of
digital resources. Your input can help shape this emerging
phenomenon.
Questions:
1. Does it make sense to, and what are the implications of
assessing the FAIRness of digital resources?
2. What are the barriers to realizing the FAIR vision?
METRICS
@micheldumontier::#DANSLOD:2017-05-01

Contenu connexe

Tendances

Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 

Tendances (20)

Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?
 
Building a Data Governance Strategy
Building a Data Governance StrategyBuilding a Data Governance Strategy
Building a Data Governance Strategy
 
Data Quality for Non-Data People
Data Quality for Non-Data PeopleData Quality for Non-Data People
Data Quality for Non-Data People
 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
Enterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data ArchitectureEnterprise Architecture vs. Data Architecture
Enterprise Architecture vs. Data Architecture
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
Introduction to Data Governance
Introduction to Data GovernanceIntroduction to Data Governance
Introduction to Data Governance
 
Graph Databases – Benefits and Risks
Graph Databases – Benefits and RisksGraph Databases – Benefits and Risks
Graph Databases – Benefits and Risks
 
DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...
DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...
DAS Slides: Building a Data Strategy - Practical Steps for Aligning with Busi...
 
Top 10 Artifacts Needed For Data Governance
Top 10 Artifacts Needed For Data GovernanceTop 10 Artifacts Needed For Data Governance
Top 10 Artifacts Needed For Data Governance
 
Is the traditional data warehouse dead?
Is the traditional data warehouse dead?Is the traditional data warehouse dead?
Is the traditional data warehouse dead?
 
Screw DevOps, Let's Talk DataOps
Screw DevOps, Let's Talk DataOpsScrew DevOps, Let's Talk DataOps
Screw DevOps, Let's Talk DataOps
 
Introduction to Open Science and EOSC
Introduction to Open Science and EOSCIntroduction to Open Science and EOSC
Introduction to Open Science and EOSC
 
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Data Quality
Data QualityData Quality
Data Quality
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 

Similaire à FAIR principles and metrics for evaluation

Similaire à FAIR principles and metrics for evaluation (20)

Towards metrics to assess and encourage FAIRness
Towards metrics to assess and encourage FAIRnessTowards metrics to assess and encourage FAIRness
Towards metrics to assess and encourage FAIRness
 
Developing and assessing FAIR digital resources
Developing and assessing FAIR digital resourcesDeveloping and assessing FAIR digital resources
Developing and assessing FAIR digital resources
 
Data-Ed: Metadata Strategies
 Data-Ed: Metadata Strategies Data-Ed: Metadata Strategies
Data-Ed: Metadata Strategies
 
McGeary Data Curation Network: Developing and Scaling
McGeary Data Curation Network: Developing and ScalingMcGeary Data Curation Network: Developing and Scaling
McGeary Data Curation Network: Developing and Scaling
 
Increasing the Reputation of your Published Data on the Web
Increasing the Reputation of your Published Data on the WebIncreasing the Reputation of your Published Data on the Web
Increasing the Reputation of your Published Data on the Web
 
Meeting Federal Research Requirements for Data Management Plans, Public Acces...
Meeting Federal Research Requirements for Data Management Plans, Public Acces...Meeting Federal Research Requirements for Data Management Plans, Public Acces...
Meeting Federal Research Requirements for Data Management Plans, Public Acces...
 
Accelerating biomedical discovery with an Internet of FAIR data and services ...
Accelerating biomedical discovery with an Internet of FAIR data and services ...Accelerating biomedical discovery with an Internet of FAIR data and services ...
Accelerating biomedical discovery with an Internet of FAIR data and services ...
 
My FAIR share of the work - Diamond Light Source - Dec 2018
My FAIR share of the work - Diamond Light Source - Dec 2018My FAIR share of the work - Diamond Light Source - Dec 2018
My FAIR share of the work - Diamond Light Source - Dec 2018
 
The Importance of Metadata
The Importance of MetadataThe Importance of Metadata
The Importance of Metadata
 
Research-Data-Management-and-your-PhD
Research-Data-Management-and-your-PhDResearch-Data-Management-and-your-PhD
Research-Data-Management-and-your-PhD
 
Data Management and Horizon 2020
Data Management and Horizon 2020Data Management and Horizon 2020
Data Management and Horizon 2020
 
Intro to Data Management Plans
Intro to Data Management PlansIntro to Data Management Plans
Intro to Data Management Plans
 
Data-Ed Online Webinar: Metadata Strategies
Data-Ed Online Webinar: Metadata StrategiesData-Ed Online Webinar: Metadata Strategies
Data-Ed Online Webinar: Metadata Strategies
 
FAIR, standards and FAIRsharing - MAQC Society 2019
FAIR, standards and FAIRsharing - MAQC Society 2019FAIR, standards and FAIRsharing - MAQC Society 2019
FAIR, standards and FAIRsharing - MAQC Society 2019
 
DMP health sciences
DMP health sciencesDMP health sciences
DMP health sciences
 
Data Profiling: The First Step to Big Data Quality
Data Profiling: The First Step to Big Data QualityData Profiling: The First Step to Big Data Quality
Data Profiling: The First Step to Big Data Quality
 
A Data Citation Roadmap for Scholarly Data Repositories
A Data Citation Roadmap for Scholarly Data RepositoriesA Data Citation Roadmap for Scholarly Data Repositories
A Data Citation Roadmap for Scholarly Data Repositories
 
Research Data Management: An Introductory Webinar from OpenAIRE and EUDAT
Research Data Management: An Introductory Webinar from OpenAIRE and EUDATResearch Data Management: An Introductory Webinar from OpenAIRE and EUDAT
Research Data Management: An Introductory Webinar from OpenAIRE and EUDAT
 
Research Data Management: An Introductory Webinar from OpenAIRE and EUDAT
Research Data Management: An Introductory Webinar from OpenAIRE and EUDATResearch Data Management: An Introductory Webinar from OpenAIRE and EUDAT
Research Data Management: An Introductory Webinar from OpenAIRE and EUDAT
 
Research Data Management Introduction: EUDAT/Open AIRE Webinar| www.eudat.eu |
Research Data Management Introduction: EUDAT/Open AIRE Webinar| www.eudat.eu | Research Data Management Introduction: EUDAT/Open AIRE Webinar| www.eudat.eu |
Research Data Management Introduction: EUDAT/Open AIRE Webinar| www.eudat.eu |
 

Plus de Michel Dumontier

FAIR & AI Ready KGs for Explainable Predictions
FAIR & AI Ready KGs for Explainable PredictionsFAIR & AI Ready KGs for Explainable Predictions
FAIR & AI Ready KGs for Explainable Predictions
Michel Dumontier
 
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
Michel Dumontier
 
Are we FAIR yet? And will it be worth it?
Are we FAIR yet? And will it be worth it?Are we FAIR yet? And will it be worth it?
Are we FAIR yet? And will it be worth it?
Michel Dumontier
 
Ontologies
OntologiesOntologies
Ontologies
Michel Dumontier
 

Plus de Michel Dumontier (20)

FAIR & AI Ready KGs for Explainable Predictions
FAIR & AI Ready KGs for Explainable PredictionsFAIR & AI Ready KGs for Explainable Predictions
FAIR & AI Ready KGs for Explainable Predictions
 
A metadata standard for Knowledge Graphs
A metadata standard for Knowledge GraphsA metadata standard for Knowledge Graphs
A metadata standard for Knowledge Graphs
 
Data-Driven Discovery Science with FAIR Knowledge Graphs
Data-Driven Discovery Science with FAIR Knowledge GraphsData-Driven Discovery Science with FAIR Knowledge Graphs
Data-Driven Discovery Science with FAIR Knowledge Graphs
 
Evaluating FAIRness
Evaluating FAIRnessEvaluating FAIRness
Evaluating FAIRness
 
The Role of the FAIR Guiding Principles for an effective Learning Health System
The Role of the FAIR Guiding Principles for an effective Learning Health SystemThe Role of the FAIR Guiding Principles for an effective Learning Health System
The Role of the FAIR Guiding Principles for an effective Learning Health System
 
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
CIKM2020 Keynote: Accelerating discovery science with an Internet of FAIR dat...
 
The role of the FAIR Guiding Principles in a Learning Health System
The role of the FAIR Guiding Principles in a Learning Health SystemThe role of the FAIR Guiding Principles in a Learning Health System
The role of the FAIR Guiding Principles in a Learning Health System
 
Acclerating biomedical discovery with an internet of FAIR data and services -...
Acclerating biomedical discovery with an internet of FAIR data and services -...Acclerating biomedical discovery with an internet of FAIR data and services -...
Acclerating biomedical discovery with an internet of FAIR data and services -...
 
Accelerating Biomedical Research with the Emerging Internet of FAIR Data and ...
Accelerating Biomedical Research with the Emerging Internet of FAIR Data and ...Accelerating Biomedical Research with the Emerging Internet of FAIR Data and ...
Accelerating Biomedical Research with the Emerging Internet of FAIR Data and ...
 
Are we FAIR yet? And will it be worth it?
Are we FAIR yet? And will it be worth it?Are we FAIR yet? And will it be worth it?
Are we FAIR yet? And will it be worth it?
 
The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...The Future of FAIR Data: An international social, legal and technological inf...
The Future of FAIR Data: An international social, legal and technological inf...
 
Keynote at the 2018 Maastricht University Dinner
Keynote at the 2018 Maastricht University DinnerKeynote at the 2018 Maastricht University Dinner
Keynote at the 2018 Maastricht University Dinner
 
The future of science and business - a UM Star Lecture
The future of science and business - a UM Star LectureThe future of science and business - a UM Star Lecture
The future of science and business - a UM Star Lecture
 
Are we FAIR yet?
Are we FAIR yet?Are we FAIR yet?
Are we FAIR yet?
 
Advancing Biomedical Knowledge Reuse with FAIR
Advancing Biomedical Knowledge Reuse with FAIRAdvancing Biomedical Knowledge Reuse with FAIR
Advancing Biomedical Knowledge Reuse with FAIR
 
A Framework to develop the FAIR Metrics
A Framework to develop the FAIR MetricsA Framework to develop the FAIR Metrics
A Framework to develop the FAIR Metrics
 
Data Science for the Win
Data Science for the WinData Science for the Win
Data Science for the Win
 
2016 bmdid-mappings
2016 bmdid-mappings2016 bmdid-mappings
2016 bmdid-mappings
 
Ontologies
OntologiesOntologies
Ontologies
 
Building a Network of Interoperable and Independently Produced Linked and Ope...
Building a Network of Interoperable and Independently Produced Linked and Ope...Building a Network of Interoperable and Independently Produced Linked and Ope...
Building a Network of Interoperable and Independently Produced Linked and Ope...
 

Dernier

Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...
Sérgio Sacani
 
Jet reorientation in central galaxies of clusters and groups: insights from V...
Jet reorientation in central galaxies of clusters and groups: insights from V...Jet reorientation in central galaxies of clusters and groups: insights from V...
Jet reorientation in central galaxies of clusters and groups: insights from V...
Sérgio Sacani
 
The solar dynamo begins near the surface
The solar dynamo begins near the surfaceThe solar dynamo begins near the surface
The solar dynamo begins near the surface
Sérgio Sacani
 
Mitosis...............................pptx
Mitosis...............................pptxMitosis...............................pptx
Mitosis...............................pptx
Cherry
 
Isolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptxIsolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptx
GOWTHAMIM22
 
The importance of continents, oceans and plate tectonics for the evolution of...
The importance of continents, oceans and plate tectonics for the evolution of...The importance of continents, oceans and plate tectonics for the evolution of...
The importance of continents, oceans and plate tectonics for the evolution of...
Sérgio Sacani
 

Dernier (20)

National Biodiversity protection initiatives and Convention on Biological Di...
National Biodiversity protection initiatives and  Convention on Biological Di...National Biodiversity protection initiatives and  Convention on Biological Di...
National Biodiversity protection initiatives and Convention on Biological Di...
 
Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...
 
The Scientific names of some important families of Industrial plants .pdf
The Scientific names of some important families of Industrial plants .pdfThe Scientific names of some important families of Industrial plants .pdf
The Scientific names of some important families of Industrial plants .pdf
 
word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings o...
word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings o...word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings o...
word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings o...
 
Jet reorientation in central galaxies of clusters and groups: insights from V...
Jet reorientation in central galaxies of clusters and groups: insights from V...Jet reorientation in central galaxies of clusters and groups: insights from V...
Jet reorientation in central galaxies of clusters and groups: insights from V...
 
Cell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptxCell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptx
 
GBSN - Microbiology Lab 1 (Microbiology Lab Safety Procedures)
GBSN -  Microbiology Lab  1 (Microbiology Lab Safety Procedures)GBSN -  Microbiology Lab  1 (Microbiology Lab Safety Procedures)
GBSN - Microbiology Lab 1 (Microbiology Lab Safety Procedures)
 
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
 
GBSN - Microbiology (Unit 7) Microbiology in Everyday Life
GBSN - Microbiology (Unit 7) Microbiology in Everyday LifeGBSN - Microbiology (Unit 7) Microbiology in Everyday Life
GBSN - Microbiology (Unit 7) Microbiology in Everyday Life
 
The solar dynamo begins near the surface
The solar dynamo begins near the surfaceThe solar dynamo begins near the surface
The solar dynamo begins near the surface
 
Mitosis...............................pptx
Mitosis...............................pptxMitosis...............................pptx
Mitosis...............................pptx
 
Isolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptxIsolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptx
 
Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...
Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...
Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...
 
The importance of continents, oceans and plate tectonics for the evolution of...
The importance of continents, oceans and plate tectonics for the evolution of...The importance of continents, oceans and plate tectonics for the evolution of...
The importance of continents, oceans and plate tectonics for the evolution of...
 
SCHISTOSOMA HEAMATOBIUM life cycle .pdf
SCHISTOSOMA HEAMATOBIUM life cycle  .pdfSCHISTOSOMA HEAMATOBIUM life cycle  .pdf
SCHISTOSOMA HEAMATOBIUM life cycle .pdf
 
Triploidy ...............................pptx
Triploidy ...............................pptxTriploidy ...............................pptx
Triploidy ...............................pptx
 
GBSN - Microbiology Lab 2 (Compound Microscope)
GBSN - Microbiology Lab 2 (Compound Microscope)GBSN - Microbiology Lab 2 (Compound Microscope)
GBSN - Microbiology Lab 2 (Compound Microscope)
 
GBSN - Biochemistry (Unit 4) Chemistry of Carbohydrates
GBSN - Biochemistry (Unit 4) Chemistry of CarbohydratesGBSN - Biochemistry (Unit 4) Chemistry of Carbohydrates
GBSN - Biochemistry (Unit 4) Chemistry of Carbohydrates
 
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxPlasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
 
Virulence Analysis of Citrus canker caused by Xanthomonas axonopodis pv. citr...
Virulence Analysis of Citrus canker caused by Xanthomonas axonopodis pv. citr...Virulence Analysis of Citrus canker caused by Xanthomonas axonopodis pv. citr...
Virulence Analysis of Citrus canker caused by Xanthomonas axonopodis pv. citr...
 

FAIR principles and metrics for evaluation

  • 1. FAIR principles and metrics for evaluation 1 Michel Dumontier, Ph.D. Distinguished Professor of Data Science @micheldumontier::#DANSLOD:2017-05-01
  • 2. Principles to enhance the value of all digital resources and their metadata. data, images, software, web services, repositories @micheldumontier::#DANSLOD:2017-05-012 http://www.nature.com/articles/sdata201618
  • 4. Rapid Adoption of Principles Developed and endorsed by researchers, publishers, funding agencies, industry partners. As of May 2017, 100+ citations since 2016 publication Included in G20 communique, EOSC, H2020, NIH, and more… @micheldumontier::#DANSLOD:2017-05-015
  • 5. Hypothesis Improving the FAIRness of digital resources will increase their reuse. @micheldumontier::#DANSLOD:2017-05-016
  • 6. What is FAIRness? FAIRness reflects the extent to which a digital resource addresses the FAIR principles as per the expectations defined by a community of stakeholders. @micheldumontier::#DANSLOD:2017-05-017
  • 7. How do we assess compliance to the FAIR principles? • Principles identify what needs to be there, but they don’t tell what is necessary and/or sufficient • They also don’t tell you how to achieve FAIR • Going beyond the principles requires some thought about what constitutes FAIRness and how do we measure it. @micheldumontier::#DANSLOD:2017-05-018
  • 8. Fundamental Questions • In what ways can we assess the FAIRness of a digital resource? • To what degree can we automate this assessment? • Must we treat each type of digital resource differently? • Who will use the metrics? The producers, the funders, or the users? • Can one resource be more FAIR than another? • Will/should FAIRness assessments impact funding decisions? • Should only one organization define these metrics? Or can anybody make their own metrics? What happens if a digital resources scores well against one set of metrics, but not another? @micheldumontier::#DANSLOD:2017-05-019
  • 9. Horizon 2020: Data Management Plan Section 2. FAIR data 1. Making data findable, including provisions for metadata (5 questions) 2. Making data openly accessible (10 questions) 3. Making data interoperable (4 questions) 4. Increase data re-use (through clarifying licenses - 4 questions) Additional sections: 1. Data summary (6 questions, 5 of which also cover aspects of FAIRness) 2. Allocation of resources (4 questions) 3. Data security (2 questions) 4. Ethical aspects (2 questions) 5. Other issues (2 questions) Total of 23 + 16 = 39 questions!! @micheldumontier::#DANSLOD:2017-05-0110 https://goo.gl/Strjua
  • 10. FAIRness of repositories • IDCC17 Practice Paper “Are the FAIR Data Principles fair?” by Alastair Dunning, Madelein de Smael, Jasmin Böhmer • web-interfaces, help-pages and metadata- records of over 40 data repositories were examined to score the individual data repository against the FAIR principles • ~2 months of work @micheldumontier::#DANSLOD:2017-05-0111 Data: http://dx.doi.org/10.4121/uuid:5146dd06-98e4-426c-9ae5-dc8fa65c549f Paper: https://zenodo.org/record/321423#.WNFNrTvytm8
  • 16. Summary of Study • Impressive first attempt at a assessment of FAIRness across repositories • Issues – Lack of fully described mechanism by which repository owners can provide the necessary information. – Fully manual effort, but AFAIK inter-annotator agreement not established. – Not easy to scale, can we automate it? @micheldumontier::#DANSLOD:2017-05-0117
  • 17. Measures for Digital Repositories • Data Seal of Approval – 6 core requirements – 16 criteria • DIN31644: Information and documentation - Criteria for trustworthy digital archives – 10 core requirements – 34 criteria • ISO16363: : Audit and certification of trustworthy digital repositories – 100+ criteria @micheldumontier::#DANSLOD:2017-05-0118
  • 18. DSA The data can be found on the Internet The data are accessible (clear rights and licences) The data are in a usable format The data are reliable The data are identified in a unique and persistent way so that they can be referred to @micheldumontier::#DANSLOD:2017-05-0119
  • 19. DSA 16 requirements 1. mission to provide access to and preserve data 2. licenses covering data access and use and monitors compliance. 3. continuity plan 4. ensures that data created/used in compliance with norms. 5. adequate funding and qualified staff through clear governance 6. mechanism(s) for expert guidance and feedback 7. guarantees the integrity and authenticity of the data 8. accepts data and metadata to ensure relevance and understandability 9. applies documented processes in archival 10. responsibility for preservation that is documented. 11. expertise to address data and metadata quality 12. archiving according to defined workflows. 13. enables discovery and citation. 14. enables reuse with appropriate metadata. 15. infrastructure 16. infrastructure @micheldumontier::#DANSLOD:2017-05-0120 https://www.datasealofapproval.org
  • 20. Data Seal of Approval • self-assessment in the DSA online tool. The online tool takes you through the 16 requirements and provides you with support. • Once you have completed your self- assessment you can submit it for peer review. @micheldumontier::#DANSLOD:2017-05-0121
  • 21. • Score data on each FAIR dimension (e.g. from 1 to 5) • Total score of FAIRness as an indicator of data quality • Scoring can only be partly automatic, not all principles can be established objectively: – scoring at ingest by data archivists of TDR – after reuse by data users (community review) @micheldumontier::#DANSLOD:2017-05-0122 Peter Doorn: https://dans.knaw.nl/nl/actueel/PresentationP.D..pdf
  • 22. DANS FAIR metrics proposal @micheldumontier::#DANSLOD:2017-05-0123
  • 27. http://hw-swel.github.io/Validata/ VALIDATA DEMO @micheldumontier::#DANSLOD:2017-05-0128 RDF constraint validation tool Configurable to any profile Declarative reusable schema description Shape Expression (ShEx) constraints Open source javascript implementation
  • 28. NIH Commons Framework Working Group on FAIR Metrics Aim: To identify and prototype methods to assess the FAIRness of a digital resource. – Identify and include initial stakeholders – Develop and discuss potential metrics – Explore ways in which to report and assess metrics. @micheldumontier::#DANSLOD:2017-05-0129
  • 29. What is a metric? • A metric is a standard of measurement. • It must provide clear definition of what is being measured, why one wants to measure it. • It must describe the process by which you obtain a valid measurement result, so that it can be reproduced by others. It needs to specify what a valid result is. @micheldumontier::#DANSLOD:2017-05-0130
  • 30. Example of a FAIRness Metric F1 (meta)data are assigned a globally unique and persistent identifier Aspect: Identifier Persistence Rationale: An identifier can be used to find, access, and reuse a resource. As such, it must be available to users in the longest term possible otherwise we will not be able to perform those functions with the identifier in hand. Relevant FAIR Principles: F,A,I,R Metric: Availability of data management plan, which includes a section dealing with continuity and contingencies related to the persistence of identifiers. The value of the metric is true or false. Procedure: Check and verify the URL in the resource metadata points to a data management plan with continuity section. Document should follow a community standard, or recommend a basic structure. @micheldumontier::#DANSLOD:2017-05-0131
  • 31. Current Thinking: FAIRness Index • A FAIRness Index is a collection of metrics that are aligned to the FAIR principles and can be consistently and transparently evaluated. • A community, comprised of clearly defined stakeholders (researchers, publishers, users, etc), may define their own FAIRness Index that expresses what makes a digital resource ideally or maximally FAIR. @micheldumontier::#DANSLOD:2017-05-0132
  • 32. Stakeholders People worried about – Findability – Accessibility – Interoperability – Reuse – Provenance – Licensing – Citation – Value @micheldumontier::#DANSLOD:2017-05-0133 People who are - Potential users - Resource creators - Academics - Publishers - Industry - The public - Funding agencies
  • 33. Ways can we gather information to assess FAIRness A) Self assessment B) Self-appointed FAIR Assessment Team C) Automated assessment D) Crowdsourcing E) All of the above @micheldumontier::#DANSLOD:2017-05-0134
  • 34. • Is there structured metadata describing the resource? – Check for embedded metadata as microdata or linked data – Check for hyperlinked documents with standardized formats: HCLS dataset description/DCAT schema.org annotations, etc • Are entries identified with a persistent identifier? – Is there a DOI with scholarly publications? – Is there a permanent URL for each item (w/out query parameters) – Is there a resource type specified, does it use a well known vocabulary such as EDAM, identifiers.org, etc. • Can the resource be found in a recognized repository? – E.g. a database in Biosharing – E.g. a tool in Elixir bio.tools – E.g. gene expression data in GEO • Can the resource be found with a web search engine? – What rank does the resource appear at when using the identifier or title in a web search? @micheldumontier::#DANSLOD:2017-05-0135 Sample Findable Metrics
  • 35. Sample FAIR Metrics Accessible metrics • Are the (meta)data accessible by permanent URL? • Can you obtain the resource as a standardized language (e.g. HTML, XML, JSON, JSON-LD)? • Are the data downloadable in bulk or in part with an application programming interface (API)? Is the API documented using Swagger, smartAPI, or follow the Hydra protocol? Interoperable metrics • Are the (meta)data described with a community vocabulary? • Are the data and metadata linked to other datasets, vocabularies and ontologies? • Are the data and metadata expressed in universal languages (e.g. XML, JSON, JSON-LD, RDF/XML) Reusable metrics • Is there a license specified? Is it a standardized license? Is it linked to in the resource metadata? • Is it clear how the work should be cited? See the FORCE11 Data Citation Implementation Pilot and bioCADDIE Working Group 5. • Is there any indication of reuse beyond its original context and original creators? • Is there any indication of access through published statistics? @micheldumontier::#DANSLOD:2017-05-0136
  • 36. michel.dumontier@maastrichtuniversity.nl Website: http://maastrichtuniversity.nl/ids Presentations: http://slideshare.com/micheldumontier 37 Early stages of thinking about assessing the FAIRness of digital resources. Your input can help shape this emerging phenomenon. Questions: 1. Does it make sense to, and what are the implications of assessing the FAIRness of digital resources? 2. What are the barriers to realizing the FAIR vision? METRICS @micheldumontier::#DANSLOD:2017-05-01

Notes de l'éditeur

  1. G20: http://europa.eu/rapid/press-release_STATEMENT-16-2967_en.htm EOSC: https://ec.europa.eu/research/openscience/pdf/realising_the_european_open_science_cloud_2016.pdf H2020: https://goo.gl/Strjua