This presentation was provided by Dave Kochalko of Artifacts during the NISO event, "Is This Still Working? Incentives to Publish, Metrics, and New Reward Systems," held on February 20, 2019.

1. BUILDING THE INDEX OF RECORD FOR SCIENTIFIC RESEARCH, DISCOVERY, AND EVALUATION
Why we should stop worrying about high impact journal
indicators and start loving high impact scientists
Dave Kochalko
NISO Virtual Conference
“Is This Still Working? Incentives to Publish, Metrics and New Reward Systems” (February 20, 2019)
0x7e1f264a606b364cb5bd61a2c304e8db44314c2bed05b9f8a26e1a43997db53e

2. 2
RECOGNITION AND IMPACT ARE MORE THAN SKIN DEEP
The Mertonian description of
normal science describes citations
as the currency of science.
Scientists make payments, in the
form of citations, to their
preceptors
— Eugene Garfield, 1962
It is better to be looked over
than overlooked.
— Mae West
You can’t judge a book by
its binding
— American Speech, 1944

3. • There is no system for sharing, discovering and attributing most scientific research outputs
• It is universally recognized that sharing data earlier in the research process will accelerate
discoveries
• Today’s metrics and indicators are optimized around a subset of published and indexed literature
3
THE PROBLEM

4. • The published journal article is the single formal
discoverable artifact of research today
4
THE PROBLEM IN SCHOLARLY COMMUNICATIONS TODAY

5. • The published journal article is the single formal
discoverable artifact of research today
5
THE PROBLEM IN SCHOLARLY COMMUNICATIONS TODAY
• Citations that are critical to discovery and researcher
reputation are formally recorded
for only a sub-set of published articles in deeply
retrospective, break-prone, and inconsistent indexes
Cited Articles & Authors

6. • Citations that are critical to discovery and researcher
reputation are formally recorded
for only a sub-set of published articles in deeply
retrospective, break-prone, and inconsistent indexes
• There is no mechanism for linking and associating related
and valuable research artifacts; inhibiting discoverability
and where attribution cannot be formally received
Cited Articles & Authors
Full Body of
Research Artifacts
6
THE PROBLEM IN SCHOLARLY COMMUNICATIONS TODAY
• The published journal article is the single formal
discoverable artifact of research today

7. • Citations that are critical to discovery and researcher
reputation are formally recorded
for only a sub-set of published articles in deeply
retrospective, break-prone, and inconsistent indexes
• There is no mechanism for linking and associating related
and valuable research artifacts; inhibiting discoverability
and where attribution cannot be formally received
Cited Articles & Authors
Full Body of
Research Artifacts
7
THE PROBLEM IN SCHOLARLY COMMUNICATIONS TODAY
• The published journal article is the single formal
discoverable artifact of research today
THE LACK OF CHANGE IS A FAILURE TO RECOGNIZE THAT THIS IS NOT JUST A
TECHNOLOGY PROBLEM, BUT A REPUTATION, RECOGNITION AND ATTRIBUTION
PROBLEM

8. 8
THE RESEARCH PROCESS HAS WELL KNOWN OPPORTUNITIES FOR IMPROVEMENT
“Researchers are increasingly questioning the time it
takes to publish their work. Many say that they feel
trapped in a cycle of submission, rejection, review, re-
review and re-re-review that seems to eat up months
of their lives, interfere with job, grant and tenure
applications and slow down the dissemination of
results”
“He found that the mean review time had roughly
doubled in the past decade, from 50–130 days to
150–250 days, depending on the journal . And when
Royle looked at eight journals that had published cell-
biology papers over the past decade, he found that
publication times had lengthened at seven of them,
mostly because review times had stretched out..”
Cycle Time

9. 9
THOUGHT LEADERS RECOGNIZE THE NEED FOR A NEW APPROACH
“We are very conscious of the huge challenges that human society has now — curing cancer, understanding the brain for
Alzheimer’s….
…But a lot of the state of knowledge of the human race is sitting in the scientists’ computers, and is currently not
shared…. We need to get it unlocked so we can tackle those huge problems”
— Tim Berners-Lee
“Good Science is good science, regardless of where it is published”
— Declaration on Research Assessment (DORA)
“Researchers are still forced to write ‘papers’ for these journals, a communication format designed in the 17th
century…. If we diversify our thinking away from the superficial field of journals and articles, and instead focus on the
power of networked technologies, we can see all sorts of innovative models for scholarly communication”
— Dr. Jon Tennant in AEON

10. 10
SCHOLARLY RESEARCH AND FUNDER POLICY POINT TO THE VALUE OF A NEW APPROACH
• “Journals’ impact factors are determined by technicalities unrelated to the scientific quality of
their articles…Article citation rates determine the journal impact factor, not vice versa...
(Seglen, 1997).
• “The real advantage: immediacy. The results…demonstrate an immense citation advantage for
articles submitted arXiv...” (Gentil-Beccot, et. al. 2009).
• “As an author team…the benefits of posting our manuscript as a preprint were
many…exhilarating to disseminate our findings while they were still ‘fresh’…as opposed to
waiting months to get the word out…” (Maggio, et. al. 2018).
• “The NIH encourages investigators to use interim research products, such as preprints, to
speed the dissemination and enhance the rigor of their work….To cite the product, applicants
and awardees must include the Digital Object Identifier and the Object type (e.g. Preprint,
protocol) in the citation...” (NOT-OD-17-050. 2017).
Seglen, P.O. (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314:498-502.
Gentil-Beccot, A., Mele, S., Brooks, T.C. (2009). Citing and Reading Behaviours in High-Energy Physics. CERN-OPEN-2009-012.
Maggio, L., Artino Jr, A. & Driessen, E. (2018). Preprints: Facilitating early discovery, access, and feedback. Perspect Med Educ 7: 287. https://doi.org/10.1007/s40037-018-0451-8
NOT-OD-17-050, March 24, 2017. Reporting Preprints and Other Interim Research Products. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-17-050.html

11. 11
RESEARCHERS AND PUBLISHERS EMPHASIZE THE NEED FOR A NEW APPROACH
• “Posters, pre-prints, and datasets are all important to me. They give me a chance to start to establish
my academic reputation and network with other collaborators. Also, my dissertation is very data
intensive so I need to have access to lots of open data. These are a 10 out of 10 for me and my
research.” PhD Student, U.S. research-intensive university
• “The value of making these types of research outputs visible is evolving—albeit slowly. In my field
(psychology / mental health), there is a growing recognition that these sorts of research outputs have
value in and of themselves, and that making them visible to others can be a good thing (rather than
information that one holds close as a private resource in one’s own lab). I am personally working to
create collaborative research resources for the field based on researchers sharing datasets. I rank this
a 10 for my work.” Tenured Professor, U.S. research-intensive university.
• “Pre-prints, dataset, protocols, all of the details surrounding the article are essentially important to us.
We may not be able to prioritize implementing features for this in all of our journals and our platform
but they are always top of mind. This is a very solid 10 for us.” Large international primary publisher
• “I see three areas where this content is of high importance: 1) research in politically sensitive and highly
charged subjects, such as environmental science; 2) Plan S-related where there is a push to get
information out and works are published outside the traditional journal system; and 3) for early career
researchers who are eager to become recognized and adopt a more open approach to sharing their
contributions. I would rank this between 8 and 10.” Large international professional society

12. 12
SOME LOGIC ABOUT INCENTIVES
• “We get what we incent” ― Anonymous
• “When you’re working in front of an audience, you have incentive to excel” ― Dave Van Ronk
• “An incentive is a bullet, a lever, a key: an often tiny object with astonishing power to change a
situation” ― Steven Levitt
• “First, I do not sit down at my desk to put into verse something that is already clear in my mind.
If it were clear in my mind, I should have no incentive or need to write about it. We do not write
in order to be understood; we write in order to understand” ― C.S.Lewis
• “If you look at history, innovation doesn’t come just from giving people incentives; it comes
from creating environments where their ideas can connect” ― Steven Johnson

13. • There is no system for sharing, discovering and attributing most scientific research outputs
• It is universally recognized that sharing data earlier in the research process will accelerate
discoveries
• Today’s metrics and indicators are optimized around a subset of published literature
13
THE PROBLEM & SOLUTION
• ARTiFACTS solves this problem by deploying indexing and attribution that can be used by
researchers to SHARE AND PROTECT THEIR WORK PRODUCT IN REAL TIME
• ARTiFACTS is adopting a collaborative approach by integrating with the current research
communication process to the benefit of all involved

14. 14
ARTiFACTS ADDRESSES THESE PROBLEMS BY ALLOWING ALL RESEARCH OUTPUTS TO BE
INDEXED AND FORMALLY CITED FROM THE EARLIEST STAGES OF RESEARCH
YEARS AFTER THE ACTUAL RESEARCH, ONLY A FRACTION OF ARTIFACTS:
• Are indexed and readily discoverable
• Can receive attribution that formerly count toward researcher reputation

15. Enabling Researchers & Scientists To Do
Three Simple Things:
• Establish proof-of-existence, authorship, and
confirm provenance at any time
• Protect and manage intellectual property while
concurrently facilitating knowledge and content
sharing
• Provide and receive valid, break-proof
attribution and assignment of credit at any point
and for any research output
THE ARTiFACTS RESEARCH WORKFLOW PLATFORM
15

16. 16
ARTiFACTS ENABLED ECOSYSTEM
Researchers secure their
pre-publication artifacts
to the blockchain, protect
their IP, and grow their
reputation and
collaboration networks.
Integrations with
research tools, software
applications, and
scientific instruments
transact newly created
works in real-time.
Researchers draw upon an
ARTiFACTS index pre-
populated with metadata
from historical archives.
Funding organizations track
outputs and impact.
Universities and research
organizations monitor, benchmark,
and report performance.
Publishers identify emerging
talent, disciplines, and enrich
value of their content
Corporate R&D gains signal
intelligence and access to
experts

17. 17
CONNECTING RESEARCH WITH REPOSITORIES FOR SHARING AND DISCOVERY

18. 18
HOLISTIC VIEW OF THE SCIENTIST OFFERS BETTER INFORMED SIGNALS
• Index all published works in very stage of development – working papers, preprints, author
manuscript, version of record (written works)
• Link and index supplementary materials that are core to claims with their written works (data)
• Maintain provenance of all work products (trust)
• Stage and frequency of sharing findings
in reusable forms (speed, openness)
• Reuse by colleagues (reliability)
• Durability of findings (influence, impact)
0
20
40
60
80
100
Publications
Data
Provenance
Sharing
Reuse
Durability
Comparative Impact
Scientist A Scientist B

19. 19
IN CONCLUSION
• How might institutions and research facilities best weld available indicators of use or
influence into a meaningful metric?
• If individual scholarship is best gauged by the value assigned to it by the larger community,
then what collection metrics should be gathered for purposes of determining appropriate
rewards in the context of academia?
• How might institutions better address this challenge and reward faculty appropriately?

20. Dave Kochalko
dkochalko@artifacts.ai
artifacts.ai
artifacts_ai
ORCiD: 0000-0002-3331-2751
20
THANK YOU