This presentation was provided by Charles Watkinson of the University of Michigan during the NISO Training Series, Assessment Practices and Metrics for the 21st Century. The session was held on Thursday, December 20, 2018.
Developing New Metrics - A Pubrarian's Perspective - Watkinson
1. Developing New Metrics
A Pubrarian’s Perspective
NISO Training Series: Assessment Practices and Metrics
Charles Watkinson, AUL, Publishing, U-M Library
Director, University of Michigan Press
December 20, 2018
3. Why I am interested in (and scared of) assessment?
Interested in assessment because:
● We have a lot of data about use of our
products pouring in. It must be useful,
right? Commercial competitors think so.
● Our authors are increasingly asking us for
data about “impact,” partly because they
are interested but also because they are
being asked for it.
● We are part of a library that is very
interested in assessment (has just hired its
first assessment specialist, Craig Smith)
and we are trying to work out how to
benefit from this.
Scared of assessment because:
● The data we have coming in is very messy
and in many cases is very partial and
obfuscated.
● We don’t have good systems to
aggregate, analyze, or communicate this
data.
● We don’t have staff who know how to do
this work, nor do the staff who are
interested have the capacity to do much.
● We’re aware of the underlying
weaknesses of the data and worried about
unintended consequences.
4. Products vs. Services / Collections vs. Services
“Many organizations aspire to regain a competitive advantage by transforming
from a product manufacture into a service provider. This requires a substantial
reinvention . . . Moving from products to services sounds like an easy and obvious
value proposition shift, but it requires substantially reengineering the business
model.” Osterwalder, A., Pigneur, Y., Bernarda, G., & Smith, A. (2014). Value
proposition design: How to create products and services customers want. John
Wiley & Sons, p. 164
We have not made this shift yet in terms of assessment . . .
6. Michigan Publishing: Books
We want a UMP book (OA or not) to:
● Make money
○ Sales
○ Licenses (e.g., for translations)
● Generate prestige for us/author
○ Peer reviews
○ Journal reviews
○ Awards and prizes
○ Rankings
○ Acceptance rate?
● Be used and useful
○ Downloads
○ Views
○ User experience improvement
○ Discovery improvement
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Syllabus adoptions
○ Geographical spread of use
● Manifest our values?
○ Accessibility? Inclusion?
12. What are the measures of
“success” for a journal/article?
13. Michigan Publishing: Journals
We want an OA journal to:
● Generate prestige for us/sponsor
○ Citations
○ Reviews
○ Acceptance rate
○ “Impact factor” type things?
○ Peer perception
● Manifest our values?
○ Accessibility? Inclusion?
● Be used and useful
○ Downloads
○ Views
○ User Experience
○ Discovery
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Spread of use
○ Syllabus adoptions
○ Accessibility? Inclusion?
16. What are the measures of
“success” for a repository object?
17. Michigan Publishing: Deep Blue docs and data
We want an OA repository object to:
● Be “recognized”
● Be used and useful
○ Downloads
○ Views
○ User experience
○ Discovery
○ Comparative downloads
(https://jusp.jisc.ac.uk/secure/irus-usa/about/)
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Spread of use
○ Syllabus adoptions
● Manifest our values?
○ Accessibility? Inclusion?
20. What are some major trends in metrics
and assessment that
publishers are focused on?
21. Some trends worthy of attention
1. Rise of the RIMS -- Research Information Management Systems
2. Use and engagement as the currency of growing open access
3. Growing sophistication of altmetrics, measuring far more than just
social media activity
4. Ethical considerations about what is being measured and what it
means
5. More attention to “the plumbing” -- identifiers (DOIs, ORCIDs, etc.)
and how supply chains do(n’t) work
27. 3. Growing sophistication of altmetrics
“How will Brexit affect health and health services in the UK? Evaluating three possible scenarios”
The Lancet, November 2017, https://doi.org/10.1016/S0140-6736(17)31926-8
28. 4. Ethical considerations (privacy, misuse, bias, etc.)
http://www.metrics-toolkit.org/
https://humetricshss.org
https://sfdora.org/
San Francisco Declaration on Research Assessment
29. 5. More attention to the plumbing
https://www.crossref.org/blog/the-article-nexus-linking-publications-to-associated-research-outputs/
ONIX vs. MARC