Presentation on open peer review, presented by Damain Pattinson at the Oxford OpenCon satellite event, November 2018.

1. 0
IN REVIEW – UNLOCKING THE
POTENTIAL OF EARLY SHARING
Damian Pattinson & Amye Kennall
OpenCon Oxford
November 2018

2. 1
0
5000
10000
15000
20000
25000
30000
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
TotalPublications
Preprints
Data from Prepubmed.org
Preprints, yay!

3. 2
0
500000
1000000
1500000
2000000
2500000
3000000
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
TotalPublications
Axis Title
Papers
Preprints
Preprints, meh

4. 3
Even after a Statement on data sharing during
public health emergencies signed by Springer
Nature, Wellcome, and many others, only 3%
of publications across both outbreaks were
associated with preprints.
Of those that were made available, most
were available more than 100 days before
publication.
Publications with preprints — the Ebola and Zika outbreaks

5. 4
More than a nice practice, sharing early is a
necessary condition of openness

6. 5
Springer Nature / Research Square history
2000 20172000 2002 2004 2006 2008 2010 2012 2014 2016
Nature Precedings
Protocol Exchange
Under
Consideration
from Nature
Comms
BMC experiments with early
preprint via "Deposited Article"
Partnership
with bioRxiv
American Journal Experts
Rubriq
Journal Guide
Journal checks

7. 6
Powered by Research Square and developed in partnership with BMC, In Review aims to
open up the peer review process to authors and beyond. The first service of its kind, In
Review will provide authors with on-demand access to the granular status of their
manuscript, including number of reviewers invited and immediate access to reviewer
reports, and will allow authors to showcase their work to funders and others and to
engage the wider community for comment and collaboration while their manuscript is
under review.
The In Review platform

8. 7
Author opt-in

9. 8
Public Article page
Editorial badges
Journal status
Hypothes.is
annotations
Review history –
updated in real time
whenever there is a
status change
https://www.researchsquare.com/company/publishers/pre-publication-platform

10. 9
Author view
Suggest Reviewers: Allows
authors to suggest
reviewers if their paper is
held up
Full Peer Review History:
Updated as soon as a review
or decision is received
Reader comments
incorporated into the
review package sent
to editor

11. 10
Journal In Review browse page

12. 11
Feedback so far
• First opt in 2 hrs after launching
• Over 100 articles opted in within first 3 weeks
• Opt in rate to early sharing: 3% à 46%
• Very positive author feedback
Journal
Submissions to "In Review"
Total Submissions (since 18
Oct)
Opt-in rate (%)
Trials 39 169 23%
BMC Neurology 61 121 50%
BMC Anaesthesiology 60 96 63%
BMC Ophthalmology 68 115 59%
TOTAL 228 501 46%
Pilot opt in rate, 29 November

13. 12
Majority of opt-ins are coming from China and Asia
generally
53%
12%
2%
4%
16%
13%
China
Europe
United Kingdom
United States
Asia (exc China)
Other
#TrialsJournal launches innovative #preprint
#platform. Great idea! @MedicalEvidence
@shauntreweek @GrimshawJeremy Tianjing Li
@JohnsHopkinsEPI
Very interesting concept. Hope it becomes the new
standard so we can stop frantically checking our
papers' status. #open #PeerReview
This is a great pilot! Your work gets a DOI (similar to a
preprint) while it is undergoing peer review
@BioMedCentral
“Cool. It’s a little like F1000’s approach but on
steroids.”
Positive qualitative feedback

14. 13
Next steps
• Expand journal participation—at Springer Nature and beyond—creating a cross-
publisher service
• Migration of Protocols Exchange and Nature Precedings (early January)
• Offer ‘Direct Submission’ pathway so that authors can post preprints directly
• Offer author tools such as automated language assessment and reproducibility checks
• Expand community review functionality

15. 14

16. 15

17. 16
Structured review form
RESEARCH SQUARE INDEPENDENT VALIDATION | Sample 1
RESEARCH QUALITY EVALUATION FAIL
MANUSCRIPT TITLE redacted (keywords: QOL, mothers of children with disabilities)
AUTHORS redacted
REPORT DATE Mar 27 2018, 07:06 (UTC -0400)
EVALUATED BY Academic Peer Reviewer
Current Manuscript Version
OBJECTIVE
Does the manuscript have a clear objective?
Yes - there is a clear objective
DESIGN
Is the current approach (including controls / protocols) appropriate for the objective(s)?
No - there are major issues
EXECUTION
Are the experiments and analyses performed with technical rigor to allow confidence in the results?
No - there are major issues
INTERPRETATION
Is the current interpretation of the results reasonable and not overstated?
No - there are major issues
Manuscript Potential
Could an appropriately REVISED version of this work represent a technically sound contribution?
No - manuscript has some fundamental flaw(s)
REVIEWER COMMENTS
This is a study comparing psychiatric diagnoses, quality of life, and global assessment of functioning of mothers of children in 3
groups: children with ASD without intellectual disability, children with ID without ASD and typically developing. This is clearly an
important question, however, the study has significant limitations and the manuscript needs to be a revised by a person who is
familiar with both the English language and with writing scientific papers.
What have the authors done well: The authors identified an important question, and made an attempt to distinguish between
mother of children with ASD without ID, and mother of children with ID. Unfortunately, this study has significant limitations, and
the manuscript itself is poorly written.
Issues with the study itself:
1. What is the rationale for this study, at one point the authors state that this question has not been investigated in the Arab
world, but then they cite studies which were in fact done in the Arab world, but state that there were methodological problems.
2. The sample sizes are quite small for this kind of study, especially if this is not the first study of its kind. Effect sizes are not
reported.
3. The authors state that the children with ASD had no intellectual disability, but did they have any other co-morbidities? It is
stated that the ID group did not have any medical or psychiatric comorbidities, but this information is not provided for the ASD
and typically developing groups.
4. The control group was recruited from social contacts of the ASD and ID groups - this could cause bias, since social contacts can
compare their lives to the lives of their friends. A better control group would have been selected from the general population.
5. There was a significant difference in age between the children in the 3 groups - this could significantly affect results.
6. It is not mentioned whether the children had siblings, what were the ages of the siblings and whether the siblings had any
diagnoses of their own. This information would significantly affect the well-being of the mothers.

18. 17
Peer review channels
Unstructured
Comments
Hypothesis
Annotation
Structured Reviews Invited Reviews
- Allows readers to
ask informal
questions to the
authors.
- Threaded to allow
replies from authors
and other readers.
- Unmoderated but
with ability to
report unsuitable
content.
- Allows readers to
make in-line
annotations.
- Comments can be
collated and
delivered to editor.
- Standard
moderation
channels.
- Free to anyone to
submit.
- Reviewers must
complete a full RQE
using our structured
template.
- Reviews are
verified before
posting.
- Invited by the
assigned journal
editor.
- Can take any form:
RQE, unstructured
or other.
- Includes editor
decision letters.

19. 18

20. 19