Scientific publishing standards: lessons learned from the Surgisphere incident

1. Scientific publishing: lessons
learned from the Surgisphere
incident
Dr James A Watson
Mahidol-Oxford Tropical Medicine Research Unit
Thailand

2. Summary
• Flagship medical journals play an important role of curation and
influence medical research and practice
• Their publishing strategy results in an increased risk of publishing
erroneous results relative to domain specific journals (incentives +
lower prior on new results vs replication)
• “Open publishing” mitigates these risks
• Post-publication review is a mechanism for correcting the record
rapidly
• Journals should embrace post-publication review, not ignore it

3. May 1st
May 22nd
May 24th
Statistical concerns: confounding by
indication?
May 25th
Data/transparency concerns: only 4 authors; no ethics
review; data locked behind NDAs; very uniform aggregated
data
May 25th Australian data don’t
add up
May 26th
Prof Joshua Davis (president of the Australasian Society
for Infectious Diseases) writes to Lancet & Mehra
Australian data are impossible. HCQ not used outside of
RCTs; CQ impossible to get; no one in Australia heard of
Surgisphere
Suspend all HCQ trials in UK
May 28th
“One hospital self-designated as belonging to the Australasia
continental designation should have been assigned to the Asian
continental designation. There have been no changes to the
findings of the paper.”
May 28th
Long list of problems: statistical analysis;
data/code sharing; ethics; data
provenance; Australia; Africa; data
homogeneity; HCQ dosing; HCQ to CQ
ratios; CIs around hazard ratios

4. June 5th
June 5th
“[..] several concerns were raised with respect to the
veracity of the data and analyses conducted by Surgisphere
Corporation and its founder and our co-author, Sapan Desai
[..]. We launched an independent third-party peer review of
Surgisphere with the consent of Sapan Desai to evaluate
the origination of the database elements, to confirm the
completeness of the database, and to replicate the analyses
presented in the paper.
Our independent peer reviewers informed us that
Surgisphere would not transfer the full dataset, client
contracts [..]”
Because all the authors were not granted
access to the raw data and the raw data
could not be made available to a third-
party auditor, we are unable to validate
the primary data sources underlying our
article, “Cardiovascular Disease, Drug
Therapy, and Mortality in Covid-19.”1 We
therefore request that the article be
retracted. We apologize to the editors and
to readers of the Journal for the difficulties
that this has caused.
Signed by Sapan Desai!
Mid June: Surgisphere has vanished: no
website, no twitter
How to reconcile this with retraction statement?

5. Post-publication review to correct the record
• Open post-publication review is better than secret peer review
• Post-publication review played the key role in Lancet and NEJM
retractions
• Post-publication review was not formally recognized by either journal

6. Minor errors
erratum
Wrong research
Massive fraud
retraction
Inconsequential Hopefully unusual..?
?
Subtle errors that could have
major impact on findings – might
not justify full retraction, but
simply a revision

7. Increased risk of wrong research in top-journals?
• Selection bias: suppose “quality” and
“importance” are uncorrelated overall, selection
based on at least one induces inverse correlation
• Incentives to embellish, overlook issues, or deceive
• A priori novel findings less likely to be true than
replication of previous results
• Harder to correctly review very broad research
field (i.e. all of medicine)

8. Current options for correcting the record
Letter to the editor: 150-250 words.
This is clearly not enough for complex arguments, e.g. showing that a
statistical analysis is flawed.
Authors used a model with 39 parameters to
explain 50 events. Overfitting is an understatement
9 page report demonstrating why the analysis is
fatally flawed
These serious concerns are not clearly indicated on the publication webpage

9. Suggested changes
1. Clear description of official journal policies for serious post-
publication concerns. Could use peer review to check validity
2. Acknowledgment of “error detectors”: formal submission type
would help guarantee recognition
3. Better discrimination between types of error: retraction is
associated with mis-conduct, whereas most wrong stuff is due to
honest error: positive incentive

10. “If you have an author who deliberately tries to mislead, it’s surprisingly easy
for them to do so” Richard Horton (Lancet editor)
Yes, but most erroneous research is probably due to honest error not fraud
“High-impact” journals need to embrace post-publication review and not
ignore it
Curation of the best science: this
should foster a conversation not be
exclude any conversation