Tutorial on reviewing your own paper (and those of others)

Reviewing Your Own Paper
(and those of others)
Tim Grant
Retired But Active Researcher (R-BAR)
Chair, ISCRAM Publications & Academic
Standards Committee (PASC)
r̅

Overview
• Goal:
– To help listeners to produce better submissions to
conferences, journals, & books by gaining
knowledge of reviewing own & other papers
• Outline:
– Reviewing: history, process, products, roles
– Reviewing criteria & how to review
– Shortcomings, criticisms & failure of reviewing
– Online possibilities
– Key resources
– An exercise for the attendee
Linköping, 5 Jun 13
Tutorial on reviewing
2

Introduction: about me
• Career:
– 1966-87: Royal Air Force officer (UK & SG)
– 1987-2004: Consultant, ICT services industry (NL)
– 2001-09: Visiting Professor, University of Pretoria (ZA)
– 2004-12: Professor, Netherlands Defence Academy (NL)
• Qualifications:
– 1969: Bachelor of Science, Aero Engineering, Bristol (UK)
– 1984: Defence Fellowship (Masters), Brunel (UK)
– 1996: PhD, Artificial Intelligence, Maastricht (NL)
• ISCRAM:
– 2009-date: Board member & PASC chair
3

Introduction: reviewing (1)
• Reviewing (aka refereeing):
– Subjecting draft proposals, papers, posters, demonstrations,
theses, courses, subjects, etc to critical evaluation
• Why?
– Quality control; help authors to learn
– Admission to “Body of Knowledge” (BoK)
– Assignment of credit & priority to authors
• By whom?
– Independent experts working in same field
– Can be board / panel / tribunal of experts
– More usually peers (other researchers / authors)
4

Introduction: reviewing (2)
• Where?
– Evaluation of applications for funding
– Review of project reports by researchers to assess
successful progress / completion
– Review of draft conference presentations, journal articles,
and monographs to check they meet quality standards
– Evaluation of set of papers after publication for review article
– Evaluation of quality of work produced by individuals, teams,
departments, and institutions to help determine:
• Appointments
• Promotions
• Levels of funding
5
RIN, 2010

Brief history (1)
• 854-931: Ishap bin Ali Al Rahwi, “Ethics of Physician”:
– 1st
description of (medical) peer review
• 1453: printing press
• 1543-1564: Copernicus, Servetus, Galileo, Versalius:
– Review by non-peers
• 1620: Francis Bacon’s New Philosophy:
– Scientific method
• 1662: Royal Society, London, chartered:
– 1st
learned society
• 1665: Philosophical Transactions:
– 1st
scientific journal; editor selects publications
6
Spier, 2002

Brief history (2)
• 1752: Royal Society adopts review procedure:
– Pioneered in Edinburgh since 1731
– Need for editorial committee (initially to fill excess space)
• 1890: Typewriter & carbon copies (1959 Xerox):
– Easier to circulate submissions to reviewers
• Approx 1940: Diversity & specialization of material:
– Excess journal space vanishes; need to discriminate
– Need for external reviewers
• 1989-98: International Congresses on Peer Review
• Since approx 1990: PC, Internet, email, WWW:
– Online journals; Open Access movement
7
Spier, 2002

Reviewing process (1)
Linköping, 5 Jun 13 8
Problem Research Write up Review Publish
Research
Funding
Teach Evaluate
Teaching
Design
Develop
Product Service
Valorization
Evaluate
citations / impact
patents
use in courses
Ideally should predict
citations / impact, patents,
& use in courses
BoK

• Steps in review process:
1. Authors submit draft paper to editors/publishers
2. Publisher’s staff log submission & acknowledge receipt
3. Editors assign draft paper to reviewers
4. Reviewers assess draft paper, recommending acceptance,
rejection, or modification
• Typically 2 or 3 reviewers
1. Editors weigh recommendations, make decision, & inform
authors
2. Authors modify paper & resubmit revised version
9
RIN, 2010

• Steps in ISCRAM review process:
1. Authors upload draft paper to ConfTool
2. ConfTool logs, acknowledges receipt, & informs Track chair
3. Track chair assigns draft paper to reviewers in ConfTool
rejection, or modification in ConfTool
5. Meta-reviewers weigh recommendations in ConfTool
6. Track chair makes decision & informs authors via ConfTool
7. Authors modify paper & upload revised version, details of
changes made & copyright form to ConfTool
8. ConfTool logs, acknowledges receipt, & informs Track chair
10
(Adapted from) RIN, 2010

• Variations on reviewing process:
– Submission:
• (Extended) abstract + paper; paper only
– Submission types:
• Full paper; short/work-in-progress paper; practitioner
paper; poster; demonstration; exhibit
– Reviewing:
• By committee; blind; double-blind; restricted; crowd (open)
– Medium:
• Paper; CD-ROM; USB; online
• Restricted (to attendees or to members) vs open access
11

• Steps in review process (with abstract, eg for book):
1. Authors submit abstract to editors/publishers
2. Editors assess abstract, make decision, & inform authors
4. Publisher’s staff log & acknowledge receipt
7. Editors weigh recommendations, make decision, & inform
authors
12

• Steps in review process (for journal):
1. Authors submit abstract to editors/publishers
2. Editors assess abstract, make decision, & inform authors
4. Publisher’s staff log & acknowledge receipt
7. Editors weigh recommendations, make decision, & inform authors
8. Authors modify draft paper & resubmit revised version
9. Reviewers assess revised paper & make new recommendation
10. Editors weigh recommendations, make decision, & inform authors
13

• Conferences:
– Usually 1 iteration; paper only; 8-10 pages
– (Less often) 2 iterations: first abstract, then paper
• Journals:
– 2 or more iterations; paper only; 20-30 pages
– At least 50% new material for conference paper
• Books:
– 2 iterations:
• Extended abstract or chapter proposal (typically 2 pages)
• Chapter / contribution; 20-30 pages
14

• Differences between disciplines:
– Humanities & social sciences:
• Double-blind common
• Authors may be allowed to nominate reviewer(s)
– Sciences:
• Single-blind common; increasingly double-blind
– Medical field:
• Increasingly open-review
15

Products from reviewing
• Paper template
• Copyright form
• Review template
• Email templates for:
– Acknowledging receipt
– Informing Track chair of upload
– Assigning papers to reviewers
– Rejection, acceptance, & conditional acceptance
– Final acceptance of camera-ready copy
16

Roles in reviewing
• Author: writes, submits & modifies paper
• Track chair: assigns submissions to reviewers
• Reviewer: reviews set (2..6) of submissions
• Meta-reviewer: moderates (2..4) reviews
• (Editor: combines Track chair & meta-reviewer roles)
• Programme Committee = {Editors}
– ISCRAM: Scientific Committee = {Track chairs}
– Moderates across tracks
• Programme Chair (& co-chairs):
– Decides conference schedule
17

Reviewing procedures
• Review procedure documents (eg ISCRAM):
– Review process:
• Overview for conference organizers
– Review timeline:
• Helps conference organizers schedule activities
– Review guidelines:
• For reviewers of full, short / work-in-progress, & practitioner
papers
– Meta-review & final decision:
• For meta-reviewers, track chairs, & Scientific Committee
– Best (Student) Paper procedure:
• For Best Paper sub-committees & conference organizers
18

Criteria for reviewing (1)
Criterion ISCRAM BNAIC PlanSIG ICIW
(Discipline) IS + CRAM AI Planning Cyber/info war
Relevance x x x x
Significance/contribution x - x x
Originality x (full only) x x -
Validity x (full only) x x x
Clarity/readability x x x x
(Others) Best Paper? Award? Method, drawings,
abstract, keywords,
references
Overall score x x x x
Reviewer’s confidence x x x -
Remarks for author(s) x x x x
Remarks for PC x x x -

• Relevance:
– Papers should be clearly relevant to conference’s subject
area, not generalist
– ISCRAM (= IS + CRAM):
• Technical papers (information systems, computer science, IT)
should describe implications for crisis response and/or
emergency management
• Vice versa for crisis response / emergency management papers
– Tips:
• Read Call For Papers!
• Ensure paper fits into topic or track
• Does this year’s conference have special theme?
– If so, try to make link to theme in paper (N.B. nice to have)
20

• Significance / contribution:
– Does paper make a significant contribution to literature?
• Significance means contribution opens up (or closes off) previously
unexplored lines of research or of development (valorization)
– Does paper state the contribution it makes?
– Does paper link this contribution to pre-existing literature?
– Does paper identify the limitations of this contribution?
• From method & from what is outside scope of paper
• Limitations can always be topics for further research!
– Tips:
• Know the relevant literature!
• Don’t overclaim, i.e. overstate contribution
21

• Originality / novelty:
– Ideas must be non-trivial, new & timely, not “more of same”
• Case study can be new
• Literature survey paper can be new if it “adds value”, eg
structuring literature in a new way
• Application of established ideas to another field can be new:
– Idea may be known in IS literature, but not yet applied to CRAM.
– And vice versa - which is why practitioner papers are so important
– Tips:
• Know the relevant literature!
• Where can, exploit multi-disciplinary nature (e.g. IS + CRAM)
22

• Validity / soundness / thoroughness:
– Is purpose of paper clearly stated? And paper’s scope?
– Is research question / hypothesis clearly stated?
– Are appropriate methods used?
• Is best practice for method followed? (eg case study, literature survey)
– Is argument logically sound? Statistics & equations correct?
– Is treatment /discussion thorough?
• In enough detail for another researcher to reproduce what you did?
– Tips:
• Ensure paper tells story:
– Not necessarily the order in which you did your research!
• Reflect on your contribution in context of related literature
23

• Clarity / readability / structure:
– Well structured? Grammatically correct? Easy to read?
– Abbreviations in full on 1st
use? Jargon known to readers?
– Citations given in full & easy to find?
– Can oldies (60+) read graphics without magnifying glass?
– Tips:
• Apply “magic number of 5 +/- 2” to number of sections
• First section should be “Introduction” or “Motivation”
• Last section should be “Conclusions & Further Research”
• Review relevant literature in 2nd
or 2nd
-to-last section
• Do not exceed page or word limit!
• Use spelling & grammar checker!
24

How to review (1)
• Reviewing is extension of reading paper
• Keshav’s three-pass method for reading:
– Pass (1): Quick-scan (5-10 mins):
• Read title, abstract, & introduction
• Read section & sub-section headings (structure)
• Glance at equations (if any)
• Read conclusions
• Glance over references to see which ones you know
– Should be able to answer “five Cs”:
• Category; context; correctness; contribution, clarity
25
Keshav, 2012

How to review (2)
– Pass (2): Read paper with more care (1 hour):
• Ignore details such as proofs
• Jot down key points, or make notes in margin
• Look carefully at figures, diagrams, & other illustrations
• Mark relevant unread references for further reading
• Answer:
– What does paper do? (Rationale/motivation, aims, hypothesis)
– How does it do it? (Participants, method)
– What did it find? (Findings, implications, falsifiable, limitations)
26
Keshav, 2012

How to review (3)
– Pass (3): Virtually re-create paper (4-5 hours):
• Making same assumptions, re-create paper in own way
• Compare re-creation with actual paper:
– Identify & challenge every statement
– Jot ideas down as you go
• Self-reflect: watch out for community groupthink / bias
• Answer:
– Weaknesses:
» Implicit assumptions; experimental / analytical errors; missing
citations
– Strengths:
» Presentation technique; support for conclusions; internal efficacy
Keshav, 2012Tutorial on reviewing

How to review (4)
• Now add reviewing-specific steps:
– Scoring according to reviewing template
– Writing “Remarks to authors”:
• Briefly summarize paper in own words
• State what you think contribution(s) are:
– Not those stated by author(s)
• Give specific comments based on jottings / margin notes:
– Separate major points from minor ones (eg typos)
– By relevance; significance; originality; validity; clarity
– Justifying your scores
• Conclude comments (eg summarize good & bad points)
• Recommend modifications
28
Roscoe, 2007

How to review (5)
• Approaches to reviewing own paper:
1. Write paper & then review it, modifying as needed
2. Write paper, already building in information reviewer needs
• I prefer 2nd
approach (next 6 slides)
• Mindset: Reviewer is always right!
– True: modify as reviewer recommends (Relevance,
Significance, Originality, & Validity)
– False: reviewer misunderstood because draft paper is poorly
expressed; express more clearly (Clarity)
• You’ll still get Remarks to Authors:
– But they should be fewer and less major

How to review (6)
• How I plan a paper (1/6):
– Look at Call For Papers
– Have I done research that is worth presenting?
• If none, STOP
– Which topic / track does it fit into?
• If none, STOP
– What message do I want to give?
• Sketch story (5 plus/minus 2 bullets)
– Write single sentence describing purpose of paper:
• “The purpose of this paper is to present / survey / show ...”
30

How to review (7)
– Extract title from purpose of paper:
• This must “sell” your paper!
Eg “Tweak the tweet: leveraging proliferation with a prescriptive syntax
to support citizen reporting” (Starbird & Stamberger, ISCRAM 2010)
– Extract top-level headings from story:
• Don’t forget 5 plus/minus 2:
1.Introduction
2.Relevant theory
3.Experiment
4.Discussion
5.Conclusions & Further Research

How to review (8)
– Expand 1. Introduction:
1.1 Background (or Motivation)
– Identifies “gap” in current Body of Knowledge
– Motivates why research is worth doing
– Shows relevance to conference & topic / track
– Outlines what paper covers, eg
“This paper focuses on technical capabilities for cyber warfare.”
1.2 Purpose & scope
– Insert one-sentence purpose here
– Scope is what your research & this paper does not include, eg
“Legal issues are outside the scope of this paper.”
1.3 Paper structure (or Layout)
– So reader knows what to expect (sections, appendices)

How to review (9)
– Expand 3. Experiment:
Eg 3.1 Design; 3.2 Execution; 3.3 Results & analysis
– Expand 5. Conclusions & Further Research:
(5.1) Summary
– Not same as Abstract
(5.2) Contribution(s)
– Just most important ones
– Don’t overstate
(5.3) Limitations
– From method used
– From what is outside scope
(5.4) Further research

How to review (10)
– Allocate page budget (see CFP or template for limit):
– Write Abstract:
• Note word limit (in CFP, author instructions, or template):
– Two-thirds background & motivation:
» Conclude with one-sentence purpose of paper
– One-third (intended) structure of paper
Title, abstract & keywords ½ page or 5%
Introduction 5%
Conclusions & Further Research 5%
References 10%
(other sections) divide up 75%

How to review (11)
– Write one-sentence objective for each section:
1. Introduction
“The objective of this section is to describe the background,
motivate the research, state the purpose and scope of the
paper, and outline its structure.”
2. Relevant theory
“The objective of this section is to summarize the theory
relevant to the content of this paper.”
• N.B. To guide writing only; may be deleted once written
– Allocate (sub-) sections to co-authors (if any):
• In consultation, taking into account roles & abilities

Key resources (1)
• Overviews (recommended):
– RIN, 2010. Peer Review: A guide for researchers. Research
Information Network (March), http://www.rin.ac.uk/our-
work/communicating-and-disseminating-research/peer-
review-guide-researchers. Accessed 2 May 13. *
– Wikipedia. 2013. http://en.wikipedia.org/wiki/Peer_review.
Accessed 2 May 2013
– Rowland, F. 2002. The Peer Review Process: A report to the
JISC Scholarly Communications Group.
http://www.jisc.ac.uk/uploaded_documents/rowland.pdf.
Accessed 2 May 2013. *
40

Key resources (2)
• Papers/articles (1/3):
– Calcagno, V., Demoinet, E., Gollner, K., Guidi, L., Ruths, D. & de
Mazancourt, C. 2012. Flows of Research Manuscripts Among
Scientific Journals Reveal Hidden Submission Patterns. Science
Magazine, 338, 1065-9 (November)
– Cormode, G. 2008. How NOT to Review a Paper: the tools and
techniques of the adversarial reviewer. SIGMOD Record, 37, 4. *
– Griswold, W.G. (not dated). How to Read an Engineering Research
Paper. http://cseweb.ucsd.edu/~wgg/CSE210/howtoread.html. *
• See also link to paperform.pdf. *
– Hill, S. & Provost, F. 2003. The Myth of the Double Blind Review?
Author identification using only citations. SIGKDD Explorations, 5,
3, 179-184
– Jefferson, T., Alderson, P., Wager, E. & Davidoff, F. 2002. Effects of
Editorial Peer Review: A systematic review. JAMA, 287, 2784-6

Key resources (3)
– Kershav, S. 2012. How to Read a Paper.
http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-
reading.pdf. (June 26). *
• See Ian McLean’s 2012 Literature Review Matrix, based on Kershav, at
http://www.psychologyinc.org/2012/06/literature-review-matrix.html. *
– Mahoney, M.J. 1977. Publication Prejudices: An experimental study
of confirmatory bias in the peer review system. Cognitive Therapy
and Research, 1, 2, 161-175
– McCook, A. 2006. Is Peer Review Broken? The Scientist Magazine
(February 1)
– Roberts. 1999. Scholarly Publishing, Peer Review and the Internet.
First Monday, 4, 4 (5 April)
– Roscoe, T. 2007. Writing Reviews for System Conferences.
http://people.inf.ethz.ch/troscoe/pubs/review-writing.pdf. *

Key resources (4)
– Smith, R. 2006. Peer Review: A flawed process at the heart of
science and journals. J R Soc Med, 99, 178-182
– Spier, R. 2002. The History of the Peer Review Process. Trends in
Biotechnology, 20, 8, 357-8 (August)
– Van Rooyen, S., Godlee, F., Evans, S., Smith, R. & Black, N. 1998.
Effect of Blinding and Unmasking on the Quality of Peer Review: A
randomized trial. JAMA, 280, 3, 234-7
– Walt, S.M. 2013. On Writing Well. Foreign Policy (February 15).
http://walt.foreignpolicy.com/posts/2013/02/15/on_writing_well. *
• Itself an example of good writing.
– Weiss, R. 2005. Many Scientists Admit to Misconduct: Degrees of
deteption vary in poll. Washington Post (June 9)

Key resources (5)
• Journals:
– Learned Publishing
– Journal of Scholarly Publishing
– Journal of American Society for Information Science
– Journal of Documentation
• Bibliography:
– Bailey, C.W. 2011. Scholarly Electronic Publishing
Bibliography. www.digital-scholarship.org/sepb/. Version 80
(30 November), accessed 2 May 13
• Learned society:
– Association of Learned and Professional Society Publishers
(ALPSP): www.alpsp.org. Accessed 2 May 13
44

Key resources (6)
• Books:
– Peek, R.P. & Newby, G.B. (eds). 1996. Scholarly Publishing: The
electronic frontier. MIT Press, Cambridge, MA
– Page, G., Campbell, R. & Meadows, A.J. 1997. Journal Publishing.
Cambridge University Press, Cambridge, UK
– Meadows, A.J. 1998. Communicating Research. Academic Press,
San Diego, CA
– Tenopir, C. & King, D.W. 2000. Towards Electronic Journals:
Realities for scientists, librarians and publishers. SLA Publishing,
Washington DC
– Fredrikson, E.H. (ed). 2001. A Century of Scientific Publishing. IOS
Publishing, Amsterdam, NL
– Abel, R.E. & Newlin, L.W. (eds). 2002. Scholarly Publishing: Books,
journals, publishers, and libraries in the twentieth century. John
Wiley & Sons, New York

Key resources (7)
• Wikipedia (English, all accessed 2 May 13):
– http://en.wikipedia.org/wiki/Peer_review
– http://en.wikipedia.org/wiki/Open_peer_review
– http://en.wikipedia.org/wiki/Peer-reviewed_scientific_journal
– http://en.wikipedia.org/wiki/Academic_journal
– http://en.wikipedia.org/wiki/Open_access_journal
– http://en.wikipedia.org/wiki/Academic_authorship
– http://en.wikipedia.org/wiki/Academic_publishing
• See also Scholarly paper and Peer review sections
– http://en.wikipedia.org/wiki/Open_access_(publishing)
– http://en.wikipedia.org/wiki/Peer_review_failure
– http://en.wikipedia.org/wiki/Plagiarism
46

Exercise for listener
• “Beer versus science” as example of open reviewing:
– Grim, T. 2008. A Possible Role of Social Activity to Explain
Differences in Publication Output Among Ecologists. Oikos, 177,
484-7. http://www.zoologie.upol.cz/osoby/Grim/Grim_Oikos_2008.pdf. *
• Inverse linear relation between beer input and publications output
– Mack, C.A. 2008. In Defense of Beer-Drinking Scientists. (March 21).
http://life.lithoguru.com/index.php?itemid=119. *
– Mack, C.A. 2008. In Defense of my Defense of Beer-Drinking
Scientists. (March 27). http://life.lithoguru.com/index.php?itemid=120. *
– Mack, C.A. 2008. More on Beer-Drinking Scientists: A response to Dr
Grim. (April 9). http://life.lithoguru.com/index.php?itemid=121. *
– Van Noorden, R. 2010. Make Mine a Double. (Sep 15).
http://blogs.nature.com/news/2010/09/make_mine_a_double.html
• Let’s do field research!

Any questions?
Tim Grant
Retired But Active Researcher (R-BAR)
tim.grant.iscram@gmail.com
+31 (0)638 193 749
With thanks to:
Julie Dugdale, Simon French, Mark S Pfaff, Murray Turoff,
Arien vd Wal

Tutorial on reviewing your own paper (and those of others)

Recommandé

Recommandé

Contenu connexe

Similaire à Tutorial on reviewing your own paper (and those of others)

Similaire à Tutorial on reviewing your own paper (and those of others) (20)

Dernier

Dernier (20)

Tutorial on reviewing your own paper (and those of others)

Notes de l'éditeur