1. Relation between the Quality of
the Review Process of Papers and
the Impact of Artificial Intelligence
Journals
José González
José-Ángel Sosa
Jorge Alarcón
Congress on Research Methodology, MUI-TIC
October 27, 2011

2. Index
• INTRODUCTION
• RELATED WORK
• RESEARCH QUESTION
• OBJECTIVES
• RESEARCH METHODOLOGY
• THEORETICAL MODEL
• EVALUATION
• ANALYSIS OF ANOMALIES
• CONCLUSIONS AND FUTURE WORK
• REFERENCES

3. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
Introduction
OBJECTIVES
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• The acceptance or rejection of an item is directly related to
the quality of the referees [1].
• The level of importance of a journal is often measured by the
impact factor of the Journal Citation Reports (JCR).
• The quality of the review article by a journal is positively
related to the impact factor of this journal .
• There are conferences supported by sponsors, where
"quantity" is preferred over "quality” [2].
3

4. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
Related Work
OBJECTIVES
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• Quality of the review process of paper has been studied in the last
50 years [1].
• Peer review has generated opinions for and against [1].
• Alternative ways of performing the review process, like the "open
review" [3] or using a public voting system [1] [4].
• Some researchers think that JCR does not include research
conducted in other languages [5].
• We did not find studies that relate the quality of the review process 4
versus the impact factor of publications on Computer Science.

5. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
Research Question
OBJECTIVES
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
Is there a positive co-relation between the
impact of Artificial Intelligence Journals and
their quality in the review process?
5

6. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Objectives RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• Global Objective:
• To determine if there is a positive co-rrelation between the quality of
the review process of papers and the impact of journals in the
Artificial Intelligence field.
• Specific Objectives:
• To analyze the factors that are involved in the quality of the review
process of articles in Artificial Intelligence.
• To propose a theorical model to measure the quality of the review
process against the impact factor of a journal of Artificial Intelligence.
• To evaluate the theoretical model with quantitative and qualitative
techniques.
• To analyze the possible anomalies (cases where no positive co-
relation occurs) by case studies and interpret the results. 6

7. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
7

8. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
Theoretical Model
OBJECTIVES
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• Quality of the review process
• Comprised by three main factors:
• Factor 1: Quality of the review process as perceived by authors who
submitted an article to the journal. Two sides of the same coin [6].
• Factor 2: Quality of the reviewers of a journal. Expertise of the
reviewers.
• Factor 3: Quality of the review criteria used in a journal.
Procedures, criteria and practices a journal follows to do the review
process.
• We will define the “quality of the review process” as follows:
• Impact 8
• “The average number of times articles from the journal published in
the past two years have been cited in the JCR year” [7].

9. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Evaluation
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
Quantitative and qualitative evaluation methodology.
15 journals selected randomly.
1. Measure of the Quality of the review process:
a. Factor 1: Survey to authors: Using open-ended questionnaires, the
results will be graded from 0 to 1
b. Factor 2: Mean of the H-Index of the reviewers, average value will
be normalized to obtain a value from 0 to 1
c. Factor 3: Survey to reviewers: Using closed-ended
questionnaires, results will be graded from 0 to 1
Quality = Factor 1 + Factor 2 + Factor 3
2. Measure of the Impact of a journal:
• Impact factor found in ISI JCR 2010. 9

10. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Evaluation
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
Quality of the review vs. Impact
1
0.9
0.8
0.7
Quality of the review process
0.6
0.5
Journal
0.4 Linear (Journal)
0.3
0.2
0.1
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Impact

11. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Analysis of Anomalies
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• We may find that some anomalies or strange cases appear.
• The anomalies may provide us with a counter-example for our
hypothesis.
• We will thoroughly analyze the journals with anomalies in a
qualitative case study to find the reasons for the anomaly.
• The results of the case study will give us information to
complete and refine our theoretical model.
11

12. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Conclusions
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• Research.
Relevant, Original and Feasible
• Relevant, we may end up finding that the impact is not a good
indicator to ensure the quality of a publication
• Original: no previous studies on this topic, and
• Feasible: The study involves low-cost research methods (online
surveys, data obtained via the Internet, etc).
• We expect to validate that the quality of the review and its
impact index are positively correlated for Artificial Intelligence
journals.
12

13. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Conclusions
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• We think that the relation between quality of the review and
its impact will be positive.
• The completion of the study will determine the possible
existence of anomalies.
• High impact journals with low quality of review in their papers.
• Low-impact journals with a high quality review process in their
papers.
• As a limitation we are aware that the factors of the quality are
a rough approximation to the real value of quality.
13

14. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
Future Work
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
• This research is work in progress.
• We reached the phase where we propose the theoretical
model.
• Our future work will pursue the completion of the evaluation
phase.
• Find the relation between Impact and Quality of the
publication itself.
14

15. INTRODUCTION
RELATED WORK
RESEARCH QUESTION
OBJECTIVES
References
RESEARCH METHODOLOGY
THEORETICAL MODEL
EVALUATION
ANALYSIS OF ANOMALIES
CONCLUSIONS AND FUTURE WORK
REFERENCES
1. CAMPANARIO, J. M. El Sistema De Revisión Por Expertos (Peer Review): Muchos
Problemas y Pocas Soluciones., 2002. Available from
<www2.uah.es/jmc/an24.pdf>.
2. OLSEN, K.A. The Economics of International Conferences, Jun. 2004. pp. 89-91.
3. WING, J. M.; CHI, E. H. Reviewing Peer Review. Communications of the
ACM, 2011, vol. 54, no. 7. pp. 10-11.
4. KESAHAV, S. Editor's Message. ACM Sigcomm Computer Communications
Review, 2011, vol. 41, no. 3. pp. 3-3.
5. Universidad Nacional de Educación a Distancia de España. Factor De Impacto
De Las Publicaciones Periódicas. 08/2008 [cited 10/23/2011] Available
from:<http://www.uned.es/biblioteca/referencia/impacto.htm#que>.
6. SORENSEN, Karsten. This is Not an Article — just some Thoughts on how to
Write One. . ed. Penti Kerola, Antti Juustila, and Janne Järvinen. Oulu University
ed., Syöte, Finland ed. , August 6–9, 1994, 1994.
7. HARZING, Anne-Wil. Publish Or Perish. [cited 10/23/2011] Available
15
from:<http://www.harzing.com/pop.htm>.

16. Relation between the Quality of
the Review Process of Papers and
the Impact of Artificial Intelligence
Journals
José González
José-Ángel Sosa
Jorge Alarcón
Congress on Research Methodology, MUI-TIC
October 27, 2011