This paper identifies five major aspects of peer review: timing, coverage, form, check list, metrics, and summarizes common practices of every aspect. The high efficient combinations of these five aspects, which help software developing teams to improve the effectiveness of peer-reviews, are recommended for various types of project stages and working products,. According to the actual situation, any organization that carries out peer-review activities should understand appropriate scenarios of these five major aspects, and combine them advisably in order to achieve better results.

1. 5th World Congress for Software Quality – Shanghai, China – November 2011
Five Major Aspects of Peer Review and Their Efficient Combinations
1st Author 2nd Author 3rd Author
Zhang Keqiang Wen Yang Huang Linfang
Shanghai, China Shanghai, China Shanghai, China
zhangkeqiang@gmail.com wangyang_178314@baosight.com huanglinfang@baosight.com
Abstract
This paper identifies five major aspects of peer review: timing, coverage, form, check list, metrics, and
summarizes common practices of every aspect. The high efficient combinations of these five aspects,
which help software developing teams to improve the effectiveness of peer-reviews, are recommended for
various types of project stages and working products,. According to the actual situation, any organization
that carries out peer-review activities should understand appropriate scenarios of these five major aspects,
and combine them advisably in order to achieve better results.
1. Introduction
Peer review is one of effective quality management methods at the product early stage. When CMM/CMMI
spreads globally, peer review is carried out widely as the basic requirement and effective practice in
CMM/CMMI. But after peer review lasts some period of time, there are still much of confusion. For example,
How to determine the coverage of peer review? How to judge the effectiveness of peer review? How to
improve the efficiency of peer review, which peer review should be hold meeting? How to measure peer
review. According to years of peer review practice, this paper identifies five major aspects of peer review
and their common practice. For different work products, high efficient combinations of five aspects are
recommended.
2. Five Major Aspects of Peer Review
Five Major Aspects of Peer Review are listed below.
1） Time
2） Coverage
3） Format
4） Checklist
5） Measurement
2.1 Time
The common time of peer review is listed below.
 Before the final version：before the final version of work product, one or several peer reviews are
arranged in a short period. Commonly, the entire work product is checked with the key points.
 By stages：the work product is reviewed at every stage. The focus of review is the incremental parts of
the work products.
 short cycle：regular review, the time of cycle is less than one week, even daily review.
 Learing style：the typical parts are selected for sharing knowledge and experience.
2.2 Coverage
The common options of coverage are
 100%. This is suitable for high quality requirement.
 About 50%, This is suitable for flexable situation.
 About 30%, This issuitable for low quality requirement.
2.3 Format

2. 5th World Congress for Software Quality – Shanghai, China – November 2011
The main formats of peer review have four.
 Single offline peer review. The reviewer receives the work product, and reviews it offline, then give the
results to the author. This is the most cost-saving format.
 Pair online peer review. The reviewer and the author work together with interactive questions and
answers. The another name of this format maybe "Desk check".
 Meeting with precheck peer review. Before the meeting, several reviewers check the work product
independently, in the meeting reviewers and authors discuss the found and affirm the issues which are
required to modify. The another name of this format is"Inspection".
 Meeting without precheck peer review. Reviewer need not check the work product before the meeting.
In the meeting authors introduce the work products, reviews raise the problems and discuss them on
the spot.
2.4 Check List
Check list of peer review have below styles.
 No checklist, reviewers judge the problems by own knowledge and experience。
 Short checklist, The short checklist is maintained meticulously according to the frequent defects, the
length of checklist is limited. Reviewers use the checklist as seeds of check point.
 Long checklist：The comprehensive long checklist is provided, reviewers check the items in the list one
by one, and need no more expanding.
2.5 Measurement
There are some methods of measurements for peer review, the main is listed below:
 No record
 Record defects
 Record efforts, size of artifact reviewed
 Record the execution of checklist
 Carry out statistics process control (SPC)
3. Recommendation of High efficient peer review combinations
3.1 Necessity of high efficient combination
The sum of five aspects combinations is a hug number, according the above aspects, the sum is
4*3*4*3*5=720. For one peer review, only one combination can be selected, it is somewhat difficult to make
the right selection. so it is worth thinking of high efficient combination for better result.
3.2 Work product of peer review
Peer review can be carried out when the work product is partlaly completed. It is an effective method of
quality control at early stage. The work products in process are worth taking peer reviews. The common
work products of peer review are requirements documents, design documents, source code, varied plans,
test-case, user documents.
3.3 Suitable situation of peer review
Even for the same work product, different peer review combinations should be taken at the different
situation, expercially in the different software develop life cycle models. There are traditional SDLC, such as
waterfall, prototype, also there are some agile software development methodology, such as eXtreme
Programing, Scrum etc.
3.4 Recommendation of requirement document peer review
Table 1. recommendations of requirement document peer review
Five Major Aspects of Peer Review and Their Efficient Combinations Page 2 of 7

3. 5th World Congress for Software Quality – Shanghai, China – November 2011
Recommendation 1 Recommendation 2
Situation Early stage, uncompleted Last stage , completed documents in
documents in traditional traditional SDLC
SDLC
Time By stage Before the final version
Coverage Several times achieve achieves 100%
100%
Format Single offline Meeting with precheck
Checklist Short checklist Short checklist
Measurement Record defects Record defects
Record the execution of Record the execution of checklist
checklist
3.5 Recommendation of design document peer review
Table 2. recommendation of design document peer review
Recommendation 1 Recommendation 2
Situation Early stage, uncompleted completed documents in traditional
documents in traditional SDLC
SDLC
Time By stage Before the final version
Coverage Several times achieve achieves 100%
100%
Format Meeting without precheck Meeting with precheck
Checklist Short checklist Short checklist
Measurement Record defects Record defects
Record the execution of checklist
3.6 Recommendation of code peer review
Table 3. previous 2 recommendation of code peer review
Recommendation 1 Recommendation 2
Situation Quality target ofZero Agile development
defect , in CMMI4or CMMI5
Time short cycle short cycle
Coverage 100% 100%
Format Single offline Pair on-line review , even pair
programing
Checklist Short checklist None
Measurement Record defects None
Record the execution of
checklist
Record efforts of review,
size of the reviewed.
Carry out statistical
process control (SPC)
The below graphic is a example of code peer review SPC.
Five Major Aspects of Peer Review and Their Efficient Combinations Page 3 of 7

4. 5th World Congress for Software Quality – Shanghai, China – November 2011
缺陷密度
缺陷密度（个/千行）
50.00 CL
45.00 UCL
40.00 LCL
35.00
30.00
25.00
20.00
15.00
10.00
5.00
0.00
106
113
127
141
148
155
162
169
176
190
204
211
15
22
29
36
50
64
71
78
85
92
99
120
134
183
197
43
57
1
8
Figure1- example of code peer review defect density SPC, the title in graphic is "defect
density(/kloc)"
Table 4. last 2 recommendation of code peer review
Recommendation 3 Recommendation 4
Situation Share experience very important code
Time Learning style short cycle
Coverage Select the typical code 100%
Format Meeting without precheck Meeting with precheck
Checklist Short checklist Short checklist
Measurement Record defects Record defects
3.7 Recommendation of plan documents peer review
Table 5. recommendation of varied plans peer review
Recommendation 1 Recommendation 2
Situation Such as project develop plan Such as testing plan
Time Before the final version Before the final version
Coverage 100% 100%
Format Meeting with precheck Single offline
Checklist Long checklist Long checklist
Measurement Record defects Record defects
Record the execution of Record the execution of checklist
checklist Record efforts of review,
Record efforts of review, Record size of the reviewed.
Record size of the reviewed.
The below graphic is a screen-shot of a plan checklist, it is shown that there are 74 check items with number
Of uses and number of defect found in this checklist.
Five Major Aspects of Peer Review and Their Efficient Combinations Page 4 of 7

5. 5th World Congress for Software Quality – Shanghai, China – November 2011
Figure 2. development plan long checklist, the columns are "content of check item", "number of
uses", "number of defect found".
3.8 Recommendation of test case peer review
Table6-recommendation of test case peer review
Recommendation 1 Recommendation 2
Situation Quality target ofZero defect , in Agile development
CMMI4or CMMI5
Time short cycle short cycle
Coverage 100% 50%
Format Single offline Pair on-line review , even pair
programing
Checklist Short checklist none
Measurement Record defects none
Record the execution of checklist
3.9 Recommendation of user documents peer review
Table 7-recommendation of user documents peer review
Recommendation 1 Recommendation 2
Situation voluminous user document Short user documents, such as
presentation, voluminous
Time By stages Before the final version
Coverage 30% 100%
Format Single offline Meeting without precheck
Checklist Short checklist No checklist
Measurement Recoord defects Record defects
Record efforts, size of the
reviewed
Record the execution of checklist
4. Rules of recommended combinations
Some rules can be concluded From the above recommendation.
 Short checklist must be maintained meticulously. The typical check items which helps to find more
defects should be included in checklist, and the check results should be recorded. From these records,
the items which can't bring defects can be found, and be replaced by new items from new defects.
 Code and test case are increase fast, only short cycle peer review can cover the increase 100%. The
shorter cycle brings more effectiveness, but more cost, these need balance. The extreme peer review
is pair work which is reviewing all the time. For long term benefits and absolute good quality, it is worth
carrying out highly frequent and full coverage peer review of code and test case.
 Single offline peer review is recommended at several places. It is the most cost-saving format which
can deal with any work product.
 User documents mostly take meeting without precheck peer review, because the reviewer in meeting
without precheck is similar with the real user who is listening the presentation.
 The learning style peer review can help the newbies to improve their skill, help work mates to share
their experience. If there is no frequent and high coverage peer review, then the learning style peer
review is highly recommended to ally all team's understanding.
 All kinds of Plan need a complete check list, because every part in plan is important which will affect
the next step. So it is worth checking the plan carefully by long complete check list.
Five Major Aspects of Peer Review and Their Efficient Combinations Page 5 of 7

6. 5th World Congress for Software Quality – Shanghai, China – November 2011
5. Example
The above combinations are used in a CMMI5 organization, the screen-shot is came from the real system
at that organization. Through the peer review and other quality management methods, the more defects are
found at the early stages, the testing defect density decreased obviously. Please see the below figure.
功能点缺陷密度（个/个）
0.2
0.1520.145
0.15
0.089
0.1
0.058
0.0333 0.0341
0.05
0
2008年7月 2009年7月 2010年7月
Figure 5. Density of testing defect trend, The title is "defect per function point density".
The test turns to final release also decrease ,so the period of test is shorten, see below.
5 4.57 平均测试轮次
3.92 3.89 3.75
4 3.53
3 2.27 2.21 2.04
2
1
0
2008年1月 2008年12月 2009年11月
Figure 6. Turns to release trend, the Title is "Average test turns"
6. Conclusion
In this paper, five major aspects of peer review are identified. The frame of peer review is established
clearly. It it more convenient to analyse peer review. The typical practise of five aspects of peer reviews are
provided. According to different situations, high efficient combinations of five aspects are proposed.
These practise in the five aspects of peer review have their strengths and weaknesses. The organization
need combine these aspects according to its own situation advisably so that the better results can be
achieved.
References
®
[1] Mary Beth Chrissis, Mike Konrad, Sandy Shrum , Capability Maturity Model Integration V1.2
http://www.sei.cmu.edu/cmmi/
[2] IEEE Std. 1028-1997, "IEEE Standard for Software Reviews"
Five Major Aspects of Peer Review and Their Efficient Combinations Page 6 of 7

7. 5th World Congress for Software Quality – Shanghai, China – November 2011
Contact author
 zhangkeqiang@gmail.com
 http://weibo.com/zhangkeqiang
 http://hi.baidu.com/hespr
 http://blog.csdn.net/zhangmike
Five Major Aspects of Peer Review and Their Efficient Combinations Page 7 of 7