Its complete standardized item development process of Punjab Examination Commission

1. Dr. Nasir Mahmood, Assessment Expert

2. 1. Internationally Recognized Examination Development
Process
2. Team of Assessment Wing
3. Main Characteristics of PEC assessment tools
4. Development of paper specification
5. Item Writing
6. Item Review
7. Pilot testing
8. Item Banking
9. Paper Generation
10.Conduct of Assessment
11.Marking of Assessment
12.Dissemination of Results
13.PEC Collaboration with other Departments
14.Areas of Collaboration with Special Education Department

4. Qualified team for assessment development recruited through competitive
mode
1. Director Assessment, PhD Education
2. Psychometrician, PhD Mathematics
3. Assessment Experts, PhD, M.Phil Education + M.Sc
4. Technical Translator (M.As in Urdu and English)
5. Subject Specialists (Masters in relevant subjects)
6. External Item Writers/Reviewers (Masters in relevant subjects)

5.  Our Instruments are:
 Valid
 Reliable
 Error free
 Bilingual
 Based on International Best Practices
▪ Framework development
▪ Alignment of SLOs with Textbooks
▪ Item development and review by experts
▪ Pilot testing
▪ Item analysis using software
▪ Equated Versions
▪ Complete record of final papers
 Ensure Uniformity and Consistency in marking through standardized Keys and
Rubrics

6.  Major activity starts from the beginning of new academic cycle i.e. April
 Checking alignment of books with curriculum and assessment design
 Old frameworks are revised to make amendment if any
 Items are prepared as per new framework
 Approved Composition of Papers is 50 : 50 (MCQs : CRQs).
 Papers are composed of cognitive skills in a general ratio of 30:50:20 for
Knowledge: Understanding : Application Plus(Higher Skills)

7.  Items are developed by Subject Specialists and Selected External
Item Writers on Flimzy /item attributes form
 Items are developed for all grades 1-8
 Items are aligned with 6 cognitive levels of Bloom’s taxonomy
 Item development ongoing process From April till December
 Items are developed by the practicing/classroom teachers in
workshop mode
 Teachers are trained before developing questions/items
 Content and pedagogy experts are engaged in item development

8.  Items are reviewed by experienced teachers/experts to improve quality
 In addition, Items are reviewed by content experts, Assessment Expert, Technical Translator and Psychometrician
as per set parameters
 Technical Translator ensures error free translation of bilingual papers (Maths, Science, Islamiyat, Computer
Science, Ethics and Social Studies)
 Assessment Expert ensure construct validity of items
 Alignment of item with SLO
 Cognitive level of items
 Plausibility of distracters
 Psychometric ensures psychometric validity of items
 Item difficulty
 Item discrimination
 Biasness, if any

9.  To sort out valid and reliable items for quality assessment
 One item is tested on at least 350 students in the field
 Generally try out is done in two phases November and December
 60% syllabus is covered in 1st phase while remaining in 2nd phase
 Piloting is done in sampled schools and in sampled districts
 Sampling strata is genders, location and school type etc.

10.  Items are analyzed using Iteman Software based on Classical Test Theory (CTT) and Xcalibre
based on IRT theory.
 PEC use Item Response Theory (IRT ) for analysis of large scale assessment instruments
 PEC has set criterion for selection of items.
 Items are reviewed on the basis of piloting output (difficulty level and discrimination factor. Dif
analysis)
 Items are provided to schools through login in a secure manner. Each paper has a bar code
showing complete information of school.
 Keys and Rubrics are also provided online for standardized and valid marking and scoring

11.  Equated multiple versions of each subject
 Paper can be generated on Single click according to specific criteria
 Multiple adjustments are available for printing of papers like two column, single
language, question placement, font size etc.
 Guidelines/videos are developed to generate paper from item bank.
 Proper record of paper development is maintained.
 Error free papers are provided over to printers/districts/schools as per SOPs
 Maximum curriculum is covered in Assessments
 Papers are generated according to set criteria through specific module

12.  SOPs are developed for the conduct of assessment stating responsibilities of:
 DEAs (DEO-EE-M, AEOs etc.)
 School heads
 Teachers
 Videos are developed on SOPs/responsibilities, paper generation, marking, filling report card template
 Conduct Manual is prepared to conduct assessments according to SOPs
 Training is imparted to CTSC heads and AEOs directly at district level
 These CTSC heads and AEOs impart the same training to heads and teachers of their clusters’ schools.

13.  Keys and Rubrics for each subject are prepared
 Marking Manual for teachers to mark papers according to rubrics and keys is prepared for valid and
standardized marking and scoring.
 Lead Master trainers for each district are selected and trained at PEC
 Lead Master trainers train head examiners and examiners in the field at district level/marking center
level.
 Subject Specialists coordinate all these trainings, and Assessment Experts, Psychometrician and
Director Assessment also impart and monitor the training.
 To ensure reliability syndicate marking is recommended.
 Marking is done at school level by the teachers
 PEC appoints monitors to ensure the marking according to rubrics and SOPs

14.  Reports are uploaded on PEC website
 Dissemination of reports may be done at divisional
and district level
 Department wise report of overall performance of
schools may be developed

15.  Assessment Wing has not only strong coordination with other wings of PEC but with other
national and international organizations
 National Institutes and Organizations:
 Agha Khan University Examination Board
 Provincial Assessment Bodies like that of Balochistan, KPK, Sindh, GB, ICT
 NEAS (National Education and Assessment System), as Provincial coordinator
 Punjab Education Foundation (PEF)
 PIEMA
 PCTB
 QAED
 PESP
 CM Roadmap Team
 DEAs
 BISEs
 Special Education
 Institute of education and Research (IER), University of the Punjab
 Some international organizations:
 UNICEF
 World bank
 Trends in Mathematics and Science Studies (TIMSS), as provincial coordinator
 British Council

16. PEC can provide support to special education department in the following
areas
 Mapping of SLOs and development of Assessment Framework
 Training of Teachers for Assessment Development
 Development of Inclusive Assessment for Mainstream schools and special
schools
 Include in Pilot testing of items
 Provision of Item Bank at Institution level through Item Banking Software
 Mapping of existing items (if available) with curriculum, their review and
standardization to be included in IBS.
 Psychometric analysis of items (if available) and their interpretation and
selection.
 Conduct of research studies on assessment issues of special children

17. Thanks

18. Back

19. Back

20. Back

21. Reaching a decision
 Step 1.An rbis is reported for every option. If none of these exceeds 0.17,
the item does not relate significantly to overall subject test performance.
REJECT.
 Step 2. If more than one option has a positive rbis, the item has more than
one answer. MODIFY one of these options.
 Step 3. If only one option has a positive rbis and that is the key, ACCEPT.
 Step 4. If the option with a positive rbis is not the Key, CHECK the KEY.
Next

22. Next

23. Next

24. Back

25. Equating of Versions
Subject: Science Grade: 5 Paper: A
Q. No. Version 1 Version 2 Version 3 Version 4
1 0.822 0.639 0.587 0.47
2 0.592 0.774 0.596 0.201
3 0.626 0.558 0.76 0.717
4 0.252 0.373 0.37 0.259
5 0.487 0.335 0.493 0.27
6 0.304 0.409 0.366 0.423
7 0.292 0.725 0.479 0.763
8 0.483 0.302 0.703 0.356
9 0.526 0.714 0.691 0.759
10 0.412 0.599 0.287 0.722
11 0.276 0.549 0.295 0.473
12 0.307 0.222 0.579 0.499
13 0.433 0.323 0.699 0.454
14 0.372 0.394 0.414 0.577
15 0.458 0.579 0.594 0.376
16 0.615 0.513 0.442 0.725
17 0.521 0.489 0.546 0.238
18 0.664 0.488 0.519 0.674
19 0.531 0.738 0.291 0.455
20 0.525 0.322 0.396 0.307
21 0.575 0.376 0.689 0.322
22 0.568 0.371 0.132 0.523
23 0.436 0.436 0.436 0.453
24 0.506 0.514 0.385 0.448
25 0.332 0.434 0.471 0.485
Average 0.4726 0.48704 0.4888 0.47184 0.48007
Back

26. Summary of Paper A
Subject: Science Grade: 5 Version: 1
Q. No.
Content
strand
Chapter
No. SLO No. Key Diagram/Table
Cog.
Level P Value
Total
Rbis Item Code
Piloting
Version
Piloting Q.
No. Item Writer
1 LS 1 1.4 C Y C 0.822 0.34
SV-18-
M260 6 10
2 LS 1 1.5 B N K 0.592 0.276 256 6 6
3 LS 1 1.6 A N C 0.626 0.314 258 6 8
4 LS 1 1.10 A N C 0.152 0.296 255 6 5
5 LS 1 1.11 C N An 0.487 0.245 259 6 9
6 LS 2 2.3 B N C 0.304 0.28 297 9 42
7 LS 2 2.6 (D) D N C 0.292 0.257 313 7 13
8 LS 2 2.2 C N K 0.483 0.329 266 6 16
9 LS 3 4.1 A N K 0.526 0.364 122 3 22
10 LS 3 4.4 A N An 0.412 0.383 125 3 25
11 LS 3 4.6 B N Ap 0.276 0.196 124 3 24
12 PS 4 5.1 D N C 0.307 0.373 34 1 34
13 PS 4 5.2 A Y C 0.433 0.243 82 2 32
14 PS 4 5.4 D Y An 0.372 0.606 85 2 35
15 PS 5 6.3 A N K 0.458 0.646 91 2 41
16 PS 5 6.5 C Y C 0.615 0.622 93 2 43
17 PS 5 6.12 C Y C 0.521 0.447 141 3 41
18 PS 6 7.1 C N Ap 0.664 0.471 331 7 31
19 PS 6 7.4 D N Ap 0.531 0.525 305 7 5
20 PS 7 8.1 D Y Ap 0.525 0.649 369 8 34
21 PS 7 8.2 B N K 0.575 0.511 382 9 12
22 PS 7 8.6 C N C 0.568 0.138 423 10 18
23 PS 7 8.7 B N K 0.436 0.502 420 10 15
24 ESS 8 10.3 D N K 0.506 0.492 442 10 37
25 ESS 8 10.6 C N K 0.332 0.337 428 10 23
Back