Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Psychometrics 101:
Know what your assessment
data is telling you
Eric Ermie – Director of Client Solutions, ExamSoft
(Form...
AGENDA
• Overview
• Types of stats
• Interpreting the item analysis
report
• Examples
• General statistical guidelines
How can I reconcile what I know about my assessment’s
past with what the data is telling me?
Item analysis is not a fool p...
TYPES
OF STATS
Common Stats:
• Item Difficulty/p Value- decimal
representation of difficulty using
the percentage of stude...
ITEM ANALYSIS
REPORT
But with any statistic it is important to
remember context matters!
ITEM ANALYSIS
EXAMPLES
Diff(p) Upper A B D E
0.98 100.00% 0.10 0 1 1 *178
0.00 0.55 0.55 98.34
0.00 0.02 -0.10 0.10
0.00 0...
Diff(p) Upper A B D E
0.66 82.00% 0.28 7 17 *120 9
3.87 9.39 66.30 4.97
-0.11 -0.19 0.28 -0.07
-0.04 -0.19 0.36 -0.04
0.00...
ITEM ANALYSIS
EXAMPLES
Diff(p) Upper A B D E
0.36 52.00% 0.22 35 34 *66 25
19.34 18.78 36.46 13.81
-0.09 0.04 0.22 -0.06
-...
ITEM ANALYSIS
EXAMPLES
Diff(p) Upper A B D E
0.55 25.00% -0.43 7 17 *120 9
3.87 9.39 55.00 7.46
-0.11 -0.19 -0.43 0.00
-0....
ITEM ANALYSIS
EXAMPLES
Diff(p) Upper A B D E
0.52 64.00% 0.18 61 21 5 0
33.70 11.60 2.76 0.00
-0.10 -0.19 0.12 0.00
-0.12 ...
ITEM ANALYSIS
EXAMPLES
Diff(p) Upper A B D E
0.71 90.00% 0.31 0 *129 30 21
0.00 71.27 16.57 11.60
0.00 0.31 -0.25 -0.11
0....
GENERAL
GUIDELINES
Desired statistical range’s - opinions differ but most commonly used are:
• Item Difficulty/p Value - A...
GENERAL
GUIDELINES
Desired statistical range’s - opinions differ but most commonly used are:
• Discrimination index – some...
GENERAL
GUIDELINES
KR-20
Used as an overall measure of reliability for the assessment.
Measured on a scale from 0.0 to 1.0...
EXTRANEOUS
FACTORS
Stats alone do not tell the whole story:
• Student behavior
– Cheating
– Return on investment
• Conflic...
WHERE DO
WE FIT IN?
• Simplified and detailed versions of item analysis
reports
• Historical item analysis data by version,
assessment and in ...
Thank you for attending!
• Check our resource library:
resources.examsoft.com to re-watch
the webinar, download a PDF of t...
Questions?
Click to edit Master title style
Click to edit Master subtitle style
For More Information:
Call: 1.866.429.8889
Email: inf...
Prochain SlideShare
Chargement dans…5
×

Psychometrics 101: Know What Your Assessment Data is Telling You

560 vues

Publié le

Presented by Eric Ermie, Executive Director of Sales, ExamSoft Worldwide, Inc.

Keep it? Throw it out? Content/teaching issue? Bad question? Too easy? Too hard? What the heck? More than likely you have asked some or all of these questions at one point or another when trying to understand the performance of questions on an assessment. With differing opinions on how to interpret the statistics provided, how do you know what all this data is trying to tell you? Join us for a webinar on the fundamentals of item analysis, how the data is derived, and the different ways they can be interpreted. This presentation will cover how to put data into a useful context that will allow you to draw your own conclusions on what it means, how you should apply them, and why you should ignore rules that others may use for their specific situation.

Publié dans : Formation
  • If you’re looking for a great essay service then you should check out ⇒ www.HelpWriting.net ⇐. A friend of mine asked them to write a whole dissertation for him and he said it turned out great! Afterwards I also ordered an essay from them and I was very happy with the work I got too.
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Psychometrics 101: Know What Your Assessment Data is Telling You

  1. 1. Psychometrics 101: Know what your assessment data is telling you Eric Ermie – Director of Client Solutions, ExamSoft (Formerly) Program Manager for Assessment and Evaluation, The Ohio State University College of Medicine.
  2. 2. AGENDA • Overview • Types of stats • Interpreting the item analysis report • Examples • General statistical guidelines
  3. 3. How can I reconcile what I know about my assessment’s past with what the data is telling me? Item analysis is not a fool proof answer to these questions. But… THE OVERVIEW YOU HAVE TO START SOMEWHERE. Where do I start? Is this a good or bad question? Can statistics even tell me that?
  4. 4. TYPES OF STATS Common Stats: • Item Difficulty/p Value- decimal representation of difficulty using the percentage of students who got the item correct. The lower the decimal the higher the difficulty • Upper 27% - what percentage of the top 27% of performers got the question correct • Lower 27% - what percentage of the bottom 27% of performers got the question correct. Common Stats Cont’d: • Discrimination index – the difference in performance between the Upper 27% and the Lower 27% • Point-Biserial- a discrimination statistic that indicates whether doing well on that specific item correlated with doing well on the exam overall. Thus was that item a good or bad predictor of overall performance on the exam.
  5. 5. ITEM ANALYSIS REPORT
  6. 6. But with any statistic it is important to remember context matters!
  7. 7. ITEM ANALYSIS EXAMPLES Diff(p) Upper A B D E 0.98 100.00% 0.10 0 1 1 *178 0.00 0.55 0.55 98.34 0.00 0.02 -0.10 0.10 0.00 0.00 -0.02 0.02 0.00 0.00 0.00 1.00 0.00 0.00 0.02 0.98Lower 27% Upper 27% Disc. Index 0.00 0.00 0.00 0.00 0 0.00 Lower Disc. Index 1 % Selected Point Biserial (rpb) 96.15% E0.04 Item # Correct Responses Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) C
  8. 8. Diff(p) Upper A B D E 0.66 82.00% 0.28 7 17 *120 9 3.87 9.39 66.30 4.97 -0.11 -0.19 0.28 -0.07 -0.04 -0.19 0.36 -0.04 0.00 0.00 0.82 0.06 0.04 0.19 0.46 0.10 Lower C Item # Correct Responses Disc. Index Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) 0.36 Lower 27% Upper 27% Disc. Index -0.09 0.21 0.12 Point Biserial (rpb) 46.15% D 28 15.47 -0.12 7 % Selected ITEM ANALYSIS EXAMPLES
  9. 9. ITEM ANALYSIS EXAMPLES Diff(p) Upper A B D E 0.36 52.00% 0.22 35 34 *66 25 19.34 18.78 36.46 13.81 -0.09 0.04 0.22 -0.06 -0.15 0.07 0.25 -0.02 0.10 0.24 0.52 0.10 0.25 0.17 0.27 0.12 Item # Correct Responses Disc. Index Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) Lower C 0.25 Lower 27% Upper 27% Disc. Index -0.15 0.19 0.04 Point Biserial (rpb) 26.92% D 21 11.60 -0.20 22 % Selected
  10. 10. ITEM ANALYSIS EXAMPLES Diff(p) Upper A B D E 0.55 25.00% -0.43 7 17 *120 9 3.87 9.39 55.00 7.46 -0.11 -0.19 -0.43 0.00 -0.04 -0.19 -0.57 0.00 0.00 0.00 0.25 0.00 0.00 0.00 0.83 0.00 Lower C Item # Correct Responses Disc. Index Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) -0.57 Lower 27% Upper 27% Disc. Index -0.09 0.17 0.75 Point Biserial (rpb) 82.50% D 28 37.54 -0.12 82 % Selected
  11. 11. ITEM ANALYSIS EXAMPLES Diff(p) Upper A B D E 0.52 64.00% 0.18 61 21 5 0 33.70 11.60 2.76 0.00 -0.10 -0.19 0.12 0.00 -0.12 -0.13 0.04 0.00 0.26 0.04 0.06 0.00 0.38 0.17 0.02 0.00 Item # Correct Responses Disc. Index Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) Lower C 0.22 Lower 27% Upper 27% Disc. Index 0.22 0.42 0.64 Point Biserial (rpb) 42.31% C *94 51.93 0.18 24 % Selected
  12. 12. ITEM ANALYSIS EXAMPLES Diff(p) Upper A B D E 0.71 90.00% 0.31 0 *129 30 21 0.00 71.27 16.57 11.60 0.00 0.31 -0.25 -0.11 0.00 0.34 -0.23 -0.09 0.00 0.90 0.06 0.04 0.00 0.56 0.29 0.13 Item # Correct Responses Disc. Index Point Biserial Correct Answer Response Frequencies (*Indicates correct answer) Lower C 0.34 Lower 27% Upper 27% Disc. Index -0.02 0.02 0.00 Point Biserial (rpb) 55.77% B 1 0.55 -0.16 34 % Selected
  13. 13. GENERAL GUIDELINES Desired statistical range’s - opinions differ but most commonly used are: • Item Difficulty/p Value - Acceptable item difficulty is not a set number but more a correlation with question intention. If you intended the item to be a mastery item you want the difficulty as close to 1.00 as possible. If you desired a discriminating question significantly lower levels are acceptable. • Upper 27% - if less than 60% of your top performers are getting a question correct a further analysis is needed to see if there are issues with the question. Also if less of your upper 27% get a question correct than your lower 27% then there is also an issue. • Lower 27% - generally you never want it to be higher than the upper 27%. As low as 0% can be acceptable as high as 100% can be acceptable if it is a mastery question.
  14. 14. GENERAL GUIDELINES Desired statistical range’s - opinions differ but most commonly used are: • Discrimination index – some set specific numbers of acceptable and unacceptable values, I would argue the more accurate guide is that the lower the p value the higher the discrimination index needs to be. Generally .2 the item is considered to have discriminated, less than that is considered no discrimination. .3 or greater is consider highly discriminating. • Point-Biserial – similarly to discrimination index some set specific numbers of acceptable and unacceptable values. Generally .2 and above is considered to have discrimination and have positive association with overall performance on the assessment, lower levels are acceptable for mastery and .3+ would be desired for discriminating questions.
  15. 15. GENERAL GUIDELINES KR-20 Used as an overall measure of reliability for the assessment. Measured on a scale from 0.0 to 1.0 with 0.0 being very poor and 1.0 being excellent. Quick notes: Heavily influenced by number of questions in assessment Heavily influenced by number of students taking the assessments The combination can FREQUENTLY lead to false positive and false negative KR-20 values.
  16. 16. EXTRANEOUS FACTORS Stats alone do not tell the whole story: • Student behavior – Cheating – Return on investment • Conflicting content/faculty • “six degrees from Sunday” Ways to increase the accuracy/usefulness of your stats: • Item review process – Format – Level of difficulty – Alternative correct options • Historical item analysis – Across assessments – Across versions • Reuse/Recycle
  17. 17. WHERE DO WE FIT IN?
  18. 18. • Simplified and detailed versions of item analysis reports • Historical item analysis data by version, assessment and in aggregate • Ability to pull item analysis by discipline/question author/category EXAMSOFT FIT THE DATA YOU NEED
  19. 19. Thank you for attending! • Check our resource library: resources.examsoft.com to re-watch the webinar, download a PDF of the presentation or access a certificate of completion. • Be sure to check out our upcoming webinars: • Creating a Secure Testing Environment for Distance Education Programs • Learning about the Learners: Using Analytical Tools to Drive Curricular Decisions
  20. 20. Questions?
  21. 21. Click to edit Master title style Click to edit Master subtitle style For More Information: Call: 1.866.429.8889 Email: info@examsoft.com Visit: learn.examsoft.com

×