Presentation at ICL 2008

1. Can Confidence Assessment
Enhance Traditional
Multiple-Choice Testing?
Josef Kolbitsch, Martin Ebner, Walther Nagler, Nicolai Scerbakov

3. Technology enhanced Learning
http://flickr.com/photos/powerbooktrance/289992273//

4. uLearning
http://www.flickr.com/photos/sovietuk/141381675

5. „Students sometimes accomplish their results
of a Mutiple-Choice Test by a combination of
partial knowledge and guesswork.“

6. Confidence Assessment
„Students do not only have to choose an
answer to a multiple-choice question, but also
have to indicate how confident they feel
about their given answer.“

7. additional confidence paramter

8. Lecture Design
- lecture „Application of Operating and
Information Systems“
- recommended to all students of Graz
University of Technology
- 720 students attend
- 432 did the final exam

9. Test Design
100 questions were generated with
following influence by confidence:

10. Interpretation of the parameter
Points if Points if
Confidence Interpretation
correct incrorrect
100% I know it 10 -5
I am not sure, but I think
66% 7 -3
I know it
I am not sure, but I
33% 3 -2
assume something
I did not know it and I
0% 0 0
have to guess

11. Test Groups
- Control group (260 students)
- Experimental group with Confidence
(172 students)
- Experimental group without
Confidence (172 students)
- Experimental group scaled (172
students)

12. Overview about groups
Minimum Maximum Range of
Type of Test
Score Score Points
Experimental group with
-50 100 150
Confidence
Experimental group
0 100 100
without Confidence
Experimental group scaled 0 100 100
Control group 0 100 100

13. General findings
$!!quot;#
,!quot;#
+!quot;#
!quot;#$quot;%&'(quot;)*+),'-./0/)1$*#quot;)
*!quot;#
)!quot;#
:72;9<2#=90>3.0?.#
(!quot;# :72;#=90>3.0?.#
:72;#=90>3.0?.@#1?/A.3#
'!quot;# =90249A#B49<C#
&!quot;#
%!quot;#
$!quot;#
!quot;#
-./0# 12/03/43#5.67/890# -.37/0# -93.#

14. Is there a difference between the results of the MCT with
confidence assessment and the result of the Control
Group?
)!quot;!!#$
/012$345678598$
(%quot;!!#$
/012$345678598quot;$:9;<87$
(!quot;!!#$ /0124=1$345678598$
3451>4<$?>4=@$
!quot;#$quot;%&'(quot;)*+),&-.quot;%&/)
'%quot;!!#$
'!quot;!!#$
&%quot;!!#$
&!quot;!!#$
%quot;!!#$
!quot;!!#$
!*&!$#$ &&*'!$#$ '&*(!$#$ (&*)!$#$ )&*%!$#$ %&*+!$#$ +&*,!$#$ ,&*-!$#$ -&*.!$#$ .&*&!!$#$
!quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)

15. Is there a difference between the scaled results of MCT
with confidence assessment and the result of the Control
Group?
+!quot;!!#$
./01$234567487$
*!quot;!!#$
./01$234567487quot;$98:;76$
./013<0$234567487$
)!quot;!!#$
2340=3;$>=3<?$
!quot;#$quot;%&'(quot;)*+),&-.quot;%&/)
(!quot;!!#$
'!quot;!!#$
&!quot;!!#$
%!quot;!!#$
!quot;!!#$
!,&!$#$ &%,(!$#$ (%,*!$#$ *%,-!$#$ -%,%!!$#$
!quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)

16. Is there a relationship between the confidence parameter
and the correctness of the answer?
$!!quot;#
,!quot;#
+!quot;#
*!quot;#
!quot;#$quot;%&'(quot;)*+)/%$*##quot;$&),%-.quot;#-)
!quot;#$quot;%&'(quot;)*+),%-.quot;#-)
)!quot;#
(!quot;#
'!quot;#
&!quot;#
%!quot;# -./012/32#456728#
$!quot;# 9/3.::23;#</8=2:8#
!quot;#
$!!quot;# ))quot;# &&quot;# !quot;#
0*%12quot;%$quot;)3'45quot;-)

17. Discussion
- Result of Experimental Group with
Confidence are slightly worse than
those of Control Group
- Result of Experimental Group scaled are
slightly better than those of Control
Group
- Relationship between confidence
parameter and percentage of wrong
answers are highly relevant

18. „By adding a
confidence
parameter
to a MCT
the result
of the MCT
deteriorates“

19. SOCIAL LEARNING
Computer and Information Services
Graz University of Technology
Graz University of Technology
Martin Ebner
http://elearning.tugraz.at
http://elearningblog.tugraz.at