From gut feeling to a structured, summative assessment of design competencies

Research Center Creating 010
Rotterdam University of Applied Sciences

From Gut Feeling To A
Structured, Summative
Assessment of Design
Competencies

Jasper Schelling, Bas Leurs,
Saskia Best and Ingrid Mulder

Antwerp, September 6 2012

Photography by Roland Pastoor

We introduce a summative method for
assessing the whole set of students’ design
competencies demonstrated in their ﬁnal
design project when graduating an interactive
media design course, and reﬂect on the use
of this assessment method.

A ﬁnal design project is often used to give
students the opportunity to demonstrate
whether they have obtained the right level
of expertise to graduate.

The level of expertise that should be acquired
by the student is often formulated as a
competency, a combination of speciﬁc
professional capacities:

Attitude

Skills

Knowledge

(Bakerman, 2005)

In our own observations, however, we have
seen many assessors struggling with lists of
ambiguous criteria used to assess the students’
competencies; it seemed that experienced
assessors often rely on their intuition or
‘gut feeling’ to grade the students, and are
not stimulated to make their choices explicit.

At the same time, recent developments in
the accreditation of higher education in
The Netherlands demand a more transparent
assessment and grading system.

“Zijlstra makes hard
intervention in higher education”
(May 20, 2011)

“Stricter supervision in higher education”

(February 3, 2012)

The ﬁnal design project consists of two elements:

Investigating state of the art
theory and knowledge. Provides
Thesis the foundation and rationale for
the design project.

A real life situation at a
company in which the students
Designproject
have to demonstrate their
competencies.

The ﬁnal assessment is a one hour session
that consists of four speciﬁc stages:

Presentation Interview Deliberation Feedback
(10 min) (20 min) (20 min) (10 min)

The students are graded on eleven criteria,
and are scored on four levels:
Research
State-of-the-art Knowledge Excellent
Thesis
Argumentation
Writing Skills and Presentation

Concept
Good
Visualization
Product Delivery
Satisfactory
Designrationale & Emotionale
‘Human Centered’ Design

Professionalism Multidisciplinary Working Unsatisfactory
& Presentation Presentation & Reﬂection

The assessment tool consists of four instruments:

Calculation Aid

Assessment Form

Question List

Manual

-".$%.',#/012#33%/,&.',#/1415%6',3*$,.17*+,8/ Institute for Communication,
Media and Information Technology
:'%$*/'/%3<*"01 =.3*0 7.'*0 study programme:
Communication and
Multimedia Design
()*+,+
G*+*."&) 9 : - ; 2#33*/'+0
@A3"B,+C43D"03)($('(+$"(%"*43&,E"'A3",3%3&,*A"FG3%'(+$"(%"H,+G$030"($"*+$'3DB+,&,-"(%%G3%E"A&%"&"
*43&,")+*G%"&$0"(%",343I&$'E"'A3",3%3&,*A"($'+"4('3,&'G,3"(%"0(I3,%3E"'A3"D3'A+0+4+H-"(%"*43&,4-"
03%*,(C30E"'A3",3%3&,*A"A&%"430"'+"&"*+$*3B'G&4"D+034E"'A3",3%3&,*A"FG3%'(+$"(%"&$%J3,30"*43&,4-
:'.'*H#IH')*H."'1F/#J6*$8* 9 : - ;
@A3"%'G03$'"A&%"G%30"%'&'36+)6'A36&,'"K$+J430H3"&$0"03D+$%',&'3%"&$"G$03,%'&$0($H"+)"'A3"
0(%*(B4($3L%"C+0-"+)"K$+J430H3E"A&%"J3446)+G$030"+B($(+$%"*+$*3,$($H"('E"&$0"A&%"*&B('&4(%30"+$"
+BB+,'G$('(3%"&$0"',3$0%M
L"8%3*/'.',#/ 9 : - ;
@A3"&,HGD3$'&'(+$"A&%"&"4+H(*&4"%',G*'G,3"&$0"%A+J%"B,+)+G$0$3%%E"&$0"%'&$0B+($'%"+)"+'A3,%"&,3"
,3*+H$(%30E"32B,3%%30"&$0"*,('(*&44-"32&D($30M
K",',/81+F,66+1./$1>"*+*/'.',#/ 9 : - ;
.',G*'G,3E"%'-43E"%B344($HE")++'$+'3%94('3,&'G,3",3)3,3$*3%E"FG&4('-E"4&-+G'"&$0")($(%A($HM

!"#$%&'
2#/&*>' 9 : - ; 2#33*/'+0

V3e
@A3"*+$*3B'"(%"+,(H($&4"&$0"($$+I&'(I3E"'A3,3"(%"&"*43&,"*+A3,3$*3"C3'J33$"'A3"'A3%(%"&$0"'A3"
B,+0G*'"'A&'"*&$"C3"',&*30"C&*K"'+",3%3&,*A"($'+"4('3,&'G,39,3)3,3$*3%"&$0"03%(H$",3%3&,*AE"&$0"'A3"

BF03.GS-BLS.NP-14jan2012
B,+0G*'",3%B+$0%"'+"'A3"J(%A3%"+)"C+'A"'A3"'&,H3'"H,+GB"&$0"'A3"*4(3$'M
M,+%.6,+.',#/ 9 : - ;
@A3"*+$*3B'"(%"D&03"N'&$H(C43OE"),+D"&$"($('(&4",+GHA"%K3'*A"'+"&",3)($30"03%(H$M"P+$I($*($H"
I(%G&4(%&'(+$%E"%K3'*A3%"&$0"+'A3,"'3*A$(FG3%"&,3"G%30"'+",3B,3%3$'"&$0"%A&,3"(03&%E"B,+C43D%"&$0"
,3%G4'%M
7*6,?*"C 9 : - ;
@A3"B,+*3%%"A&%"430"'+"'A3"*,3&'(+$"+)"&"B,+'+'-B3"+)"&$"($'3,&*'(I3"B,+0G*'"'A&'"A&%"C33$"'3%'30"($"
B,&*'(*3M"@3%'",3%G4'%"A&I3"430"'+"&0&B'&'(+$%"&$0",3*+DD3$0&'(+$%M
G.',#/.6141;3#',#/.6 9 : - ;
@A3"%'G03$'"*&$"%GC%'&$'(&'3"03%(H$"03*(%(+$%E"*43&,4-"03D+$%',&'3"'A3"*+A3,3$*3"C3'J33$"
,3%3&,*A"&$0"B,+0G*'E"&$0",3I(3J"'A3"*+$%3FG3$*3%"+)"B3,%+$&4"03%(H$"03*(%(+$%M
NO%3./12*/'*"*$N1$*+,8/ 9 : - ;
@A3"%'G03$'"%A+J%"&C(4('-"'+"3DB&'A(Q3E"A&%"BG'"A(D"+,"A3,%34)"($"'A3"B+%('(+$"+)"'A3"'&,H3'"H,+GB"
&$0"+'A3,"%'&K3A+403,%E"A&%"&*'(I34-"($I+4I30"'A3"'&,H3'"H,+GB"&$0"+'A3,"%'&K3A+403,%"($"'A3"03%(H$"
B,+*3%%E"A&%"+C%3,I30"J344"&$0"0,&J$"*+$*4G%(+$%"),+D"'A(%M

!"#I*++,#/.6,+3141!"*+*/'.',#/
5%6',$,+&,>6,/."C1J#"F,/8 9 : - ; 2#33*/'+0
@A3"%'G03$'"(%"&C43"'+"*+44&C+,&'3E"+,H&$(%3E"*+DDG$(*&'3"&$0"B&,'(*(B&'3E"&$0"03D+$%',&'3%"3))+,'"
)+,"&$0"*+44&C+,&'(+$"J('A"*+443&HG3%E"'A3"*4(3$'"&$0"+'A3,"%'&K3A+403,%M
!"*+*/'.',#/141G*I6*&',#/ 9 : - ;
@A3"B,3%3$'&'(+$"A&%"&"*43&,"%',G*'G,3"&$0"(%"%GBB+,'30"I(%G&44-E"'A3"FG&4('-"+)"'A3"&$%J3,%"(%"
*+$I($*($HE"'A3"%'G03$'"(%"&C43"'+",3)43*'"+$"A(%"+,"A3,"B3,%+$&4"03I34+BD3$'E"A&%"+B3,&'30"($036
B3$03$'4-E"A&%"(03$'()(30"&$0"G'(4(%30"43&,$($H"+BB+,'G$('(3%E"&$0"A&%"J+,K30"+$"B3,%+$&4"03I34+B6
D3$'"&$0"'A3"&*FG(%('(+$"+)"$3J"K$+J430H3"&$0"G$03,%'&$0($HM

:%>*"?,+#"1@&).,"A :%>*"?,+#"1@$*'.&)*$A :%>*"?,+#"1@&#3>./CA ;D'*"/.61*D>*"' E,/.613."F
>565=?

=.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"*
!"#"!$%&'(%)&*'+,-"."#".&'(%)&*'+,-"/"#"/++0"1"#"12*3443$'""""""""567"#"!$%&'(%)&*'+,-"89:"#".&'(%)&*'+,-";"#"/++0"<95="#"12*3443$'

-".$%.',#/012#33%/,&.',#/1415%6',3*$,.17*+,8/ Institute for Communication,
Media and Information Technology
:'%$*/'/%3<*"01 =.3*0 7.'*0 study programme:
Communication and
077234 Jeroen Stappenmaecker June 16, 2011 Multimedia Design
()*+,+ ;D.3>6*1#I1&.6&%6.',#/
G*+*."&) 9 : - ; 2#33*/'+0
@A3"B,+C43D"03)($('(+$"(%"*43&,E"'A3",3%3&,*A"FG3%'(+$"(%"H,+G$030"($"*+$'3DB+,&,-"(%%G3%E"A&%"&"
*43&,")+*G%"&$0"(%",343I&$'E"'A3",3%3&,*A"($'+"4('3,&'G,3"(%"0(I3,%3E"'A3"D3'A+0+4+H-"(%"*43&,4-"
()*+,+
If an item has been graded with
03%*,(C30E"'A3",3%3&,*A"A&%"430"'+"&"*+$*3B'G&4"D+034E"'A3",3%3&,*A"FG3%'(+$"(%"&$%J3,30"*43&,4- ‘unsatisfactory’, the student
9 9/+.',+I.&'#"C does not pass!
:'.'*H#IH')*H."'1F/#J6*$8* 9 : - ;
@A3"%'G03$'"A&%"G%30"%'&'36+)6'A36&,'"K$+J430H3"&$0"03D+$%',&'3%"&$"G$03,%'&$0($H"+)"'A3"
0(%*(B4($3L%"C+0-"+)"K$+J430H3E"A&%"J3446)+G$030"+B($(+$%"*+$*3,$($H"('E"&$0"A&%"*&B('&4(%30"+$"
: :.',+I.&'#"C 2 P1QR11S1 3 0
+BB+,'G$('(3%"&$0"',3$0%M - -##$ 2 P1TU11S1 4 0
L"8%3*/'.',#/ 9 : - ;
@A3"&,HGD3$'&'(+$"A&%"&"4+H(*&4"%',G*'G,3"&$0"%A+J%"B,+)+G$0$3%%E"&$0"%'&$0B+($'%"+)"+'A3,%"&,3" ; ;D&*66*/' P1TR11S1 +
,3*+H$(%30E"32B,3%%30"&$0"*,('(*&44-"32&D($30M
K",',/81+F,66+1./$1>"*+*/'.',#/
.',G*'G,3E"%'-43E"%B344($HE")++'$+'3%94('3,&'G,3",3)3,3$*3%E"FG&4('-E"4&-+G'"&$0")($(%A($HM
9 : - ;
7 0 P1V11S 2 8 0 B
!"#$%&'
!"#$%&' If an item has been graded with
2#/&*>' 9 : - ; 9 9/+.',+I.&'#"C
2#33*/'+0 ‘unsatisfactory’, the student

V3e
does not pass!
@A3"*+$*3B'"(%"+,(H($&4"&$0"($$+I&'(I3E"'A3,3"(%"&"*43&,"*+A3,3$*3"C3'J33$"'A3"'A3%(%"&$0"'A3"
B,+0G*'"'A&'"*&$"C3"',&*30"C&*K"'+",3%3&,*A"($'+"4('3,&'G,39,3)3,3$*3%"&$0"03%(H$",3%3&,*AE"&$0"'A3" : :.',+I.&'#"C 2 P1QT11S1 2 4

BF03.GS-BLS.NP-14jan2012
B,+0G*'",3%B+$0%"'+"'A3"J(%A3%"+)"C+'A"'A3"'&,H3'"H,+GB"&$0"'A3"*4(3$'M
M,+%.6,+.',#/ 9 : - ;
- -##$ 2 P1QW11S1 3 2
@A3"*+$*3B'"(%"D&03"N'&$H(C43OE"),+D"&$"($('(&4",+GHA"%K3'*A"'+"&",3)($30"03%(H$M"P+$I($*($H"
I(%G&4(%&'(+$%E"%K3'*A3%"&$0"+'A3,"'3*A$(FG3%"&,3"G%30"'+",3B,3%3$'"&$0"%A&,3"(03&%E"B,+C43D%"&$0"
; ;D&*66*/' 1 P1TU11S1 2 0 +
,3%G4'%M
7*6,?*"C
@A3"B,+*3%%"A&%"430"'+"'A3"*,3&'(+$"+)"&"B,+'+'-B3"+)"&$"($'3,&*'(I3"B,+0G*'"'A&'"A&%"C33$"'3%'30"($"
9 : - ; 7 6 P1V11S 3 0 4 B
B,&*'(*3M"@3%'",3%G4'%"A&I3"430"'+"&0&B'&'(+$%"&$0",3*+DD3$0&'(+$%M
!"#I*++,#/.6,+3141!"*+*/'.',#/
G.',#/.6141;3#',#/.6 9 : - ; If an item has been graded with
@A3"%'G03$'"*&$"%GC%'&$'(&'3"03%(H$"03*(%(+$%E"*43&,4-"03D+$%',&'3"'A3"*+A3,3$*3"C3'J33$" 9 9/+.',+I.&'#"C ‘unsatisfactory’, the student
,3%3&,*A"&$0"B,+0G*'E"&$0",3I(3J"'A3"*+$%3FG3$*3%"+)"B3,%+$&4"03%(H$"03*(%(+$%M does not pass!

NO%3./12*/'*"*$N1$*+,8/ 9 : - ; : :.',+I.&'#"C P1XU11S1
@A3"%'G03$'"%A+J%"&C(4('-"'+"3DB&'A(Q3E"A&%"BG'"A(D"+,"A3,%34)"($"'A3"B+%('(+$"+)"'A3"'&,H3'"H,+GB"
&$0"+'A3,"%'&K3A+403,%E"A&%"&*'(I34-"($I+4I30"'A3"'&,H3'"H,+GB"&$0"+'A3,"%'&K3A+403,%"($"'A3"03%(H$" - -##$ 1 P1VU11S1 4 0
B,+*3%%E"A&%"+C%3,I30"J344"&$0"0,&J$"*+$*4G%(+$%"),+D"'A(%M
; ;D&*66*/' 1 P1RU11S1 5 0 +
!"#I*++,#/.6,+3141!"*+*/'.',#/
5%6',$,+&,>6,/."C1J#"F,/8 9 : - ; 2#33*/'+0 9 0 P1T11S 1 8 0 B
@A3"%'G03$'"(%"&C43"'+"*+44&C+,&'3E"+,H&$(%3E"*+DDG$(*&'3"&$0"B&,'(*(B&'3E"&$0"03D+$%',&'3%"3))+,'"
)+,"&$0"*+44&C+,&'(+$"J('A"*+443&HG3%E"'A3"*4(3$'"&$0"+'A3,"%'&K3A+403,%M
!"*+*/'.',#/141G*I6*&',#/
@A3"B,3%3$'&'(+$"A&%"&"*43&,"%',G*'G,3"&$0"(%"%GBB+,'30"I(%G&44-E"'A3"FG&4('-"+)"'A3"&$%J3,%"(%"
9 : - ; E,/.613."F 7 6 4 / 100 = 7, 6 4
*+$I($*($HE"'A3"%'G03$'"(%"&C43"'+",3)43*'"+$"A(%"+,"A3,"B3,%+$&4"03I34+BD3$'E"A&%"+B3,&'30"($036
B3$03$'4-E"A&%"(03$'()(30"&$0"G'(4(%30"43&,$($H"+BB+,'G$('(3%E"&$0"A&%"J+,K30"+$"B3,%+$&4"03I34+B6
D3$'"&$0"'A3"&*FG(%('(+$"+)"$3J"K$+J430H3"&$0"G$03,%'&$0($HM E,/.613."F1@"#%/$*$A

:%>*"?,+#"1@&).,"A :%>*"?,+#"1@$*'.&)*$A :%>*"?,+#"1@&#3>./CA ;D'*"/.61*D>*"'
8
E,/.613."F
>565=?

Els Karreman Niels de Worp Bernadette Delfgauw
=.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"* =.3*1B1:,8/.'%"*
!"#"!$%&'(%)&*'+,-"."#".&'(%)&*'+,-"/"#"/++0"1"#"12*3443$'""""""""567"#"!$%&'(%)&*'+,-"89:"#".&'(%)&*'+,-";"#"/++0"<95="#"12*3443$'

During the assessments of the ﬁnal
projects we observed assessor teams
when they deliberated on the assessment
of competencies and discussed the
resulting ﬁnal grade.

After the assessment, the use of the
instruments was evaluated with assessors from
the teaching staff. They were asked to give
feedback on how the instruments supported
them in grading the graduation work and
what their experiences were using this tool.

Observations during the deliberation phase
of the ﬁnal assessment showed that assessors
were less likely to rely on their general
impression of the work and the presentation.
Instead, they were actively discussing the
students’ results on the criteria set forth in
the assessment form.

When the assessors finished their discussion
and asked the student to reappear before the
committee to receive the final result, the
committee was better prepared to link
their feedback to the criteria set forth in the
assessment and seemed to be more confident
motivating the resulting final grade.

In general, the assessors mentioned that the
discussion during the grading was about the
criteria set forth in the assessment form and
not about the ﬁnal grade.

Although the assessment instrument was
helpful in assessing the design competencies,
some assessors indicated that the ﬁnal grade
did not match with their 'gut feeling'.

A piece of visualization software was written
to visualize the scores of the assessments for
each student. This visualization was created
for each of the 58 students.

These visualizations in turn provide a
fertile ground for continuing discussion on
assessment procedures, for individual students
as well as for the curriculum in general.

Being able to visually compare the outcomes
of various assessments allows us to gain
insight into the different assessments and
to ﬁnd patterns as well as points of interest,
which inform further course development.

The current study demonstrated that structured
assessment instruments have dual purposes:

Assessing the quality of the
work of the students in a
Hollistic Assessment manner ﬁtting with competence
based education.

Improving the Supporting the explicit motivation
discussion among of the nuances of the speciﬁc
design competencies.
assessors

A side effect of revealing the tacit expertise
of experienced assessors was that the junior
assessors felt they improved their
understanding of the competencies and
associated criteria and proﬁciency level.

This seems to be beneficial for the students
as well. Usually students complained about
perceived subjectivity of the final results;
interestingly, the current cohort hardly
discussed their final grades.

All in all, it can be concluded that structured
assessment instruments, when properly
designed, help assessors to distinguish
between a ‘gut feeling’ and their intuition,
as well show the relative merit of a speciﬁc
competency and matching proﬁciency level.

Questions?

Contact Information Jasper Schelling
Research Center Creating 010 j.a.schelling@hr.nl
Rotterdam University of
Bas Leurs
Applied Sciences
b.l.f.leurs@hr.nl
creating010@hr.nl

From gut feeling to a structured, summative assessment of design competencies

Recommandé

Recommandé

Contenu connexe

Similaire à From gut feeling to a structured, summative assessment of design competencies

Similaire à From gut feeling to a structured, summative assessment of design competencies (20)

Dernier

Dernier (20)

From gut feeling to a structured, summative assessment of design competencies