How do Faculty Ensure Student Competency at Course Completion?

1. How do Faculty Ensure
Student Competency at
Course Completion?
Department of Engineering Fundamentals
Michigan Technological University
Gretchen Hein, Mary Fraley, Amber Kemppainen, AJ
Hamlin
Friday, October 28, 2016, 3:50PM, PCC, 113c

2. Outline
• What is “Competency”?
• First‐Year Engineering Program
• Competency Assessments in ENG1100 and
ENG1102
• Competency Data and Analysis
• Conclusions

3. What is “Competency”?
Assessment
• Students are “tested”
• Some will succeed and
others will fail
• There is no barrier to
continuing if they fail
Competency
• Student demonstrates
the ability to complete a
given set of tasks
• Definition: Cannot move
to next task until they
have proven their ability
• Michigan Tech: Grade
consequences for not
mastering skills, but can
continue to next task

4. First‐Year Engineering Program
Calculus Ready
Path
(2 semesters)
ENG1101
(Engineering
Analysis and
Problem Solving)
ENG1102
(Engineering
Modeling and
Design)
Pre‐
Calculus/College
Algebra Path
(3 semesters)
ENG1001
(Engineering
Analysis)
ENG1100
(Engineering
Problem Solving)
ENG1102
(Engineering
Modeling and
Design)
• Common program for all
entering engineering
students
• Paths:
1. Calculus‐Ready:
a. ENG1101
b. ENG1102
2. Pre‐Calculus and
Algebra Ready:
a. ENG1001
b. ENG1100
c. ENG1102

5. Courses where Competency was
assessed
• ENG1100
• Units and unit conversions (Week 3 out of 14)
• Basic MATLAB programing (Week 8 out of 14)
• Logistics:
• 35 minutes to complete
• Could repeat until they earned at least an 80% or earn a C in the
course
• ENG1102
• 3D modeling (NX) (Week 7 out of 14)
• MATLAB programing (Week 13 out of 14)
• Logistics:
• 30 minutes to complete
• Could retake it once if they scored less than 88%
• Retake grade was capped at 88%

6. ENG1100 Competency Tests
Units (on paper)
• Work out problems
• 3 questions on unit
conversions
• Length conversion
• Areal or volume conversion
• Mixed unit conversion (e.g.:
mg/L, N∙m)
• 2 questions of units in
equations
• Unit homogeneity
• Find the missing units on
one variable in the equation
MATLAB
(Programming)
• On a computer
• Import and use a data set
• Use a “for” loop
• Count occurrences
• Use built‐in functions (e.g.
min, max)
• Output a plot and/or table

7. ENG1100 Competency Test
Results (n = 63)
• All students who
could retake either or
both tests, completed
at least one re‐take
• Highest score was
recorded
• 100% was possible on
retakes
• Overall averages
improved as number
of attempts increased
Units MATLAB
# Attempts Possible 5 3
Average, 1st Attempt 76.6% 73.4%
Average, Last Attempt 87.4% 88.4%
Table 1. ENG1100 Competency Test
Performance

8. ENG1100 Competency Test Results
• Approximately 50% of
students were
competent after first
attempt
• Increased to 80% after
second attempt
• Less than 10% did not
master the skills (6% for
Units and 10% for
MATLAB)
• 1 student did not master
either skill
0
10
20
30
40
50
60
70
1 2 3 4 5
Number of Students
Attempt
Figure 1. Number of ENG1100
Students Achieving 80% Threshold
Units MATLAB

9. ENG1102 Competency Tests
(Completed on a computer)
MATLAB
(Programming)
• Problem 1:
• Accepted keyboard input
• Created and called a user
defined function
• Completed calculations
• Formatted output statements
• Problem 2:
• Import and use a data set
• Use a “while” loop
• Complete calculations
• Graph data
• Format output
NX (3D Modeling)
• Problem 1:
• Sketch‐ and feature‐based
modeling
• Fully constrained geometry
and dimensions
• Geometry with linked
dimensions and other
constraints
• Problem 2:
• Use pattern and mirror
features along with datum
planes to create a part
• Problem 3:
• Assemble and constrain
multiple parts to create a
larger model

10. ENG1102 Competency Test Results
• Overall averages
improved with
retake
• Maximum grade on
the retake was 88%
NX MATLAB
Number of Sections 5 (out of 13)
# Attempts Possible 2
Average, 1st Attempt 88.8% 81.0%
Average, Last Attempt 91.7% 84.9%
Table 2. ENG1102 Competency Test
Performance (n = 257)

11. ENG1102 Competency Test Results
• Scores < 80% who
chose not to retake:
• NX: 4/76 (5%)
• MATLAB: 37/155 (24%)
• 80% < Score < 88% who
chose to retake:
• NX: 5/36 (14%)
• MATLAB: 16/58 (28%)
• Most students in this
range did not complete
the retake
NX MATLAB
Average, 1st Attempt 88.8% 81.0%
Average, Last Attempt 91.7% 84.9%
1st Attempt, Scored < 88%
76/257
(30%)
155/257
(60%)
1st Attempt, 80 < Score < 88
36/257
(14%)
58/257
(22.6%)
Completed 2nd Attempt
33/76
(43%)
60/155
(39%)
Table 3. ENG1102 Competency Test
Attempt Summary (n = 257)

12. ENG1102 Competency Test Results
• Approximately 70%
of students were
competent (>88%)
after first
assessment
• Increased to 80%
after retake 0
50
100
150
200
250
NX MATLAB
Number of Students
Competency Test
Figure 2. ENG1102 Students who
Were Competent after Each
Attempt (n = 257)
1st Attempt 2nd Attempt

13. ENG1102 Competency Test Results
NX MATLAB Both
<88% 57 (22%) 125 (49%) 42 (16%)
• Students perform
better on NX than
MATLAB
• Some students
(16%) find NX and
MATLAB difficult to
master
Table 4. ENG1102 Students who Did
not achieve the Threshold of 88% on
the Competency Tests (n = 257)

14. Competency Tests
Disadvantages
• Time consuming
• Different tests for each
attempt
• Proctoring each attempt
• Assessing each attempt
• Limited time within the
semester to administer
tests
• MATLAB was given in week
13 and retake was during
week 14 (end of the
semester)
• Limited computer
resources
Advantage
• More students leave a
program with skill mastery

15. Conclusions
• Competency Assessments:
• More attempts and no cap on final test grade,
encourages students to retake the tests (ENG1100)
• Threshold grade on test (ENG1102, 88%):
• Students close to the threshold chose not to retake
• Students performing much below the threshold chose to
retake
• Both models encourage students to retake to improve
their grade (which also increases their competency
on that topic)
• Questions: Contact Gretchen Hein
(glhein@mtu.edu)

16. WE16 Summary
How do Faculty Ensure Student Competency at Course Completion?
At Michigan Technological University, faculty within the first‐year engineering program are beginning to go from assessing
student knowledge to determining if students are competent in a given skill. The first‐year engineering program consists of a
calculus‐ready track (2 course sequence, ENG1101 followed by ENG1102) and a pre‐calculus‐ready track (3 course sequence,
ENG1001, ENG1100 and ENG1102). The purpose of the program is for students to leave the program with the necessary
technical, communication and programming/modelling skills required to be a successful engineering student.
In two courses, ENG1100 and ENG1102, students were given competency assessments in the Spring 2016 semester. In both
courses, students were given the opportunity to repeat the assessments if they did not meet the competency threshold on
their first attempt.
Students in ENG1100 were given two competency assessments, one covering units and the other basic MATLAB programming.
The units competency assessment involved students solving problems on paper that ranged from basic unit conversions to
homogeneous units in equations. The basic MATLAB programming competency required students to write a short program
using MATLAB in a computer lab. The program included a loop and selection structure. Students had 35 minutes to complete
their program. For the ENG1100 competency assessments, students were required to retake a new version of the assessment
if they earned less than 80%. Students needed to achieve the target threshold of 80% on each assessment, or the highest
grade they could earn in the course was a C. Students were able to repeat the assessment until they achieved the threshold 0f
80%.
The ENG1102 competency assessments were given to students in 3D modeling (NX) and programming (MATLAB). The 3D
modeling assessment was given in a computer lab. Students were given 30 minutes to complete three separate tasks using NX.
These tasks included creating and fully constraining a basic shape, using the pattern and mirror tools with reference geometry
to modify a separate part, and properly constrain two parts together in an assembly. In the programming assessment, students
completed two short programs using MATLAB as the programming language. The programs were more advanced than those in
the ENG1100 competency tests and included indefinite loops and functions. Students had 30 minutes to complete both
programs.
While still using the competency threshold of 80%, the ENG1102 competency process allowed students to re‐take the
assessment if their score was anything lower than 88%. If a student completed the re‐take, they could not earn a grade
exceeding 88% even if their actual re‐take performance exceeded that level. This was to encourage students who did not
perform well to improve their grade, and not “penalize” the students who had already performed well. The data were
analyzed
During this presentation, data will be presented to show how successful the competency assessment process was during the
Spring 2016 semester and the number of students met or exceeded the competency level. By the end of this session,
participants will be able to apply these concepts to their own institution to create competency assessments in their courses.