What Students Know and What They Don't Know - Course Technology Computing Conference Presenter: Cheryl Reindl-Johnson, Jennifer Day & Jennifer Romero, Sinclair Community College We have been hearing for years that our jobs will be obsolete as students enter college knowing more about computers than their college faculty members know. At Sinclair Community College we have NOT found that to be the case. In fact, we have struggled with success and retention rates in our Computer Concepts and Applications course. We offer roughly 80 sections of our BIS 1120 Computer Concepts and Applications course each semester, enrolling approximately 4,000 students per year. We realize that there are myriad reasons why students are not successful in a course, but to begin to find a solution to address low success and retention rates in this high enrollment course, we felt we needed to do some analysis of our student population. We are using SAM to administer a pre-test to all students enrolled in a section of the course during the second week of the semester, so that we can better analyze student knowledge of the material when they arrive in our classes, and collect information on student mastery of the material at the end of the semester using the same content in a post-test. SAM's reporting features allow us to analyze section level results to compare: Day classes that tend to include more “traditional” students who are straight out of high school, to evening classes which tend to include more non-traditional students; Face-to-face to online sections; Full semester classes to 12-week or 8-week sections; Sections taught by full-time faculty versus adjunct faculty. When the pre-test and post-test are scheduled by our SAM faculty administrator, we can also analyze the data at the question level (frequency analysis) across all sections to examine: What content most students (70% or more) come to the class knowing (specific skills that are used across applications and groups of skills by application); What content most students do not know when they begin the class; What content most students struggle with at the end of the class.

1. What Students Know and Don’t Know
Jennifer Day, Cheryl Reindl-Johnson, and Jennifer Romero
Sinclair Community College

2. The Issues
• Belief that students come to college knowing MS
Office from high school classes and/or work
experience.
• BIS 1120 Computer Concepts and Applications - high
enrollment course (4275 per year), but low success
rates (53.7%).

3. BIS 1120
• Three semester-hour, 16-week course covers
computer concepts and introductory
Word, Excel, Access, PPT.
• Custom New Perspectives text and SAM.
• A variety of approaches to explore low success rate
over the years.
• Pre-exam/Post-exam approach

4. Student Enrollment
4%
58%
3%
4%
2%
29%
Location
CVCC
Dayton
ELC
HHLC
PCLC
WWW

5. Exam Breakdown
• 70 Items in 75 Minutes
– 6 tasks – Windows 7
– 13 tasks – Word
– 14 tasks – Excel
– 14 tasks – Access
– 14 tasks – PowerPoint
– 5 MC – Browser & Email Basics
– 4 MC – Computer Concepts

6. A computer interprets every signal as either “on”
or “off” using numbers known as _____________
1 2 3 4
25% 25%25%25%
1. millibytes
2. binary digits
3. ASCIIs
4. hexadecimals

7. The rules that establish an orderly transfer of data
between sender and receive are called
1 2 3 4
25% 25%25%25%
1. protocols
2. channels
3. NICs
4. priorities

8. ________ changes every time you add or remove
hardware; therefore, it is often referred to as
semipermanent memory.
1 2 3 4
25% 25%25%25%
1. BIOS
2. CMOS
3. ROM
4. The CPU cache

9. If a Web page provides an RSS feed, the orange
Feeds button appears on the _______________.
1 2 3 4
25% 25%25%25%
1. Command bar
2. Status bar
3. Address bar
4. Favorites Feeds tab

10. Online (29%) vs. Face-to-Face (71%)
Population # Students Average Min Max
Online (Full Term) – 225 80% 11% 99%
Online (12-Week) 13 84% 67% 94%
Online (8-Week Combined) 74 79% 19% 99%
Online (A-Term) 21 84% 49% 99%
Online (B-Term) 53 78% 19% 99%
F2F (On Campus) 532 79% 6% 99%
F2F (Off Campus) 137 76% 20% 97%
F2F (8-Week Term) 7 85% 70% 96%

11. The RESULTS

12. Exam Questions
• Common questions missed among high scorers
– Multiple choice questions
• CMOS question
• A URL’s ___ is a standardized procedure the computers use to exchange data
(protocol)
• RSS Feed question
– Task Modify the style of the selected table to create a new table style
option named “MODEL” with the Header Row using Italic Font and
apply it to the table
• click Format as Table drop down arrow, right click selected style (modify is
dim) – select “Duplicate…” – that opens dialog box which then you select
Header Row and click Format, select Italics, save, click OK, then select Table
drop down arrow and apply formatting

13. 13Fall Pretest Frequency Analysis
• Multiple choice questions missed on the pre-exam
were also missed on the post-exam.
• Task EX2870 no students answered correctly on pre-
exam. Frequently missed (87%) on post-exam.
• Exam average – 34%

14. 14Spring Pretest Analysis
• Multiple choice questions missed on pretest are the
same three missed on fall pretest.
• Task EX2870 (Modify table) most missed question
on exam (3 students).
• Exam average 33%

15. Scores – Fall 2013 – Post-Exam
– 835 registered for sections did not take exam
– 108 scores removed because of irregularities
• 990 Scores used from Post exam
• 55 sections
– 21 taught by FT Faculty (5 faculty members)
– 34 taught by Adjunct faculty (23 faculty members)

16. Average by Application
Subject # Tasks
Pre-test
SP14
Pre-Test
F13
Post-Test
F13
Improvement
MS Access 14 24% 24% 86% 62%
MS Excel 14 19% 19% 68% 49%
MS PowerPoint 14 39% 40% 82% 42%
MS Windows 7 6 60% 59% 88% 29%
MS Word 13 33% 34% 83% 49%
Overall Average 62 32% 32% 80% 48%

17. High Scorers
Students Pretest
Average
Post Test
Average
Women 10 37.2% 98%
Men 15 46.4% 98%
Total Students 25 41.8% 98%

18. Day vs. Evening Sections
Students Average Min Max
Day 511 79% 6% 99%
Evening 117 76% 14% 99%

19. Time
Shortest time = highest scores Highest time = lowest scores
• 41 student took 75 mins (max)
– 10 (24%) scored 50% or
– 28 (68%) did not pass
Score Time
94% 0:12:56
97% 0:15:38
96% 0:15:58
97% 0:16:16
94% 0:16:34

20. Full-time v. Adjunct
• 59% students taught by adjuncts, 41% by Full-time
• Overall Average = 79%
– Adjunct Average = 78%
– Full-time Average = 80%
• 9 of 56 sections average below 75% - 7 were taught
by adjuncts – 2 of those brand new faculty –
• Highest class average =86%, lowest 65%

21. Highest Average by Faculty
• Highest average 86%
– Day (FT - online) - 21 students with only 1 student not
passing post-exam (scored 64%)- 10 of 21 scored 90% or
higher.
– Paranjpe (Adjunct - 8 week A term) - 2 of 15 did not pass
(57% and 70%) – 9 of 15 scored 90% or higher
– Aungst – (FT – F2F) 2 of 10 students did not pass (57%
and 74%) – 6 of 10 scored 90% or higher

22. Post-Exam Statistics
• 70.4% overall passed (75% or higher)
• 73% male passed
• 68% female passed

23. Post-Exam by Gender
Male – 423 (42.7%)
A - 57 (13.5%)
B – 142 (33.6%)
C - 109 (25.8%)
D – 66 (15.6%)
F – 49 (11.6%)
13%
33%
26%
16%
12%
A B C D F

24. Post-Exam by Gender
Female – 567 (57.3%)
A - 65 (11.5%)
B - 141 (24.9%)
C - 182 (32.1%)
D – 102 (18%)
F – 77 (13.6%)
11%
25%
32%
18%
14%
A B C D F

25. Lessons Learned
• More faculty involvement in test creation
• Better communication about significance of exam
with adjunct faculty
• Naming/scheduling of exam by Superuser
• SAM paths allow students to focus on what students
don’t know

26. Next Steps
• Involved faculty reporting other data (not SAM data)
• Improving the pre/post exam to focus on course
objectives
• Department changing course content to focus only
on apps