Injustice - Developers Among Us (SciFiDevCon 2024)
Aims course managementsoftwarereport (1)
1. Usability Report for AIMS Course Management
Software
Research Team: Tristan Carkuff, Samuel Childs, Derek Rumpler, Stuart Yamartino
Client: Armstrong Institute for Interactive Media Studies
1
4. ● Competitive Analysis: After identifying Niihka and Coursera as the primary competitor to
our online course management platform, we developed a series of best practices that
stressed contact with the professor, ease of navigation, and means to gauge progress.
Issues and Recommendations
After the totality of our research, we decided the best course of action would include the following
recommendations:
● Streamline site features and focus on those features that are important to course goals
(in other words, avoid feature creep).
● Design an interface that makes it easy for students to contact their classmates and the
professor if necessary.
● Ease the tension in the homework submission process. Assist students in trusting that
the assignment submission system is reliable and trustworthy.
Prototype Testing and Design
We implemented the recommendations in Issues and Recommendations. Our redesigned
obtained a high A grade on the System Usability Scale opposed to Niihka’s D grade. Thus we
can say that the data supports that we have sufficiently dealt with the usability issues presented
in Niihka and our methodology seems sound for designing course management software.
4
5. Methods and Results
Following is a detailed report of the specific quantitative and qualitative methods that we engaged
in leading up to our redesign.
Competitive Analysis
We decided to look at other course management platforms to inform our design for our course
management site. This is the result of our research:
Actionable Intelligence
Based on our feature audit and input we have received from students at Miami University, we
have developed the following best practices:
● There should be a system whereby students can easily track their progress within the
course. This includes being able to visit previously completed weeks via some interface
such as a calendar and being able to know their grade at any given moment.
● Course platforms should focus on the students’ needs. Unnecessary elements which
complicate student goals should not be part of a course platform. Students should be
able to navigate a course platform with minimal to no tutorials or orientation.
● While many of the online course platforms that we analyzed did not include instructor
feedback, a frequent complaint from some students of online courses was that students
felt they were unable to contact their instructor. Other features, such as online forums
with other students, may benefit the student, but few things were as appreciated as
having easy access to instructors. Therefore, a course platform should highlight this
functionality.
These best practices were determined after identifying our competition, conducting a website
features audit, and doing comparative research.
Identifying the Competition
When conducting our competitive analysis, we identified a number of competitors based on their
use across Miami University and from our knowledge of other popular course platforms. We
confirmed our list with the client and additional suggestions were made. From this research, we
identified two key competitors to the AIMS platform. We chose these two platforms since their
goals seemed closer to those of the AIMS platform and thus provided a significant benchmark
whereby we could judge our redesign.
[A thorough list of competitors can be found in Appendix A: List of AIMS Competitors.]
5
6. Table 1.1: Tier I Competitors to Our Course Management Site
Company/Platform URL Category Description
Coursera https://www.coursera.org/ Tier I Coursera is an online course
platform with both course
management and tools to
facilitate online classes.
Niihka http://niihka.muohio.edu Tier I Site used to display student
grades, post assignments
and resources, as well as
forums and email support to
encourage communication
within the class.
Website Features Audit
In conducting our websites feature audit, we first compiled a number of features that we found
on the different sites that we analyzed. Based on our analysis from these sites and the feedback
that we received from students, we determined that the following features were important for our
redesign:
● An interactive calendar
● Email resources
● Grade features
● Online submission
● Teacher feedback
● Forums
[A thorough table of website features can be found in Appendix B: Website Feature Audit Table.]
Comparative Research
The next step was to determine perceived advantages and foibles in each of the sites that we
analyzed. We decided to optimize visual aspects of the Coursera website in our redesign: a
simple, clean interface which focused on student needs first and foremost. We strived to avoid
the major issue that we perceived in the Niihka platform: needless information. Further, we
wanted to promote social features such as email and forum posts. Our analyses of both Niihka
and Coursera can be found in the following table:
6
8.
User Testing Analysis (Niihka Usability Testing)
Further, we conducted three usability testing sessions using the Niihka platform and eyetracking
software. After a preinterview, eyetracking, and postinterview, we discovered that students
generally were familiar with the platform but did take some time in finding what they were
seeking. This confirmed our presupposition that the Niihka platform might be too complicated for
most students.
We then designed a prototype including the feedback that we received from this analysis. After
we gathered five different students as test subjects, we found that there was a significant
decrease in time between the Niihka platform and our proposed AIMS platform. This is
summarized in the following chart:
Table 1.3: Usability Task Times
Task Original Time in Seconds
(Niihka)
New Time in Seconds (AIMS
Platform)
Time Reduced
Find News Pertinent to the
Course
9.57 4.48 53.19%
Find New Assignment 10.90 1.96 82.02%
Find Due Date for
Assignment
9.27 4.22 54.48%
Find Description on How to
Do Homework
11.83 5.34 54.86%
Find Current Grade for
Semester
12.37 4.76 61.52%
Find Quiz Grade 8.07 3.56 55.89%
It should be noted that since this was the second semester of the school year, our test subjects
already had significant experience with Niihka. Even though there was a drastic improvement in
time for those already experienced with Niihka, we suspect that new students without exposure
to Niihka will adapt to our platform at a faster rate than if they had started with Niihka.
[See Appendix E: Usability Testing for our sample script and more detailed results.]
User Testing Analysis (In-Depth Interviews)
As a team, we conducted a total of nine indepth interviews with Miami University students
possessing various levels of experience with online courses and course platforms. We
discovered that these students generally distrusted online platforms and desired personal
interaction with professors. They also despised the lack of consistent structure across courses
on Niihka. Therefore we concluded that online courses must include features that seek to deal
with these anxieties and that provided close correspondence with professors.
8
9.
Included is our script guideline we used to facilitate discussions with students:
● Have you taken a class where the primary component involved being online? These can
include completely online classes or hybrid classes. What was the platform?
● Would you take/retake an online class if you had the opportunity to do so?
● If you think about online classes, what are your initial reactions?
● What would be the key attributes of a successful online class?
● Do you have any fear or apprehension about taking online classes? What are they?
● Can you share an experience in which an online learning platform such as Niihka has
been detrimental to a class?
● Can you talk about the experience?
● Can you share an experience in which an online learning platform such as Niihka has
been beneficial to a class?
● Can you talk about the experience?
[See Appendix F for the Summary of InDepth Interview Responses.]
System Usability Questionnaire
We asked 5 different students for their feedback regarding Niihka using John Brooke’s System
Usability Scale. Based on our feedback, we discovered that Niihka received a grade of 56%
which is considered a D on this scale. Anything above 68% is considered above average using
this scale.
However, after testing our redesign, we discovered that it was a significant improvement over
Niihka. Our final score using the System Usability Scale resulted in an average of 96.5%, a high
A on the scale.
[See Appendix G for System Usability Scale Survey Data.
Site Persona Development
In order to best understand how to redesign the site, we created a site persona that we felt was
symbolic of the variety of user data that we obtained. Our persona is a female college student
named Salley O’Malley taking online classes for the first time. We decided it would be most
informative to use a persona who was capable of using computers for simple tasks, but was not
sufficiently techliterate. This kind of persona was useful for us because it exposed any site
characteristics that were difficult for students unskilled in computers.
[See Appendix H for Persona Supplements.]
9
18. Appendix A: List of AIMS Redesign Competitors
The following is a complete list of the competitors that we analyzed. Within the body of our
report, we chose to only focus on competitors that we deemed to be Tier I. This designation was
chosen since their goals seemed to overlap what we were doing with our redesign.
Table A.1: Complete List of Competitors to Our Course Management Site
Company/Platform URL Category Description
Chalk and Wire https://ep.chalkandwire.com/
ep2_muohio/login.aspx?cus=
228
Tier II Chalk and Wire is the online
course management
software for Miami
University’s Honors Program.
Coursera https://www.coursera.org/ Tier I Coursera is an online course
platform with both course
management and tools to
facilitate online classes.
Desire2Learn http://www.desire2learn.co
m
Tier II System that allows teachers
to focus on teaching, leaving
the design and management
of the online course to
experts at Desire2Learn.
The platform allows for
online assignments,
resources, and grading as
well as other analytic
services.
Khan Academy https://www.khanacademy.
org/
Tier II Khan Academy is an online
learning platform that is
based around user
achievements. It also has
features that can be used by
instructors to track user
progress.
Niihka http://niihka.muohio.edu Tier I Site used to display student
grades, post assignments
and resources, as well as
forums and email support to
encourage communication
within the class.
18
19. Appendix B: Website Feature Audit Table
In conducting our websites feature audit, we first compiled a number of features that we found
on the different sites that we analyzed. We then took note when a site had a given feature (x) or
when it did not (). The following table is a list of these features:
Table B.1: Complete Website Feature Audit
Website Features and
Functionality
Chalk and Wire Coursera Khan Academy Niihka
Calendar/
schedule
x x
Email resources x
Help x x x x
Tutorials x x x x
Grades*
*measured feedback,
not necessarily
lettered
x x x x
Online submission x x x x
Embedded videos x x x
Teacher feedback x x x x
Forums x x x
Blog x
Course roster x*
*if course roster is
understood to include
students in forums
x x
Online tests/quizzes x x x
Multiplatform support x x
Portfolio x x x
Student selects
courses
x x
Tracking progress x x
19
20.
Appendix C: Comparative Research Table
The following concerns the competitors outlined in Table A.1. Within this table, we outline both
the positive and negative design decisions of each of our competitors.
Table C.1: Complete Comparative Research Table
Platform Advantages/Core Competencies Obstacles/Foibles
Chalk and Wire Chalk and Wire is arranged by portfolio.
Within the Miami University Honors
portfolio, Honors requirements are
marked as categories in the left toolbar.
This clear design allows students to
readily access what is needed for the
program and to submit their work within
sections relevant to program goals.
Chalk and Wire has a steep learning
curve that requires a great deal of
outside help and tutorials to master. It is
difficult to find material on the site since
the mappings do not seem logical, menu
items are unclear, and there is too much
information on the pages. Further, the
information on how to operate the site
is spread across several Miami
University websites.
Coursera Coursera is very similar to Niihka in its
design. However, the design has more
white space, the font size is a size 11
Arial font, and the left tool bar only
contains information essential to the
course. General and weekly
announcements are on the main course
page and students can access other
important things (such as quizzes or
homework) through the left toolbar.
A casual observation of the site reveals
that Coursera course learning is
primarily discussion forum based. While
the site is large enough for individuals
to learn from students covering a wide
variety of experience, it is up to the
instructor to determine if they want to
provide individualized responses. In one
course (Gamification), the instructor
decided to address overwhelming
student concerns through the forums
and videos.
Khan Academy Khan Academy is a free online learning
platform that allows people to tackle
any subject that interests them.
Progress is tracked through a progress
map that shows their “trek” across
various interrelated domains of
knowledge. Students can also earn
merit badges which they can display
publicly. The site also contains metrics
of the individual’s progress, which
either they or “coaches” can use to
improve learning.
Like Coursera, discussions are
relegated to an Internet forum (unless a
coach such as an instructor
provides outside feedback). The site is
designed to be selfpaced and
selfmotivated with students able to
engage themselves with anything that
interests them. This design might not
benefit all students.
Niihka Niihka possesses a number of
customization options for navigation
purposes. They can add tabs at the top
of the webpage through preferences.
This allows them to access each
course. They can also access their
current courses through the “More
Through extensive interviewing, we
have discovered that several people
are not content with the Niihka platform.
The primary reason being that the
platform requires a substantial time
commitment to master. While students
have demonstrated proficiency with the
20
22. Appendix D: Data for Heuristic Analyses of Niihka
For more information on the Forrester Scorecard, you can peruse their website for more
information: http://www.forrester.com/home
Following is a table of individual group scores corresponding to the Forrester Scorecard with the
group scorecard scores provided.
Table D.1 Table of Individual and Group Scores
Individuals Score
Individual 1 0
Individual 2 15
Individual 3 1
Individual 4 N/A
Group Consensus 6
Due to a number of technical issues, we are unable to include the score cards in the appendix.
Nonetheless, they can be accessed here: http://goo.gl/r10Gr
22
23. Appendix E: Usability Testing
Script
The following is the script that we used for both usability tests (note: we omitted everything
before Familiarization and the Post Test Section for the test of our redesign since the Niihka test
was diagnostic):
Introduction
We are going to do a number of usability tests to ensure that the site operates the way that you
think it should. Because of this, we are testing the site, not you. If you are confused about
something or can’t find anything, tell us and we will communicate that to the site’s creators.
Since we did not create the site, you should not worry about offending us.
Niihka/AIMS Platform Usability Process
Do you have any experience with courses where the online component was predominant?
These can include courses that were fully online or hybrid courses. Could you describe your
experience with these courses?
What is important to you when trying to locate course work or class materials online?
Do online course platforms work how you would expect them to work? If not, why/how?
Familiarization
Have the user look at the material for a minute.
Can you tell me briefly what you can do on the site?
Could you discuss how the site is organized?
Tasks (eyetracking)
1. Your instructor posted a new assignment. Can you show me where you would access this
information?
2. Your instructor graded the quiz that you took yesterday and posted this grade online. Show me
how you would find this grade.
3.You wanted to find your grade on a homework assignment that you took at the beginning of the
semester. Show me where you would go.
4. Show me where you would go if you want a description on how to do the homework.
5. Your instructor recently posted news pertinent to the course. Show me where you would go.
23
24. 6. Show me where you would go if you wanted to know the due date of an assignment.
PostTest
1. What was the most difficult task we asked you to do?
2. What did you like visually about the site?
3. What didn’t you like visually about the site?
4. What is something that you have had difficulty doing with Niihka over the semester?
5. What is something that you wish you could do with this interface?
6. Do you prefer a consistent navigational structure or does it matter for your courses?
Detailed Usability Results
The following table is the detailed results of the usability data between Niihka and our redesign
(termed AIMS Platform in this table):
Table E.1: Complete Usability Results
Niihka’s Initial Survey
Miami University Students’ initial reactions to handling the Niihka interface were documented in
the form of a survey when Niihka was first launched. The majority of the student population
agreed that it was helpful to be able to access class content through the web and improved
student teacher communication. Few students experienced problems, most of which were
minor.
On the written portion of the survey several students expressed interest in some sort of help
system for the Niihka interface. When asked to describe the nature of the problems they
experienced, a great deal of students reported problems with either the layout or network/server
errors. This data is located at http://goo.gl/EhVYu
24
25. Appendix F: Summary of In-Depth Interview Responses
Table F.1: Summary of Coded InDepth Interview Responses
25
26. Appendix G: System Usability Scale Survey Data
We distributed the survey for the System Usability Scale to different two groups of five people
each. The first group used Niihka and the second used our redesign. As the survey data
demonstrates, we have made a significant improvement over Niihka. Our data can be
reconfirmed by using this guideline to calculate the results into a letter grade:
http://www.measuringusability.com/sus.php
Survey data continues on the next page.
26
38. Appendix H: Persona Supplements
Interview Notes
We conducted interviews with seven different students. Included are the important highlights of
these interviews. These highlights informed our redesign.
Student 1
○ never taken online class
○ would like to take online class
○ desires contact with professor beyond simply email
○ wants lots of resources offered in the course
○ afraid professors might not be accessible
Student 2
○ never taken online class
○ willing to take online course but dependant on topic, because he wants to be able
to have contact with professor for explaining
○ doesn’t feel that online classes are really focused on by professors
○ thinks that online classes are only easy course, more flexible than standard
courses, require less effort
○ a good online class should have more structure, which isn’t what he would like
○ good course would have lots of interaction between professor/students
○ afraid he would forget to do assignments, since you don’t go to classes
○ feels that some professors are very unorganized in an online course/platform
Student 3
○ never taken online class
○ wouldn’t want to take online class, because he wants personal interaction with
professor
○ feels like online classes have lack of structure
○ feels that professors have less to do with online classes than with regular classes
○ would like online classes to be structured
○ feels like you will learn more from interaction with a real professor (not online)
○ more concrete courses (Mathematics, Statistics) would work better than flexible
(philosophy, social sciences)
○ afraid of computer/platform errors, such as niihka not submitting homework
Student 4
○ has taken online course (on Niihka)
○ enjoyed online class
○ enjoyed interaction with other students (forums)
○ thinks online classes will be easier and no group projects
38
39. ○ exams easier, because they are all open book, long time to complete
○ blogs very helpful, videos helped
○ wasn’t sure if she could talk to professor for help
○ used virtual office hours through Skype
○ interaction with other students is really helpful
Student 5
○ has taken an online class
○ wouldn’t retake online class, it was very difficult to access
○ not in touch with the professor, has trouble keeping up with assignments
○ access should be simplified
○ online classes require lot of independent work
○ dislikes only being able to communicate with professor through email
○ would like access to old material
○ forgets to turn things in, no reminders
○ site is not standardized enough to allow people to use it properly from class to
class
Student 6
○ has taken online class before
○ would retake an online class
○ liked online class (useful during the summer)
○ feels that online classes are very independent
○ feels that online classes need good communication
○ thinks that an online classes requires a good platform
○ afraid of miscommunication in the course
○ afraid of missing assignments
○ afraid that it will be difficult to contact professors for help
○ afraid of platform not working properly (not submitting assignment)
○ likes that online resources can be accessed through any computer
Student 7
○ never taken online class
○ would like to take online class
○ likes that online classes allow more freedom
○ feels that online classes are easy
○ likes that online classes can be taken anytime of the day
○ feels that a good online class should have at least one assignment a week
○ fears video chatting teacher because it would be awkward
○ fears platform will not work properly
○ likes how online class allow for submission times at midnight
Student 8
39