User testing and focus group report at Manchester University (C-SAP collections project)

Report on C‐SAP (Higher Education Academy Subject Centre for
Sociology, Anthropology and Politics) focus group and user testing at
the University of Manchester, July 2011
_________________________________________________________________________________

Isabelle Brent
C‐SAP, August 2011

This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England &
Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/

1

Manchester Focus Group and User Testing Report

Introduction and Summary
As part of the OER Phase 2 C-SAP Collections Project a user-testing of the front-end
website took place at the University of Manchester on 27th July 2011 together with a mini-
focus group.

The user testing benefited from the wide range of backgrounds of participants and generated
a number of specific suggestions that have been forwarded to the developer. The main
overall finding was the lack of understanding about the purpose of the website in promoting
resource reviews. When this aspect was explained there was widespread support for the
idea but this is not currently communicated in the website. The Google custom search was
particularly popular together with the links page. It was felt that the site looks too much like a
blog and needs more design to highlight the contents of the website.

The focus group provided more support for the increasing use of Google and YouTube in
academia. There was a general enthusiasm for using digital resources, particularly videos,
both in lectures and on VLEs; reservations related to technical rather than intellectual issues.
There was also a widespread acceptance of sharing teaching online and the use of Creative
Commons licenses.

Participants
Emily Bannister (Researcher Development team. User testing only)
Andrew Gold (Learning Technologist. User testing only)
Elisa Pieri (PhD researcher. User testing only)
Keir Martin (Lecturer Social Anthropology)
Mark Brown (Teaching Fellow Centre for Statistics and Survey research)
Kamie Kitmitto (Landmap manager/MIMAS)
Ian Fairweather (C-SAP)
Graham Gibbs (C-SAP, Focus Group Chair)
Anna Gruszczynska (C-SAP)
Isabelle Brent (C-SAP)


2

Focus Group

Searching Online
Though individual social science sites were mentioned, Google emerged as the clear
favourite for online searches of academic materials. The attachment to Google is clearly not
because of lack of awareness of other sources and participants expressed the view that
Google’s search capabilities have improved.

Increasingly I’m using Google as the widest net to cast a search

Increasingly Google is actually easier

The use of the word ‘increasingly’ suggests that the academics do not set out with the
expectation of using Google but that it is proving more effective than other strategies. The
general faith in Google is expressed well by one participant:

If I want to find a theatre ticket, you know, whatever I want to find I put it into
Google and I’ve found in terms of doing academic searches that pretty much
every time I’ve looked, if it’s there I’ll find it on the first two pages, whether it’s
a book about Robert Owen and socialist humanitarianism in the 1930s . . . I’ll
find it on Google.

The only resources that Google was not the primary search tool for was searching for
videos. YouTube was the preferred site and was considered to be the video equivalent of
Google:

Google is for searching and YouTube is for videos

Participants were aware of other sites like Vimeo and particular sites hosted by institutions
such as methods@manchester, however it was felt that YouTube was the first place to look.

Using digital resources in teaching
All the focus group participants were comfortable with incorporating digital resources into
their teaching though it is clearly quite a new practice. The change to Blackboard 9 was cited
as having made a big difference in teacher’s ability to integrate digital sources. As well as
YouTube, ‘Box of Broadcast’ was mentioned as a useful resource because results can be
embedded in a VLE:

. . . it pulled up a Panorama programme and a couple of radio programmes
[on demography] that just gives you a link to embed within a virtual learning
environment, something like Blackboard and it’s great

Other digital resources such as newspapers and websites were also mentioned.


3

‘Live’ vs VLE use of Digital Resources
Two contexts for the use of digital resources were discussed; during lectures and as
additional materials provided within Blackboard. Participants were aware that there was a
definite limit to the amount of time spent watching a video during a lecture. There was
general agreement when one participant said:

If you do too much video in the actual lecture students think it is a cop‐out

During lectures videos are used to illustrate points and to point to further resources. For
example one participant mentioned the use of a YouTube clip to illustrate a particular
anthropological ritual which was particularly popular with students. As he commented, it
shows that it is:

real people and you can see that there are emotions attached to it

Another strategy is to show a clip and refer the students to Blackboard to watch the full
version. Participants described collecting relevant resources to put on course pages
including videos, websites and newspapers. In addition to a concern with not wanting to play
clips for too long during a lecture, the most common reason for focusing on the VLE was a
concern that technology would not work:

I would be wary about having a lecture that relied on it, just because I don’t trust
any of the infra‐structure . . . Whereas I trust my brain and my mouth to get through
sixty minutes.

This reservation was expressed at a number or points and suggests that more extensive use
of digital resources is being held back by a basic concern for technical support.

Quality
Assessing the quality of digital resources, particularly videos was discussed and identified as
a barrier to further use. This was particularly the case with YouTube:

The problem is quality control and you end up watching a lot of crap made by
people who don’t really know what they are talking about . . .

Since the focus group took place after the user-testing session there was a consensus that
the idea behind the review of resources of the C-SAP collections project was a valuable one:

There is a lot of very bad stuff and the real appeal of what you were showing us
was that . . . somebody’s done the work a bit before

Intute was referred to as being a valuable resource because the materials had been vetted.
The labour-intensiveness of this process was acknowledged and different star rating review
systems were discussed such as Amazon and music review sites as a potential alternative.


4

The incentives for contributing to such a site generated an interesting discussion about how
sites become popular enough to generate contributions:

It depends on who I imagine the community of people to be who are clicking
‘like’ or ‘dislike’ so that is a way of building a brand. And then I would be more
likely to do it myself if I thought it was part of that community as well, so I imagine
it is how you get to that critical mass where people imagine there are enough
like‐minded people involved.

Participants were interested in systems which marked out resources as being particularly
interesting. The number of views on YouTube was one indicator that acted as an initial filter
and the ‘likes/dislikes’ bar.

The focus group described trust and quality in a particular way which relates to a filtering
process. Participants reserved the judgment of quality for themselves and were unsure that
any indicator would of itself make them ‘trust’ a resource. However the issue is more how to
have a filtered short-list. This is particularly the case with teaching resources which, in
addition to general concerns with quality, have to fit into a particular curriculum and teaching
approach:

The key thing with a video is that students can watch it in their own time about how
to do a method but you have to be very careful at the way that is taught and the way
the video matches your teaching.

The issue of ‘fit’ was identified as a problem with some digital teaching resources which
cannot be ‘chopped down to fit into a pre-existing course’.

Attitudes to Openness
All participants were comfortable with the notion of sharing resources and the concept of
Creative Commons licenses. Michael Wesch’s (Wesch 2008) work on YouTube was cited as
a positive example within anthropology whose work has been viewed by millions online.

The increasing use of videos and other digital resources available online has also affected
the attitude of our focus group to making materials open themselves. All agreed that they
would be happy to see their materials freely available online given they were of sufficient
quality. The potential barriers they identified related to institutional restrictions rather than
personal concerns over intellectual property. This suggests that the use of freely available
digital resources (whether covered by CC licenses or not) provides a model of sharing that
encourages users of such materials to contribute their own content.

The reservations expressed were mainly practical, such as wanting to veto poor quality
materials. One participant felt that it would be difficult if the university decided to suddenly
make everything CC-licensed because of concerns over copyright issues for images that
might not have been attributed. The main obstacle to opening up more materials was
perceived to be institutional with some universities not allowing materials to be open.


5

Notes From User Testing

Front Page
 Who is the site for? Know it is social research methods but who is the site for? Is it for
teachers or students?
 Needs to have clear focus for teachers.
 Should we be making it more attractive to students?
 Say what the site does for users. Its benefits, resources, teaching materials.
 Website not upfront about what it is about
 Could identify type of user and route them to separate pages e.g. portals.
 Looks like a trustworthy academic site
 Front page undersells itself, too academic – need to put clear statement of benefits on
front page
 Make title say it is for teaching
 Image looks like it may be about business/consultancy
 ‘our recent review’ not necessary on front page.
 ‘Most recent’ assumes you are a regular visitor. May not have regular visitors, - seems
disproportionate.
 Don’t want to have to scroll up and down
 Needs RSS on home page
 Same list in sidebar and main page not necessary
 Sidebar contains things that say ‘active community’
 Video of the week – only makes sense for regular visitors
 Confusion over whether it is a blog or a website – ‘posts’ may need replacing
 Should include more web2.0 features in addition to Google +1

Videos
 Main box on home page should show the core contents of the site – our pitch or USP
 Video item, should open video directly. Can we embed them all?
 At the end of long lists only a few items show and then there is ‘older posts’ at the
bottom. This should say More… or some such
 Video titles sometimes very broad. Needs abstract of video with the resource. (Can we
pick up YouTube description??)
 Arrows confusing – should be to play video.
 Looks like a PowerPoint with sideways arrows – also makes ‘video of the week’ unclear
 ‘Video of week’ wrong name
 Wanted to be able to browse through video resources


6

Reviews
 In general participants didn’t ‘get’ that the reviews were a main part of the site
 One user thought they were critical appraisal of method itself rather than review of
teaching resource
 ‘Reviews’ is an odd title. (Use ‘Resource Reviews’?)
 When they were explained they were popular - not too long and useful. Pedagogic
content especially useful
 ‘Write a reply’ is odd wording under a resource/video. Reword to ‘comment on resource’
 Checklist for reviews needs to be more visible (add an icon link to this on login)
 Reviews a little too specific? Need glossary or more general ones - perceived to be
aimed at teachers of methods already knowing about the subject
 Having search box above list of reviews in panel to the right makes it appear as if it is a
search of reviews

Resources
 Categories for taxonomy should be more logically organised. Retain logical order (not
alphabetic)
 Resources. Say they will be relevant to you
 Wants to find resources first and then link to its reviews
 Don’t like 2-step process of getting to resources – intermediate stage looks like blog
post

Google Custom Search
 Custom search – need to say clearly what it does
 Is Intute in the custom search tool?
 Thought custom search tool was for the site – needs new name.
 Search – Make clear the scope of the custom search tool – e.g. that it searches outside
this site. (Call it ‘UK resources search’?)
 Custom search – make a big selling point for the site. Say it searches high quality sites.
 Impressed by Custom Search Tool (is Jorum in the custom search)
 Compared with Google, all stuff looks relevant
 Problem of pop-up page from Custom Search – lose it when you go to another site.
 Show sites we use in a pop-up list?
 Is there a case for including American sites?

Appearance
 Likes the graphic. Arresting, combines things you might use


7

 Use different background colour.
 Background colour described as ‘hospital green’

Sign‐Up
 Needs better sign up for contributors.
 Tell people that e-mail address will not appear – to preserve confidentiality or don’t use
e-mail.

Links
 Add ESDS to links page – ESDS International especially useful
 Popular to have links in one place

Events
 Categorise results and avoid past events

Misc
 Andy Field has a very good website (linked to his books – add to our custom search)
 Other ESRC funded projects will now run for 2-3 years and then will need to be added to
this site.
 There is a JISC funded project in Philosophy doing event listing. – involves the Institute
of Philosophy and will finish in November (could use their way of getting events??)
 Personalisation through registration suggested

Reference
Wesch, M. (2008). An anthropological introduction to YouTube, YouTube.


8

User testing and focus group report at Manchester University (C-SAP collections project)

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à User testing and focus group report at Manchester University (C-SAP collections project)

Similaire à User testing and focus group report at Manchester University (C-SAP collections project) (20)

Plus de CSAPSubjectCentre

Plus de CSAPSubjectCentre (20)

Dernier

Dernier (20)

User testing and focus group report at Manchester University (C-SAP collections project)