How to analyse questionnaire data: an advanced session

1. How to analyse questionnaire
data: an advanced session
Prof Bart Rienties
13 July 2020
A special thanks to Dr Rachel Slater, Dr Christine Thomas & Dr Jenna
Mittelmeier (University of Manchester)

2. Workshop objectives
• By the end of this session you will familiar with:
–how to analyse the questionnaire data using common
psychometric and linguistic techniques.
–Concepts introduced in this session are computing of
constructs, factor analysis, missing values, reliability,
and validity,
–Reporting on these questionnaire results and
advanced techniques (e.g., ANOVA, correlations,
regressions, SEM)
–Triangulation of quantitative data with qualitative data.

3. What is a questionnaire
• A research tool for data collection
• Usually a set of structured questions for which answers
can be coded and analysed quantitatively
• Can also include open questions
• Can be self-administered or through interview
• On-line, postal, telephone, face-to-face
• Can also be used for qualitative analysis using semi-
structured questions (face-to-face or by telephone)

4. Questionnaire design in the
survey process
• Research aim and research questions
• Identify the population and sample
• Decide how to collect replies
• Design your questionnaire
• Run a pilot survey
• Carry out main survey
• Analyse the data
• Report findings and dissemination

6. Questionnaire design in the
context of the survey process
Research aim &
questions
Research
population &
sample
Survey type &
method for
collecting replies
Think about
analysis
Questionnaire
design
Pilot (always !) Run survey
Analyse data
Report findings

7. Why use a
questionnaire?
Strengths and
limitations?

8. Strengths
• Can be quick and relatively simple way to collect data
• Insightful when large number of participants involved
• Reach respondents in widely dispersed locations
• Can be relatively low cost
• Standardised and structured questions
• Analysis can be straight-forward
and responses
pre-coded

9. Strengths
• Can cover activities and behaviour, knowledge,
attitudes, preferences
• Use to describe, compare or explain
• Effective for collecting quantitative data – information
that can be counted or measured
• Low pressure for respondents
• Lack of interviewer bias
(possibility of ‘ghost interviewer’ effect)

10. Limitations
• Biases
–non response, self selection, questionnaire fatigue,
acquiescence, extreme response styles
• Quality of data? Confidence in results? Reliability?
• Unsuitable for some people
–e.g. poor literacy, visually impaired, young children,
not online
• Question wording can have major effect on answers
• Misunderstandings cannot be corrected
• Can be difficult to account for cultural and language
differences

11. Limitations
• No opportunities to probe and develop answers –
breadth vs depth
• No control over the context and order questions are
answered in postal surveys
• No check on incomplete responses
• Design issues with moving through online surveys
• Seeks information only by asking, can we trust what
people say? e.g. issues with over-reporting

12. Questionnaire/Survey
design: follow the pros
There is already a lot known about
your research question
– Better to “re-use” and replicate (in
order to compare) than to create
yourself. Find the instrument at:
• http://inn.theorizeit.org/
• www.scholar.google.com
• www.eric.ed.org
– A good validated questionnaire
enhances generalisation
• Possibility to compare and contrast your findings
to others.
• Identify different/similar trends, thereby
increasing relevance of your research
• Easier to publish (if you want to) in good, ISI-
ranked journals
• Possibilities to get more citations, as others can
use and refer to your research

13. Search for your instrument:
http://inn.theorizeit.org/

14. Case study 1: Self-determination Theory

15. Example of Academic Motivation Scale
Academic Motivation Scale
developed by Vallerand et al
1992
• 28 items
• Likert Response scale 1-7
• Questions randomised
• Questions/constructs
developed based upon
theoretical concepts Self-
Determination Theory
• 3 specific constructs for
intrinsic motivation
Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senécal, C., & Vallières, E. F. (1992). The academic motivation scale: A measure of intrinsic, extrinsic, and
amotivation in education. Educational and Psychological Measurement, 52, 1003–1017.

16. Case Study 2:What predicts (international) student progression?
Input Process Output
Learner characteristics
(incl. prior education, gender,
cultural background)
Academic adjustment
(incl. personal-emotional adjustment,
attachment to institute)
Social adjustment
(incl. study support, satisfaction with social
Environment, financial support)
Family characteristics
(incl. support, finance, child-
care)
Learning design
(incl. assessment, learning
materials, communication)
Engagement with learning
(incl. VLE engagement, attending sessions,
submitting assignments, social media)
Academic performance
over time
(incl. grades, credits, GPA)
Degree outcomes
(incl. Employment, migration)
Interviews with 140+ UNISA students, 40+ stake holders, country reports
Madge, C., Breines, M., Beatrice Dalu, M.T., Gunter, A., Mittelmeier, J., Prinsloo, P., & Raghuram, P. (2019). WhatsApp use among African international
distance education (IDE) students: transferring, translating and transforming educational experiences. Learning, Media andTechnology, 44(3), 267-282.
Raghuram, P., Breines, M. R., & Gunter,A. (2020). Beyond #FeesMustFall: International students, fees and everyday agency in the era of decolonisation.
Geoforum.
Roos Breines, M., Raghuram, P., & Gunter,A. (2019). Infrastructures of immobility: enabling international distance education students in Africa to not
move. Mobilities, 1-16. doi: 10.1080/17450101.2019.1618565

17. Hypotheses
■ H1 Internationalisation at Home (IaH) students have higher academic adjustment scores relative to
Internationalisation Abroad (IA), and Internationalisation at Distance (IaD) students.
■ H2 IaH students have higher social adjustment scores relative to IA, and IaD students.
■ H3 IaH students have higher personal-emotional adjustment scores relative to IA, and IaD
students.
■ H4 IaH students have higher attachment scores relative to IA, and IaD students.
■ H5 Access to technology at home is positively related to academic adjustment
■ H6 Access to technology at home is positively related to social adjustment
■ H7 Access to technology at home is positively related to personal-emotional adjustment
■ H8 Access to technology at home is positively related to attachment at UNISA
■ H9 Being from South Africa and having access to technology has a positive impact on academic
adjustment
■ H10 Academic adjustment is positively predicted by social adjustment, personal emotional
adjustment, attachment, access to technology, and being from South Africa.
Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment
in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12

18. SACQ Questionnaire
■ Student Adaptation to College Questionnaire
• measures how well students manage the educational demands of the
university experience.
Academic Adjustment
• measures how well students deal with interpersonal experiences at the
university (e.g., making friends, joining groups)
Social Adjustment
• measures how well students maintain emotional equilibrium (particularly in
the face of adjustment stressors), and indicates whether the student
experiences general psychological distress or shows somatic symptoms of
distress
Personal Emotional Adjustment
• assesses the degree of identification with and commitment towards the
university
Attachment
Baker, R.W., and Siryk, B. (1999). SACQ Student Adaptation to College Questionnaire. Los Angeles: Western Psychological Services.
Rienties, B., Beausaert, S., Grohnert,T., Niemantsverdriet, S., and Kommers, P. (2012). Understanding academic performance of international students:

19. Data collection
■ First, in our initial study (Mittelmeier et al. 2019) we sampled 2634 students from a first-
year level course unit with undergraduate students studying for a Bachelor of Science
degree in Mathematics and Programming in the College of Science, Engineering and
Technology: 320 (11.77%) students (IaH = 270, IaD = 36) responded.
■ In the second phase, we broadened our sampling approach to additional STEM
qualifications, whereby we specifically sampled IaD and IA students using MIS data. 5273
students in the selected programmes were invited to participate through an email sent to
their university email address, which included a link to the online survey.Altogether, in
the two phases 1295 students participated in this study, which is a large sample of
participants with a very reasonable response rate of 16.38% (Nulty, 2008)
Mittelmeier, J., Rogaten, J., Long, D., Sachikonye, M., Gunter, A., Prinsloo, P., et al. (2019). Understanding the adjustment of first-year distance education students in South
Africa: Factors that impact students’ experiences. The International Review of Research in Open and Distributed Learning 20(3). doi: 10.19173/irrodl.v20i4.4101.

20. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment
in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12
■ Substantial differences in demographics and socio-economic conditions between
three groups of students

21. Using Explorative and Confirmatory Factor Analysis
Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senécal, C., & Vallières, E. F. (1992). The academic motivation scale: A measure of intrinsic, extrinsic, and
amotivation in education. Educational and Psychological Measurement, 52, 1003–1017.
• Factor analyses allow you to
determine whether there is
a particular structure in
your data (which hopefully
is in line with your
theoretical model)
• Identify which items stick
together, and which are in
other categories
• AMS indicate that items fall
neatly into expected
categories

22. Checking for reliability using Cronbach Alphas
• Cronbach alpha scores allow
you to identify whether
items within a construct are
reliable (>.60)
• Identify which constructs
are not reliable (<.60)
• Correlations between items
might indicate why α <.60
• Pre-vs post-test (aka test vs
re-test) good approach to
test stability of construct

23. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment
in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12
■ No significant differences between the three internationalisation categories in terms
of academic and social adjustment (H1 –H2).
■ Significant differences were found in terms of emotional adjustment, whereby South
Africans living in SouthAfrica (IaH) indicated significantly lower personal-emotional
adjustment scores relative to their peers. Similarly, significant differences were found
in terms of attachment (-H4)

24. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment
in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12
■ Substantial and significant differences were found between the three
internationalisation categories, whereby IaD students had significantly higher access
to technology, all medium to large in effect size

25. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment
in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12
■ The three categories of internationalisation had a significant impact on social adjustment, emotional adjustment, and attachment, but not
academic adjustment. South Africans living in South Africa (IaH) had higher social adjustment relative to IaD (H2), but lower emotional
adjustment and attachment relative to IaD (-H3, -H4).
■ Access to technology significantly predicted academic adjustment and emotional adjustment, indicating that students with better technology
access also had better academic adjustment (H5) and emotional adjustment (H7). Age positively predicted all four SACQ scores, indicating that
relatively older learners had better adjustment than younger learners. Gender and type of occupation (being a full time student, looking after
the family, and other occupation (i.e., retired from paid work, unable to work due to long-term sickness, unemployed) had no significant impact
on the SACQ scores. English as a first language also had no impact on SACQ scores, except for a negative impact on emotional adjustment,
while working part-time had a positive impact on emotional adjustment.
■ The majority of black UNISA students felt better adjusted to the distance learning setting than others. Finally, relative to first-year students
those who studied in second-year, third-year, and post-graduate level had significantly lower attachment towards UNISA. Furthermore, those
at third-year and post-graduate level had significantly lower (self-reported) academic adjustment and personal-emotional adjustment relative
to first-year students.

26. A potential model for predicting school dropout
• Psychometric instruments
can be used to understand
why and how people
behave, act, think, etc.
• Build theoretical models
how one construct links to
another

27. Role of motivation on dropout vs persistent students
• Can be used to compare
basic descriptives
• Test hypotheses

28. Case Study 2:What predicts (international) student progression?
Input Process Output
Learner characteristics
(incl. prior education, gender,
cultural background)
Academic adjustment
(incl. personal-emotional adjustment,
attachment to institute)
Social adjustment
(incl. study support, satisfaction with social
Environment, financial support)
Family characteristics
(incl. support, finance, child-
care)
Learning design
(incl. assessment, learning
materials, communication)
Engagement with learning
(incl. VLE engagement, attending sessions,
submitting assignments, social media)
Academic performance
over time
(incl. grades, credits, GPA)
Degree outcomes
(incl. Employment, migration)
Interviews with 140+ UNISA students, 40+ stake holders, country reports

29. Thank you!
Questions?

30. How to analyse questionnaire data:
an advanced session
Prof Bart Rienties
A special thanks to Dr Rachel Slater, Dr Christine
Thomas & Dr Jenna Mittelmeier