Scale Evaluation including determination of reliability and validity of questionnaire

1. Scale Evaluation
(Reliability and Validity)
By
Dr. Suhail Ahmad Bhat

2. Learning Objectives
• Basic understanding of scale evaluation methods.
• Understanding regarding different reliability techniques.
• Learning about various methods of testing scale validity.

3. Scale Evaluation

4. Reliability
• Reliability refers to the extent to which a scale produces consistent
results if repeated measurements are made.
• Systematic error does not affect reliability.
• Random error produces inconsistency.
• Reliability can be defined as the extent to which measures are free
from random error.

5. Test-re-test Reliability
• Identical scale
• Respondents- two different times
• Correlation between the scores is determined

6. Alternative Form Reliability
• Two equivalent forms of scale are constructed
• Addressed to same respondents at different times
• Scores from two alternative forms are correlated
• Problem- Difficult, time consuming, expensive, two alternative forms
should have same mean, variance, etc.

7. Internal Consistency
• Items are summated to form a total score for the scale.
• Two Approaches
Split-half
Coefficient Alpha

8. Validity
• Validity is the extent to which a test measures what we actually wish
to measure.
• Perfectly validity requires that there is no measurement error
• systematic error = 0
• random error = 0

9. Content Validity
• Also called face validity.
• It is subjective but systematic evaluation of how well the content of a
scale represents the measurement task at hand.
• For example, Bank image
• Major dimensions- range of products, quality of products, services of
the bank personnel, etc.

10. Criterion Validity
• Criterion validity reflects whether a scale performs as expected in
relation to other selected variables (criterion variables) as
meaningful criteria.
Example: Customer loyalty scale (Measurement variable)
Repeated purchasing (criterion variable)
• Based on time period involved criterion validity has two types-
• Concurrent validity
• Predictive validity

11. Construct Validity
• Construct validity addresses the question of what construct or
characteristic the scale is, in fact, measuring.
• Construct validity is the most sophisticated and difficult type of
validity to establish.
• Construct validity includes convergent, discriminant and
nomological validity.

12. • Convergent validity is the extent to which the scale correlates
positively with other measurements of the same construct.
• Discriminant validity is the extent to which a measure does not
correlate with other constructs from which it is supposed to differ.
• Nomological validity is the extent to which the scale correlates in
theoretically predicted ways with measures of different but related
constructs.

13. Graphical Display of 5 Construct CFA Model
Security
R23
R22
R24
R21
R20
WInf5
WInf6 WInf7 WInf8
S18
S17 S19
S16
I32
I33
I31
I34
PU41
PU40 PU42
Note: Measured variables are shown as a box with labels corresponding to those shown in slide. Latent constructs are an oval.
Each measured variable has an error term, but the error terms are not shown. Two headed connections indicate covariance
between constructs. One headed connectors indicate a causal path from a construct to an indicator (measured) variable. In CFA
all connectors between constructs are two-headed covariances / correlations.
PU43
Web
Information
Interactivity
Reliability
Perceived
Usefulness
WInf9
PU44
S15

14. Factor Loadings –
Convergent Validity
When examining convergent validity, we look at two additional measures:
(1) Average Variance Extracted (AVE) by each construct.
(2) Construct Reliabilities (CR).
The AVE and CR are not provided by AMOS software so they have to be calculated.
Factor loadings are the first thing to
look at in examining convergent validity.
Our guidelines are that all loadings
should be at least .5, and preferably .7 or
higher. All loadings are significant as
required for convergent validity. The
lowest is .658 (R21) and there are only
four below .70 (WInf5, R21, PU42 &
S19).
These are factor loadings but in
AMOS they are called “standardized”
regression weights.

15. Thank You