This document summarizes and critiques a research paper that examined the effects of handicap source, frequency, and performance on observer impressions. It used a 2x2x2 factorial ANOVA/MANOVA design. The summary found partial support for the paper's 3 hypotheses but noted low variance explanation and some contrary findings. The critique raised concerns about not testing assumptions, excluding variables from MANOVA, and small effect sizes. It concluded the paper advanced the field but improvements could have been made by addressing the critique's methodological points.

1. July 2014 updated
Prepared by Michael Ling Page 1
QUANTITATIVE RESEARCH METHODS
SAMPLE OF
ANOVA/MANOVA ANALYSIS
Prepared by
Michael Ling
Reference: McElroy, J. C, & Crant, J. M. (2008). “Handicapping: The effects of its
source and frequency,” Journal of Applied Psychology, Vol. 93, 893-900.

2. July 2014 updated
Prepared by Michael Ling Page 2
I. INTRODUCTION
Prior to the event of performance, handicapping meant setting up to “deflect
blame away” in case of failure and “accept credit for success by having overcome
difficulty” in case of success. Handicapping was considered a “direct impression
management tactic” in relation to the “credit or blame attributed to and level of affect”
allocated towards the actor by an observer.
Past research in handicapping was limited to single instance of self-
handicapping. This paper contributed to handicapping research by extending a
single instance self-handicapping event into a much broader framework where
multiple instances of handicapping and indirect handicapping were evaluated.
The research question was to which extent that frequency and sources of
handicapping could influence reactions of an observer and his/her perceived
credibility about the actor concerned. These effects were further examined in the
context of success or failed performance. Three hypotheses were proposed and
ANOVA/MANOVA procedures were used as the research method.

3. July 2014 updated
Prepared by Michael Ling Page 3
II. SUMMARY
A 2x2x2 factorial ANOVA/MANOVA design was used to examine the main and
interaction effects of three factors - handicap source (self or other), frequency (once
or multiple times) and performance (success or failure) – on credit/blame,
interpersonal affect and perceived credibility. The sample size was 246 with an
average cell size of 28 to 34, and participants were randomly assigned to one of
eight cells. Manipulation checks were conducted on the measurement scales to
ensure their consistency and reliability.
Partial support was found in hypothesis 1, which stated that frequency
moderated the source of handicapping and the observer impressions where (i)
impressions (credit/blame and interpersonal affect) were more favourable in third-
party handicaps than self-handicaps; and (ii) increased frequency reduced
impressions more strongly in self-handicaps then third-party handicaps. The results
showed that there was (i) a significant Source*Frequency effect (p < .05) on the set
of dependent variables (MANOVA); (ii) a significant Source*Frequency effect (p<.05)
on interpersonal affect (ANOVA); and (iii) a moderating effect of Frequency on
Source and Interpersonal affect. However, no significant Source*Frequency
interaction effect was found on credit/blame (ANOVA).
Partial support was found for hypothesis 2, which stated that frequency
moderated the source of handicapping and the perceived credibility of the
handicapping information where (i) handicaps were more credible in third-party
handicaps than self-handicaps; and (ii) increased frequency reduced credibility more
strongly in self handicaps than third-party handicaps. The results showed that there

4. July 2014 updated
Prepared by Michael Ling Page 4
was Source*Frequency interaction effect (p < 0.05) on Credibility (ANOVA). Despite
the significant interaction effect, Eta-squared (η2) was 2 percent. Contrary to
expectations, self-handicaps were found more credible than third-party handicaps in
single handicaps.
Partial support was found for hypothesis 3, which stated that performance
moderated the source and frequency of handicapping and the observer impressions
and perceived credibility where (i) impressions were more favourable following failed
than successful performance; and (ii) perceived credibility was more favourable
following failed than successful performance. It was found that multiple handicaps
decreased credibility for all handicaps except in the case of third-party handicaps
following failed performance.

5. July 2014 updated
Prepared by Michael Ling Page 5
III. CRITIQUE
The use of ANOVA/MANOVA procedures were appropriate because it
examined the main and interaction effects of independent categorical variables
(Source, Performance and Frequency) on multiple dependent interval variables
(Credit/blame, Interpersonal affect and Credibility) by comparing group differences.
The sample design suggested that a balanced design was adopted. In
MANOVA, the cell sizes should be roughly equal because normality of the dependent
variables was important.
The coefficient alpha values for the measurement scales were high and the
outcomes of the manipulation checks were satisfactory.
A key weakness of the paper was that it did not specify the assumptions of the
ANOVA/MANOVA tests. For example,
i. No results were provided for univariate and multivariate normality that
the dependent variables, and their combinations, were distributed
normally. No Scatterplots were checked for linear relationships among
the dependent variables.
ii. No results of multicollinearity were provided to examine the correlations
of the dependent variables.
iii. No results of multivariate outliners, such as Maximum Mahalanobis
Distance, were provided.
iv. No results of homogeneity of the covariance matrices, such as Box’s M
test, were provided. If Box’s M test showed the covariance matrices
were significantly different across levels of the independent variables, it

6. July 2014 updated
Prepared by Michael Ling Page 6
indicated an increased possibility of Type I error and hence there was a
need to use a smaller error region than p < .05.
v. No evidence was provided for the independence of observations. As
the questionnaires were completed by the respondents in their
workplaces, it was probable that the respondents might have discussed
the questions amongst themselves. Data quality would be a potential
issue.
Hypothesis 1
Despite that the Source*Frequency interaction effects were reported
significant in Hypothesis 1, there were a few areas of concerns:-
i. Wilks’s λ was a measure of the percent of variance in the dependent
variables that was not explained by differences in the level of the
independent variable. Wilks’s λ for Source*Frequency was .97 in the
MANOVA test, which meant that 97 percent of variance was still
unexplained.
ii. The Eta-squared for Source*Frequency was 3 percent, which meant
the percent of total variance in the dependent variable explained by the
variance between groups formed by the independent variable was only
3 percent, which not very impressive despite a significant result.
iii. If the MANOVA omnibus test was significant, it was common practice to
conduct separate ANOVAs. However, considerations such as
Bonferroni adjustment should have been given to adjust the

7. July 2014 updated
Prepared by Michael Ling Page 7
significance cut-off level of ANOVAs in order to minimize the probability
of Type I error.
Hypothesis 2
An area of concern was that credibility had not been included in the MANOVA
omnibus test. The authors argued that they “felt that these reactions to handicapping
were conceptually distinct enough to preclude us from including all of the dependent
variables in just one… (MANOVA)”. This argument was not relied upon any
substantive literature but was based on what the authors “felt” it should be. On the
contrary, it was possible that credibility, credit/blame and affect were correlated. If
this was the case, the independent ANOVA would have ignored their interrelations
and substantial information would be lost. The resultant p values for the 1-way
independent ANOVA would have been incorrect.
The Bartlett’s Test of Sphericity could have been used to test the hypothesis
that the population correlation matrix was an identity matrix. If the determinant was
small, independence of the variables would be rejected and there was a need for
MANOVA.
Again, the Eta-squared for Source*Frequency was 2 percent, which was not
impressive at all despite a significant result.
No discussions were provided on the adjustments in relation to the treatment
of experiment-wise and testwise errors in the univariate ANOVA tests.
Hypothesis 3

8. July 2014 updated
Prepared by Michael Ling Page 8
The MANOVA omnibus result failed to find significance in the three-way
interactions effect of Performance*Source*Frequency on impressions. A three-way
interactions effect was found on credibility in the ANOVA test. Despite the interaction
effects were reported significant, a concern was that Eta-squared (η2) of the three-
way interactions effect was very low (0.02) in the ANOVA result.
Although all forms of handicaps were found more credible following failure
than success, their mean differences were very small in the range of 0.13 to 0.71, as
shown in the table below. A larger sample size could have been considered to test if
reasonable effect sizes could be achieved. Proper measures of effect sizes such as
Cohen’s d, as the difference in group mean divided by the pooled standard deviation,
should have been provided to measure effect sizes.
Success
Performance
Failure
Performance
Mean
Difference
Self-handicaps Single 4.11 4.82 0.71
Self-handicaps Multiple 3.65 3.92 0.27
Other handicaps Single 3.98 4.11 0.13
Other handicaps Multiple 3.61 3.92 0.31
Note: the mean values above were extracted from Figure 3.
Overall Assessment
Though the authors reported partial support of the three hypotheses, the low
values of Wilks’s λ and Eta-squared created some concerns. A large portion of the
variance was not accounted for in the model. Not all relationships, despite their
importance, suggested in the hypotheses were found significant. The assumptions of

9. July 2014 updated
Prepared by Michael Ling Page 9
ANOVA/MANOVA procedures were left untested. The exclusion of Credibility in the
MANOVA test was not adequately supported. No consideration was given to the
need for adjustment of significance criteria in ANOVA. Contrary evidence was found
with respect to self-handicaps and multiple handicaps in hypotheses 2 and 3
respectively. As a result, the conclusions drawn about the hypotheses were not
totally trustworthy.

10. July 2014 updated
Prepared by Michael Ling Page 10
III. CONCLUSION
The contribution of the paper rested on advancing a theoretical framework in
handicapping research, from single instance self-handicapping into multiple
instances of handicapping and indirect handicapping.
The authors started off on the right track by using a full factorial design and
ANOVA/MANOVA procedures as the research method to establish their hypotheses.
Unfortunately, the results were not quite satisfactory as all hypotheses were only
partially supported and the variances explained by the independent variables were
very negligible.
The research could have improved by addressing the concerns raised in this
critique. In particular, assumptions of the ANOVA/MANOVA procedures needed to
be tested and data quality needed to be reinforced, especially independence of
samples; a redesign of the scenario-based experiment methodology, for example, by
replacing questionnaire with testing respondents in a laboratory setting. Given that
there was only partial support of the three hypotheses, an important step would be a
review of the handicapping model from the theoretical perspective.