1. West Coast Data Conference 2016: Las Vegas
Toward a Measure of
Mission Transparency
Mark A. Hager
Rodney Machokoto
Chia-Ko Hung
Terry Driscoll
Arizona State University

2. West Coast Data Conference 2016: Las Vegas
Gist of the Project
 Nonprofits vary on their level of
reporting “transparency” on Form
990.
So?
 Pressures  Transparency
 Transparency  Donor
decisions

3. West Coast Data Conference 2016: Las Vegas
Form 990

4. West Coast Data Conference 2016: Las Vegas
“Transparency”
 Quality of mission statement and
program descriptions
 Completeness of narrative and
financial fields
 Reporting on form rather than
attachments

5. West Coast Data Conference 2016: Las Vegas
Data
 200 public charities drawn from
BMF with fiscal years ending in
first half of 2011 and at least
$200,000 in reported revenues.
 Currently working on full coding
of approximately 1200 cases.

6. West Coast Data Conference 2016: Las Vegas
Toward a Transparency
Index
 Part I
Description
Max
value
Quality of description of mission (Pt1) 5
Reports number of board members (Pt1) 1
Reports number of employees 1
Reports number of volunteers 1
Reports revenue streams 3
Reports expense category totals 3
Reports net assets or fund balance 3 17

7. West Coast Data Conference 2016: Las Vegas
Toward a Transparency
Index
 Part III
Description
Max
value
Part 3, Line 1
Quality of description of mission
(Pt3) 5
Part 3, Line 4a
Quality of description of program
service 5
Part 3, Line 4a
Expenses
Reports primary program expenses 2
Part 3, Line 4a
Revenues
Reports primary program revenues 2 14

8. West Coast Data Conference 2016: Las Vegas
Toward a Transparency
Index
 Part VI
Description
Max
value
Part 6, Line 1a Reports number of board members (pt6) 1
Part 6, Line 1b Reports number of independent board members 1
Part 6, Line 8a
Reports whether board meetings are
documented 1
Part 6, Line 12a
Reports whether organization has conflict of
interest policy 1
Part 6, Line 13
Reports whether organization has a
whistleblower policy 1
Part 6, Line 14
Reports whether organization has a document
destruction policy 1
Part 6, Line 15a
Reports whether organization has independent
review of compensation of CEO 1
Part 6, Line 15b
Reports whether organization has independent
review of compensation of other key employees 1

9. West Coast Data Conference 2016: Las Vegas
Toward a Transparency
Index
 Part VII
Description
Max
value
Part 7, Line 1
et al
Lists officers, directors, trustees, and
key employees
5
Part 7, Line 1
et al, Col. D
Reports their compensation 5
Part 7, Line 1
et al
Lists board members
3
Part 7, Line 2
Reports number of individuals
compensated more than $100,000
1 14

10. West Coast Data Conference 2016: Las Vegas
Description
Max
valu
e
Part 8, Line
1a
Reports revenues from federated
campaigns
1
Part 8, Line
1b
Reports revenues from membership
dues 1
Part 8, Line 1c
Reports revenues from fundraising
events 1
Part 8, Line
1d
Reports revenues from related
organizations 1
Part 8, Line Reports revenues from government
Toward a Transparency
Index
 Part VIII

11. West Coast Data Conference 2016: Las Vegas
Description
Max
valu
e
Part 9
Expenses reported in relevant
sections and generally add to total
10
Part 9, Col.
D
Penalty (-8) for organizations
reporting substantial contributions
and no fundraising expenses
0
1
0
Toward a Transparency
Index
 Part IX

12. West Coast Data Conference 2016: Las Vegas
Description
Max
valu
e
Part 10, Lines 1-
16
Assets reported on relevant lines
and generally add to total
2
Part 10, Lines 17-
26
Liabilities reported on relevant
lines and generally add to total
2 4
Part 12, Line 2a
Reports whether financial
statements had review by
independent accountant
1
Reports whether financial
Toward a Transparency
Index
 Parts X and
XII

13. West Coast Data Conference 2016: Las Vegas
Distribution: 75 test cases
Range: 39 to
70

14. West Coast Data Conference 2016: Las Vegas
Coding Mission Statements
 The Transparency Index asks:
“How helpful is this mission
statement?”

15. West Coast Data Conference 2016: Las Vegas
Coding Mission Statements
 Skipping the question is not
transparent
 If present, coders rate from 1 to 5:
1: Very general; unhelpful
2:
3:
4:
5: Detailed; thorough

16. West Coast Data Conference 2016: Las Vegas
 JK rates this a 1
 Terry rates this a 4

17. West Coast Data Conference 2016: Las Vegas
 Terry rates this a 1
 Rodney rates this a 5

18. West Coast Data Conference 2016: Las Vegas
Inter-rater Reliability
 Degree of agreement among
raters
 Different approaches to
measurement:
- Joint probability of agreement
- Cohen’s kappa, Fleiss’ kappa
- Correlation
- Limits of agreement

19. West Coast Data Conference 2016: Las Vegas
Krippendorf’s alpha
 Kα = 1 –
 Not an option in standard
statistical packages
 Andrew Hayes has publically
available macros for SAS,
SPSS, & Mplus
observed
disagreement
chance
disagreement

20. West Coast Data Conference 2016: Las Vegas
Krippendorf’s alpha
 DeSwert 2012: “Kα =.80 is often
brought forward as the norm for
a good reliability test, with a
minimum of .67 or even .60.”
 Kα for P1 mission statement,
200 test cases: 0.592

21. West Coast Data Conference 2016: Las Vegas
DV: Variance in raters; n=75
M1 M2
Expenditures (ln) * *
% Revenues from
Contributions ns
Transparency Index ns
TI: Part 1 ns
TI: Part 3 ns
TI: Part 6 ns
TI: Part 7 - **
TI: Part 8 ns
TI: Part 9 ns
Constant ns ns

22. West Coast Data Conference 2016: Las Vegas
Complexity = Variance
 The organization size
increases, variance in mission
transparency assessment
increases.

23. West Coast Data Conference 2016: Las Vegas
Opacity = Variance
 As transparency of officer participation
and compensation decreases,
variance in mission transparency
assessment increases.

24. West Coast Data Conference 2016: Las Vegas
Next Step 1
 Finish all the data entry and
coding.

25. West Coast Data Conference 2016: Las Vegas
Next Step 2
Come to grips with inter-rater
variation
 Old school: Train coders better,
discuss and “fix” variant cases.
 Alternative: Accept rater
variance as part of the reality of
user assessments. Ie, donors
are going to differ in their

26. West Coast Data Conference 2016: Las Vegas
Next Step 3
 Bring ratings for missions (part
1) and programs (part 3) into
Transparency Index.
 Press forward on larger
research agenda with the Index.

27. West Coast Data Conference 2016: Las Vegas
Thanks!
Mark A. Hager
Rodney, JK, and Terry
School of Community Resources &
Development
Arizona State University
Mark.Hager@asu.edu