Presentation by Michael Best, Columbia, Jonas Hjort, Columbia, David Szakonyi, George Washington. SITE, Stockholm School of Economics, December 15th, 2017.

1. Individuals and Organizations as Sources of State Effectiveness
and Consequences for Policy Design
Michael Best
Columbia
Jonas Hjort
Columbia
David Szakonyi
George Washington
Public Sector Efficiency and Effectiveness:
Implications for Developing and Transition Countries,
SITE-Stockholm School of Economics
December 15 2017

2. Introduction
Same policy → different outcomes in different settings
1 / 14

3. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
1 / 14

4. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
1 / 14

5. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
1 / 14

6. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
Bureaucrats and public organizations different across settings
1 / 14

7. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
Bureaucrats and public organizations different across settings
How much of variation in policy outcomes due to bureaucracy?
1 / 14

8. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
Bureaucrats and public organizations different across settings
How much of variation in policy outcomes due to bureaucracy?
What do effective bureaucracies do differently?
1 / 14

9. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
Bureaucrats and public organizations different across settings
How much of variation in policy outcomes due to bureaucracy?
What do effective bureaucracies do differently?
Towards: How does optimal policy depend on who implements it?
1 / 14

10. Introduction
Same policy → different outcomes in different settings
VAT in low/high income countries
NREGA across India
Post office returning mail
Bureaucrats and public organizations different across settings
How much of variation in policy outcomes due to bureaucracy?
What do effective bureaucracies do differently?
Towards: How does optimal policy depend on who implements it?
We investigate these questions in the Russian public sector
1 / 14

11. Procurement in Russia
Decentralized procurement with centralized rules. 104,000
Federal, regional, municipal public bodies procure independently.
2 / 14

12. Procurement in Russia
Decentralized procurement with centralized rules. 104,000
Federal, regional, municipal public bodies procure independently.
No centralized civil service, procurement officers hired much like
private sector.
2 / 14

13. Procurement in Russia
Decentralized procurement with centralized rules. 104,000
Federal, regional, municipal public bodies procure independently.
No centralized civil service, procurement officers hired much like
private sector.
Key: Organizations observed working with multiple bureaucrats &
bureaucrats observed working with multiple organizations.
2 / 14

14. Procurement in Russia
Decentralized procurement with centralized rules. 104,000
Federal, regional, municipal public bodies procure independently.
No centralized civil service, procurement officers hired much like
private sector.
Key: Organizations observed working with multiple bureaucrats &
bureaucrats observed working with multiple organizations.
Majority (52%) of purchases by electronic descending auction.
2 / 14

15. Procurement in Russia
Decentralized procurement with centralized rules. 104,000
Federal, regional, municipal public bodies procure independently.
No centralized civil service, procurement officers hired much like
private sector.
Key: Organizations observed working with multiple bureaucrats &
bureaucrats observed working with multiple organizations.
Majority (52%) of purchases by electronic descending auction.
Detailed data on all procurement available in online procurement
register – the Unified Register of Federal and Municipal Contracts
We collect data on all auction requests, review protocols, bids,
and final contracts 2011–2015
2 / 14

16. What Happens When Bureaucrats Move?
0
1
2
3
−1 0 1 2
Time (0 = last day with old bureaucrat)
StandardizedPriceResiduals
Trajectory
1 to 1
4 to 4
3 / 14

17. What Happens When Bureaucrats Move?
0
1
2
3
−1 0 1 2
Time (0 = last day with old bureaucrat)
StandardizedPriceResiduals
Trajectory
1 to 1
1 to 4
4 to 1
4 to 4
4 / 14

18. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
5 / 14

19. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
5 / 14

20. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
4 Challenges
5 / 14

21. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
4 Challenges
1. Like-for-like comparisons: quality-adjusted prices
5 / 14

22. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
4 Challenges
1. Like-for-like comparisons: quality-adjusted prices
2. Effects only identified within “connected sets” linked by mobility
5 / 14

23. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
4 Challenges
1. Like-for-like comparisons: quality-adjusted prices
2. Effects only identified within “connected sets” linked by mobility
3. Causal identification requires “exogenous mobility”
5 / 14

24. Decomposing Variation in Prices Paid
Model outcome yi ∈ {log (pi) , ni} for item i purchased for
organization j by bureaucrat b (i, j) as
yi = Xiβ + ˜αb(i,j) + ˜ψj + εi
Xi includes log quantity, good FEs, month FEs, size, region
Goal: Estimate magnitude of Var (˜α) and Var ˜ψ
4 Challenges
1. Like-for-like comparisons: quality-adjusted prices
2. Effects only identified within “connected sets” linked by mobility
3. Causal identification requires “exogenous mobility”
4. Sampling error inflates variances
5 / 14

25. Decomposing Variation in Prices Paid
Yi =Xiβ + αb(i,j) + ψj + γs(b,j) + εi
Var (Yi) =Var αb(i,j) + Var (ψj) + . . .
Prices (P) (s.e.) Participation (N) (s.e.)
(1) (2) (3) (4)
s.d. of Y 2.417 1.355
s.d. of Y | good, month 1.646 1.241
s.d. of Bureaucrat Effects 1.031 (0.0462) 0.919 (0.0418)
s.d. of Organization Effects 1.068 (0.0496) 0.888 (0.0468)
s.d. of Total Bur + Org Effects 1.036 (0.00126) 0.710 (0.00358)
Adjusted R-squared 0.955 0.837
Sample Size 11,228,122 11,228,122
6 / 14

26. Crude Counterfactual: Improving Bureaucrats
7 / 14

27. Counterfactual: Improving Bureaucrats & Organizations
8 / 14

28. Correlates of Bureaucrat/Organization Effectiveness
What do good and bad bureaucrats and organizations do
differently?
What attributes do good and bad bureaucrats have?
9 / 14

29. Correlates of Bureaucrat/Organization Effectiveness
What do good and bad bureaucrats and organizations do
differently?
What attributes do good and bad bureaucrats have?
Using detailed data on the levers they control at the “qualification
stage”, the “auction stage”, and “the contracting stage” – and
how firms respond to their behaviors
9 / 14

30. Correlates of Bureaucrat/Organization Effectiveness
What do good and bad bureaucrats and organizations do
differently?
What attributes do good and bad bureaucrats have?
Using detailed data on the levers they control at the “qualification
stage”, the “auction stage”, and “the contracting stage” – and
how firms respond to their behaviors
Large number of rhs variables. For variable selection present
1. Pairwise regression coefficients (standardized rhs vars)
2. Post-LASSO coefficients
9 / 14

31. Pairwise Regressions Post−LASSO Regression
−0.10 −0.05 0.00 0.05 0.10 −0.10 −0.05 0.00 0.05 0.10
Supplier Turnover
Supplier Assets
Supplier log Employees
Number of Contract Revisions
Average of Losing Bids / Winning Bid
1[Winner is From Same Region]
1[Auction Held]
Number of Items Purchased
Success Rate
In−house Bureaucrat
Admission Rate to Auction
Number of Applicants
Time to Prepare Documents
Deposit / Reserve price
Number of Products
Lot Size
Standardized Coefficient
Variable
10 / 14

32. Bid Preferences for Domestic Products
Motivation: classic mercantilist “support local business”
Each spring presidency issues list of goods receiving preferences
Winner supplying foreign good receives 85% of winning bid
Winner supplying domestic good receives 100% of winning bid
11 / 14

33. Bid Preferences for Domestic Products
Motivation: classic mercantilist “support local business”
Each spring presidency issues list of goods receiving preferences
Winner supplying foreign good receives 85% of winning bid
Winner supplying domestic good receives 100% of winning bid
Sources of time × product variation:
Intra-year: Preferences switch on in late spring, off on 12/31
Across goods: Different goods on list each year
⇒ Difference-in-differences strategy:
11 / 14

34. Bid Preferences for Domestic Products
Motivation: classic mercantilist “support local business”
Each spring presidency issues list of goods receiving preferences
Winner supplying foreign good receives 85% of winning bid
Winner supplying domestic good receives 100% of winning bid
Sources of time × product variation:
Intra-year: Preferences switch on in late spring, off on 12/31
Across goods: Different goods on list each year
⇒ Difference-in-differences strategy:
⇒ Triple-differences: Interact with ˆαb, ˆψj from regular auctions
11 / 14

35. Average Policy Impact: Graphical Analysis
2011lawcomesintoEffect
2013lawcomesintoEffect
2012lawcomesintoEffect
2014lawcomesintoEffect
−0.4
−0.2
0
0.2
0.4
PriceResiduals
Treatment Group
Not Preferenced
Preferenced
−0.5 SD
[−0.70]
0
0.5 SD
[0.70]
2011 2012 2013 2014 2015
Time
PriceResiduals
Difference between Groups:
Preferenced − Unpreferenced
12 / 14

36. Policy Impact Depends on Bureaucratic Effectiveness
q
q
q
q q
q
q
q
q
−0.4
−0.2
0.0
−1 0 1
Bureaucrat Effectiveness
TreatmentEffect(rel.todecile1)
Panel A: Heterogeneity in Effect on Price by Bureaucrat Effectiveness
q
q
q q
q
q q
q
q
−0.4
−0.2
0.0
−1 0 1
Organization Effectiveness
TreatmentEffect(rel.todecile1)
Panel B: Heterogeneity in Effect on Price by Organization Effectiveness
13 / 14

37. Conclusion
Procurement as window into determinants of state’s effectiveness
14 / 14

38. Conclusion
Procurement as window into determinants of state’s effectiveness
Weber (1922) “Bureaucracy develops the more perfectly, the more
it is ‘dehumanized’... The individual bureaucrat cannot squirm out
of the apparatus into which he has been harnessed”
14 / 14

39. Conclusion
Procurement as window into determinants of state’s effectiveness
Weber (1922) “Bureaucracy develops the more perfectly, the more
it is ‘dehumanized’... The individual bureaucrat cannot squirm out
of the apparatus into which he has been harnessed”
∼ 60% of variation in prices paid due to bureaucrats & public
organizations. Far from Weberian ideal.
14 / 14

40. Conclusion
Procurement as window into determinants of state’s effectiveness
Weber (1922) “Bureaucracy develops the more perfectly, the more
it is ‘dehumanized’... The individual bureaucrat cannot squirm out
of the apparatus into which he has been harnessed”
∼ 60% of variation in prices paid due to bureaucrats & public
organizations. Far from Weberian ideal.
What do effective bureaucrats do?
Attract entry by diverse firms: ⇒ policy to encourage participation
Experienced/in-house burs better: ⇒ training/end-user feedback?
14 / 14

41. Conclusion
Procurement as window into determinants of state’s effectiveness
Weber (1922) “Bureaucracy develops the more perfectly, the more
it is ‘dehumanized’... The individual bureaucrat cannot squirm out
of the apparatus into which he has been harnessed”
∼ 60% of variation in prices paid due to bureaucrats & public
organizations. Far from Weberian ideal.
What do effective bureaucrats do?
Attract entry by diverse firms: ⇒ policy to encourage participation
Experienced/in-house burs better: ⇒ training/end-user feedback?
Policy responses to limited effectiveness
Optimal bid preferences decreasing in effectiveness
Tailoring policy to capacity substitute for raising capacity
14 / 14