This document discusses using statistical benchmarking to evaluate the efficiency of authorities that maintain highway networks. It explains that benchmarking can identify potential cost savings by determining authorities' minimum possible costs given their characteristics, quality levels, and citizen satisfaction. The document outlines models developed to estimate cost frontiers for highway maintenance, street lighting, and winter services. Authorities performing above the cost frontier are deemed inefficient, and the analysis can estimate potential savings if they adopted best practices. However, the document notes the analysis does not explain why costs gaps exist or are a maximum potential saving rather than absolute. It concludes benchmarking should be used alongside other analyses to help authorities understand how to improve.

1. Institute for Transport Studies
FACULTY OF ENVIRONMENT
Statistical Benchmarking:
What it can add to CQC
Phill Wheat
Senior Research Fellow
15th
October 2013

2. What is Good?
• Consider trying to determine if an authority is maintaining it’s
highway network efficiently
• Some authorities will perform well against some criteria and
badly against others
• Lack of a single measure which gives a handle on potential
cost savings that could be made
Cost per
highway km
Cost per
head
Different answers from
different measures
Which to
use?

3. What is Good?
• Even if there is only one measure, it maybe the case that
larger authorities have an implicit (scale) advantage over
smaller authorities
• Also other factors which influence (characterise) cost:
Customer perception and output quality
An authority has high cost, high quality
Another has low cost, low quality
The question we really want to know is whether either
can reduce cost and still maintain quality

4. What is Good?
Cost
Highway
Length
Cost frontier (drawn
for a given level of
quality)
.
.
.
A
B
C
O
Notice not a straight line…
Economies of Scale (unit
costs fall with the size of
the authority)

5. What is Good?
Cost
Highway
Length
Cost frontier (quality
= high)
.
.
.
A
B
C
O
Cost frontier (quality
= low)
.D
E
.
The bottom line here is that authorities A, B, C, D
and E are all achieving the minimum cost possible
for their size and quality, even though their unit
costs are quite different

6. • Authorities X and Y are above the frontier and so are inefficient
• By adopting best practice Authorities X and Y can reduce costs
without sacrificing output
What is Good?
Cost
Highway
Length
Cost frontier (drawn
for a given level of
quality)
Y
.
..
.
.X
A
B
C
.X’
O
Cost efficiency (TE)
= Minimum cost/
Actual cost
TEA=OX’/OX
0<=TE<=1
TE shows the
proportion of actual
cost which is needed if
the authority adopted
best practice (all other
things equal)

7. Approach
• Need to model the explicit relationship between cost and the
various cost drivers
– Include quality, physical outputs and citizen satisfaction
• Allows us to see the trade-off that exists between cost and
quality and citizen satisfaction
• Do this by exploiting the rich dataset collected
– Use of regression analysis to model minimum costs and then
compute the efficiency gap for each authority
– Gives clear indication as to the cost of incrementing quality and
citizen satisfaction holding everything else equal
Log(min cost) = a0 + a1 . Log (cost driver 1) + a2 . Log (cost driver 2)
+ …
… + ak . Log (cost driver k)

8. The cost categories that we
consider and cost drivers
Cost Category Cost Drivers Number of
complete
observations1
Highway Pavement
Maintenance (Reactive +
Structural Maintenance)
Total Highway Network Length
Average Highway Condition
CSM Condition of Road
Surfaces
90
Street Lighting Spend Number of lighting columns
% of units operational
CSM Street Lighting
107
Winter maintenance
spend
Length of Precautionary
Salting Routes
Number of Salting Runs
CSM Winter Maintenance
135
1
an authority for a given year

9. Overall Model Evolutions
• Highways model is fairly good in terms of the shape of the
frontier
• The street lighting model is okay but has some limits
• The winter service model is more of a concern… missing
key variables?

10. Summary of cost savings by
model
Model Average Score Minimum Score
Average
potential saving
by LA
Highway 71% 33% £ 4,669,803
Lighting 72% 33% £ 900,738
Winter_Op 86% 53% £ 318,688
• The Efficiency Score gives the minimum % of actual cost required
to produce the existing level of output and quality i.e. after
eliminating cost inefficiency.
• Average Scores seem plausible
• Minimum Scores of 33% are not uncommon from this type of
analysis. Probably a good reason (other than pure inefficiency)
for this LA to be different – Important: all we really measure is a
gap after controlling for a set of cost drivers

11. What we DO get from this
analysis
• An estimate of the minimum cost for each authority tailored to its
own characteristics, quality and citizen satisfaction
• A tool to conduct what if analysis e.g. How do (minimum) costs
change if:
– Authorities merged highway functions and increased network size for a
given operation,
– an Authority were to change quality (e.g. to improve average condition by
1%)
– Authority was prepared to allow public satisfaction to reduce
• A potential cost saving for each LA (if they closed the gap)
• Identification of the best performing authorities which is useful to
direct more process oriented analysis (to establish why there is a
gap)

12. The estimated cost frontier:
highway maintenance model
Unit Cost at:
5000km of highway
length = £2350 per km
20000km of highway
length = £2035 per km
ECONOMIES OF SCALE
Implication: A large authority
can be expected to have unit
costs 15% cheaper than a
small authority even when
both are efficient (producing at
minimum possible cost)

13. The cost of incrementing quality:
highway maintenance model

14. What we DO get from this
analysis
• An estimate of the minimum cost for each authority tailored to its
own characteristics, quality and citizen satisfaction
• A tool to conduct what if analysis e.g. How do (minimum) costs
change if:
– Authorities merged highway functions and increased network size for a
given operation,
– an Authority were to change quality (e.g. to improve average condition by
1%)
– Authority was prepared to allow public satisfaction to reduce
• A potential cost saving for each LA (if they closed the gap)
• Identification of the best performing authorities which is useful to
direct more process oriented analysis (to establish why there is a
gap)

15. What we DO NOT get from
this analysis
• An understanding of WHY there is a gap between actual
observed cost and minimum cost
– Analysis is useful to identify potential saving magnitude and direct to
which LAs do look like they are performing close to minimum cost
– But supplementary work is required to establish and disseminate
best practice
• The ‘gap’ is what is left over once we control for the effects
of cost drivers
– We can only control for the cost drivers (variables) that we can
collect
– Thus there maybe other reasons (outside of an LA’s control) which
my explain the gap
• The gap is likely to be a maximum possible saving rather than an
absolute

16. Going forward
• We have made a good start in the pilot study
• Clear ways to take this forward (more years of data, more
cost drivers, more cost categories)
• But needs to be taken forward along side other analyses
– Process benchmarking is important too
– Authorities need the why as well as the what!

17. Institute for Transport Studies
Phill Wheat
Senior Research Fellow
Institute for Transport Studies
University of Leeds
p.e.wheat@its.leeds.ac.uk
We offer taught courses, bespoke training and research
consultancy across a range of transportation policy,
regulation and economics areas