Paul Njoku's presentation to evaluation workshop organised by Scottish Communicators Network

1. Scottish Communicators Network
Evaluation Workshop
Paul Njoku
Cabinet Office, UK 14th & 15th May 2014

2. A
• Welcome & Introductions
• GCS Mandatory Evaluation Project (MEP) - Overview
B
• Why evaluate
• Barriers to evaluation
C
• How to go about it
• 4 stage evaluation process - The big IDIA
• Main performance categories - KPI indicators sources
• Take Away - Tips
D
• GCS Mandatory Evaluation Project (MEP) - Update & Next steps
Agenda
Group interactive session

3. Government Communications Review:
Summary findings
• Tactically strong, strategically weak
 Half of departments had no communication strategy
• Poor links between policy and communications
• Major activity rarely evaluated
• Inconsistent and variable standards and approaches
• Little join up across departments
• Digital skills falling behind private sector

4. GCS Mandatory Evaluation Project – Overview
Mandatory Evaluation Project (MEP) is a key part of
Government Communication reform programme.
Objective: To create a ‘step change’ in approach to
evaluation of communication activities across departments.
Four Key deliverables:
1.Develop high-level performance frameworks and dashboards.
2.Introduce mandatory evaluation training across all comms disciplines.
3.Enhance evaluation capability across the GCS.
4.Develop an evaluation ‘centre of excellence’ & promote best-practice.
A

5. Agenda
13.45 – 14.00 Welcome and introduction – Paul
14.00 – 14.10 Scene setting: Conference theme, aims and objective
14.10 -14.30 Guest Speaker: Selvin Brown - DWP Testimony
14.30 – 14.50 Vision, objectives and challenges for Government Communications
Alex Aiken, Exec Director of Government Communications
15.00 -15.45 Group Exercise
15.45 – 16.00 TEA BREAK
16.00 – 16.30 Guest Speaker: Spenser Fox (CEO, Reputation Institute
Topic: Significance of Reputation as a key outcome measure
16.30 – 16.50 Guest Speaker: Gareth Evans (Independent Consultant-MR2)
Topic: Communication ROI case study for public sector
16.50 – 17.05 Guest speakers: Q & A Panel
17.05 -17.15 Evaluating communication activity – Never an optional extra
Sean Larkin, Head of Government Communications Policy and Capability
17.15 -17.20 Next Steps & Close
Departments & ALBs - Membership
40 Depts. /ALBs
80 ‘evaluation
champions’
A1

6. STEPCHANGE
No Evaluation happening at
Campaign / Project level
Evaluation @ Project / Campaign
level but no overall Comms KPI’s
(Outcome measures)
Evaluation & Comms KPI’s
exist but partially aligned
with dept. strategic
priorities
Fully functional Comms
performance centre (hub) aligned
with dept. strategic priorities
Stage 1
Stage 2
Stage 3
Stage 4
Dept. Outcomes
measures aligned
with Govt. priorities
TIME
Journey (start)…
39%
5%
56%
A2

7. 7
Why evaluate?
B

8. John Wanamaker
Inventor of mass
retailing in the United
States
"Half the money I spend on
advertising is wasted;
the trouble is I don’t know
which half."

9. Group Exercise (15mins)
• Why do we need to evaluate?
• What are the barriers ?

10. Context
• Austere times
 Need to make every £ count
 View as an investment not an expense
• Media landscape & consumption patterns
 Evidence of what works and what does not
 Optimise use of scarce resources
• The role of communications under scrutiny
 How it supports achievement of policy outcomes
 Business planning & activity prioritisation
 Sharing best practice
B1

11. 1111
Barriers – stopping it happening
Insufficient time /
resource / budget
Lack of SMART or
unrealistic policy
objectives / targets
Difficulty accessing the
right data / tools
Culture & entrenched
behaviours
Gaps in evaluation
standards & capability
Lack of integrated
communication strategy
linked to policy
objectives
B2

12. 12
Strategic Alignment
 Start with the policy aim, then develop communications objectives that
help deliver this.
 Objectives should be measurable, focused on outcomes not
outputs, and related to changing attitudes and/or behaviour.
Setting the right objectives is crucial
C1

13. 13
Strategic AlignmentC2
Strategic Objectives
& KPIs
Comms Objectives,
& KPIs
Activity objective

14. Evaluation stages –The Big IDIAC3
Identify
The scope of your project
Develop
Your evaluation plan
Implement
Gather data to measure
performance
Analyse & report
Performance against plan
1
2
3
4
Task 1: Define what you need to evaluate by asking:
• What activity am I evaluating?
• What do I know & what factors could affect the outcome?
• What is my evaluation expected to achieve?
Output: Summary of your proposed evaluation approach
Task 2: Define how you’ll measure success:
• Set SMART objectives & define your target audience
• Map out how activity will work
• Set performance metrics (KPIs) & agree baseline / targets
Output: Draft evaluation plan
Task 3: Identify and gather evaluation data:
• Make most of existing data
• Gather additional data (research, feedback & proxies)
• Review data gaps (more budget ?) manage expectations
Output: Completed evaluation plan
Task 4: Assess the success of your activity:
• Analyse effectiveness & provide insights for future
• Demonstrate efficiency and value for money
• Demonstrate role of communications in supporting
policy objective delivery (outcome)
Output: Final evaluation report

15. Key performance indicator basketC4
Activity Effectiveness Result

16. 1. Strategic alignment – Ensure activity objectives are SMART,
aligned and supports policy delivery.
2. Business impact – Always measure true business impact
(outcome) rather than outputs and out-takes
3. Big IDIA – Try to adopt the suggested frameworks and follow
the big IDIA stages.
4. Continuous improvement – Ensure results drive appropriate
actions and any learnings inform future activities.
5. Best practice – Be objective, share results and make
evaluation an integral part of your communications approach.
Top TipsC5

17. GCS Mandatory Evaluation Project
Progress update & Next steps
D

18. D1: Performance Frameworks & Dashboards
1. Performance frameworks /dashboards (Phase one: 37 departments /ALBs)
 33 depts. /ALBs (89%) with complete or near complete frameworks.
 27 depts./ALBs (73%) expected to have fully functional dashboards by April 14.

19. Headline measures Audience Team activity / Performance indicators Resource Risks / Concerns
Stakeholders
Partners
Staff
Claimants
Staff
People
• FTE (actual vs. plan)
• FTE (team and profiles)
• Flexible resource stats
Financials
• Allocations (Staff vs. Non-staff costs)
• Spend vs. Forecast
• ERF submissions / spend (teams)
1 2 3 4 5
Directorate Targets
• Sickness levels: AWDL (teams)
• Directorate engagement score
• HSE incident:
• Compliance targets (e.g.)
 mandatory training
 performance discussions held
 Skills manager
Risk Impact Action Lead
1 H
2 M
3 L
Risk register
No Raised by date Status
1
2
3
3 C’s / Hot Topic register
Press office ‘Forward Look’ – up coming events
12 months rolling activity communications plan
Prioritisation activity list
Planning Section
New
Existing
6
SRP1: Welfare Reform
Business Objective: Introduce the Universal
Credit and other reforms to simplify the
welfare system; Ensure that the system
always incentivises work and work always
pays; Help to make the welfare system
affordable longer term.
Comms Objective / Outcome:
W
S
R
Team / Area Input / output / outcome indicators)
Universal
Credit
• Milestones – performance up date
• Nos. engaged / events held
• No. of web visits / downloads.
• No. of comms issued
• Staff wavelength / People survey
• % Net positive media coverage
SRP2: Labour Market (GBW)
Business Objective: Implement and manage
the Work Programme; Deliver personalised
package of support to get people into work.
Comms Objective / Outcome:
Stakeholders
Employers
Individuals
Staff G
B
W
Team / Area Input / output /Outcome (indicators)
Labour
Market
• Milestones – performance up date
• Nos. events held / attendance levels
• No. of web visits / downloads.
• No. of comms issued
• Staff wavelength / People survey
• % Net positive media coverage
SRP3: Tackling Poverty
Business Objective: Develop a welfare
system that recognises work as the primary
route out of poverty; Implement a the Social
Justice and Child Poverty Strategies that
focuses on eradicating child poverty by 2020.
Comms Objective / Outcome:
P
O
V
E
R
T
Y
Team / Area Input / output metrics (indicators)
Social
Justice
• Milestones – performance up date
• Nos. events held / attendance levels
• No. of web visits / downloads.
• No. of comms issued
• Staff wavelength / People survey
• % Net positive media coverage
Stakeholders
OGDs
General public
Staff
SRP4: Pensions Reform (PR)
Business Objective: Provide decent State
Pensions; Encourage employers to provide
high quality pensions and make automatic
enrolment and higher pension saving a
reality; Phase out the default retirement age
Comms Objective / Outcome:
P
R
Team / Area Input / output metrics (indicators)
Pensions
• Milestones – performance up date
• Camp tracker data (Ind / employers)
• No. of web visits / downloads.
• No. of comms issued
• % Net positive media coverage
• Auto enrolment data
Stakeholders
OGDs
General public
SRP5: Disability Equality
Objective: Develop and implement a disability
strategy to enable disable people fulfil their
potential; Support more independent living for
disable people through an effective strategy
and evidence-based policies for disability
benefits; Support disabled people take up
employment opportunities through specialist
programmes. Comms Objective / Outcome:
D
I
S
A
B
I
L
T
Y
Team / Area Input / output metrics (indicators)
A2 work
DLA
PIP
ESA
• Milestones – performance update
• No. of events (dialogues) carried out
• No. of partnerships established
• No. of web visits / downloads.
• No. of comms issued
• % Net positive media coverage
• % take up rates (PIP)
Stakeholders
OGDs
General public
Disabled people
Claimants
Staff
SRP6: Improving our services
Objectives
• Effective communications delivery
• Staff engagement improvements
• Public and Corporate info delivery
• Secure and effective digital comms
• Resource management / R&E
Outcomes:
People Survey – Headline scores
People survey scores for: B31,41,47,50-
54, B06-08.
S
E
R
V
I
C
E
S
Teams Input / output metrics (indicators)
Internal
Comms
• Milestone – performance update
• No of staff comms issued
• No. of proactive briefings (SCS & Line mgrs)
• No. of bright ideas (submitted vs. Implemented)
• Intranet staff visits to:
 Home page / DWP Story pages
 Have your say / forum discussions
• Wavelength scores (QB1, QB5,QC1,QE1, QE3)
Corp Info • No. of leaflets / forms delivered or reduced
• No. of evaluation reports requested / delivered
• No. of jobs successfully delivered
• L&D core training delivered / attendance levels
• People survey scores for: B22, B25-26
R&E
Business MI*
Existing

20. D2: Mandatory Evaluation Training
2. Evaluation training - Free half-day courses:
 34 training sessions delivered.
 541GCS staff trained to date.
 4,402 unique page views to our online evaluation guide.
Evaluation Training (400 free places)*
• Introduction to Evaluation for Marketing
• Introduction to Evaluation for Press & Media
• Guest speaker Events
* NB: The nature of the programme this year 2014/15 under review (TBC)

21. D3: Evaluation Standards & Capability
• New evaluation standards – Three levels
• Capability audit to start soon - April / May 2014
• Tailored package of support & training for each level
Advance
Practitioner
Foundation

22. D4: Evaluation ‘ centre of excellence’
• Established an Evaluation Council - Industry expert who meet regularly
and act as a sounding board for new evaluation thinking as well as to review and validate
government evaluation plans and outputs.
• Creating a repository for:
 Case studies
 Evaluation Guide
 Performance Framework Top tips
 Standard Metrics

23. What Next
• GCS Live - 1st April 2014
• Govt. Comms Plan launched – 12th May
• Capability Audit – April /May 2014
• Developing a tier package of support for each level
• Phase 2 roll-out across all departments / ALBs (330)
• Regular updates to GCS Programme board and Council
 Invite departments / ALB to share their experience & approach
• Lots more events planned – Register on GCS now!

24. 24
Thank you!
Contact: Paul Njoku
Email : paul.njoku@cabinet-office.gsi.gov.uk
Web link to guide: https://gcn.civilservice.gov.uk/guidance/evaluation/

25. Appendix

26. Products
Customer
Service
Investments
Employment
Branding
Public
Relations
Marketing
Social
Responsibility
MEDIA
(Traditional,
Social)
Topic Experts,
Leaders,
Friends/Family
Perceptions &
expectations
Supportive
Behaviour
Results
What your department
Says/Does
Direct Experience
What Others Say
How reputation is created

27. The attributes and dimensions have
different meanings and importance for
different stakeholders. Beneath the 7
dimensions, 32 attributes underpin the
individual dimension themes. Different
stakeholder groups typically have unique
attributes that are found more important
than others (reputation drivers).
Reputation Attributes
Reputation Dimensions
The seven dimensions specify at a more operational level,
which aspects are most important for stakeholders’
perceptions and expectations – i.e. what’s driving a
company’s reputation
Reputation Strength
A measure of the emotional connection.
Reputation has a positive/negative impact on
support. An increase in reputation = an increase
in support. Support (such as buying products
and services, saying something positive, giving
the benefit of doubt in times of crisis (etc.) leads
to increased business results
Supportive Behaviour
Reputation drivers & dimensions

28. Measuring outputs
28
Content metrics
Favourability
Messages
Themes / Issues
Campaigns
Article type
Proactive / reactive
Prominence
Calls to action
Delivery metrics
Volume
Opportunities to see
Reach and Frequency
Media type
Key publications
Journalists
Spokespeople
Regional breakdown
Impact measures
Output Metrics
Are we saying the right thing... ....to the right audience?
Activity
Intermediary
Effect
Audience
Effect
‘Earned’ media
Model
Amplification
from social
media

29. Government analysis audit
29
Are we
saying the
right thing...
....to the right
audience
Metric
% of
projects
Favourability 100%
Message delivery 91%
Campaign tracking 48%
Theme / issues tracking 57%
Spokespeople / commentators 78%
Prominence 57%
Article type 43%
Proactive / reactive 22%
Calls to action 39%
Volume (number of articles) 100%
Opportunities to see (OTS) 74%
Reach and frequency 78%
Impact measures 26%
Media type 83%
Regional split 57%
Publications 87%
Journalists 87%
Advertising value equivalent AVE) 57%
Cost per thousand (CPM) 22%
Linking ouput to outcomes 22%
Content Metrics
Delivery metrics
Efficiency /
effectiveness
metrics