Scottish Communicators NetworkEvaluation Workshop

Paul NjokuCabinet Office, UK 14th & 15th May

2014

A

• Welcome & Introductions

• GCS Mandatory Evaluation Project (MEP) - Overview

B• Why evaluate

• Barriers to evaluation

C

• How to go about it• 4 stage evaluation process - The big IDIA• Main performance categories - KPI indicators sources• Take Away - Tips

D• GCS Mandatory Evaluation Project (MEP) - Update & Next steps

Agenda

Group interactive session

Government Communications Review: Summary findings

• Tactically strong, strategically weak Half of departments had no communication strategy

• Poor links between policy and communications

• Major activity rarely evaluated

• Inconsistent and variable standards and approaches

• Little join up across departments

• Digital skills falling behind private sector

GCS Mandatory Evaluation Project – Overview

Mandatory Evaluation Project (MEP) is a key part of Government Communication reform programme.

Objective: To create a ‘step change’ in approach to evaluation of communication activities across departments.

Four Key deliverables:

1.Develop high-level performance frameworks and dashboards.

2.Introduce mandatory evaluation training across all comms disciplines.

3.Enhance evaluation capability across the GCS.

4.Develop an evaluation ‘centre of excellence’ & promote best-practice.

A

Agenda13.45 – 14.00 Welcome and introduction – Paul

14.00 – 14.10 Scene setting: Conference theme, aims and objective

14.10 -14.30 Guest Speaker: Selvin Brown - DWP Testimony

14.30 – 14.50 Vision, objectives and challenges for Government Communications

Alex Aiken, Exec Director of Government Communications

15.00 -15.45 Group Exercise

15.45 – 16.00 TEA BREAK

16.00 – 16.30 Guest Speaker: Spenser Fox (CEO, Reputation Institute

Topic: Significance of Reputation as a key outcome measure

16.30 – 16.50 Guest Speaker: Gareth Evans (Independent Consultant-MR2)

Topic: Communication ROI case study for public sector

16.50 – 17.05 Guest speakers: Q & A Panel

17.05 -17.15 Evaluating communication activity – Never an optional extra

Sean Larkin, Head of Government Communications Policy and Capability

17.15 -17.20 Next Steps & Close

Departments & ALBs - Membership

40 Depts. /ALBs

80 ‘evaluation champions’

A1

ST

EP

CH

AN

GE

No Evaluation happening at Campaign / Project level

Evaluation @ Project / Campaign level but no overall Comms KPI’s

(Outcome measures)

Evaluation & Comms KPI’s exist but partially aligned

with dept. strategic priorities

Fully functional Comms performance centre (hub) aligned

with dept. strategic priorities

Stage 1

Stage 2

Stage 3

Stage 4

Dept. Outcomes measures aligned

with Govt. priorities

TIME

Journey (start)…

39%

5%

56%

A2

7

Why evaluate?

B

John WanamakerInventor of mass

retailing in the United States

"Half the money I spend on advertising is wasted;

the trouble is I don’t know which half."

Group Exercise (15mins)

• Why do we need to evaluate?• What are the barriers ?

Context• Austere times

Need to make every £ count View as an investment not an expense

• Media landscape & consumption patterns

Evidence of what works and what does not Optimise use of scarce resources

• The role of communications under scrutiny

How it supports achievement of policy outcomes Business planning & activity prioritisation Sharing best practice

B1

1111

Barriers – stopping it happening

Insufficient time / resource / budget

Lack of SMART or unrealistic policy

objectives / targets

Difficulty accessing the right data / tools

Culture & entrenched behaviours

Gaps in evaluation standards & capability

Lack of integrated communication

strategy linked to policy objectives

B2

12

Strategic Alignment

Start with the policy aim, then develop communications objectives that help deliver this.

Objectives should be measurable, focused on outcomes not outputs, and related to changing attitudes and/or behaviour.

Setting the right objectives is crucial

C1

13

Strategic Alignment

C2

Strategic Objectives

& KPIs

Comms Objectives,

& KPIs

Activity objective

Evaluation stages –The Big IDIAC3

IdentifyThe scope of your project

DevelopYour evaluation plan

ImplementGather data to measure

performance

Analyse & reportPerformance against plan

1

2

3

4

Task 1: Define what you need to evaluate by asking:• What activity am I evaluating?• What do I know & what factors could affect the outcome?• What is my evaluation expected to achieve?Output: Summary of your proposed evaluation approach

Task 2: Define how you’ll measure success:• Set SMART objectives & define your target audience• Map out how activity will work• Set performance metrics (KPIs) & agree baseline / targets Output: Draft evaluation plan

Task 3: Identify and gather evaluation data:• Make most of existing data• Gather additional data (research, feedback & proxies) • Review data gaps (more budget ?) manage expectations Output: Completed evaluation plan

Task 4: Assess the success of your activity:• Analyse effectiveness & provide insights for future• Demonstrate efficiency and value for money • Demonstrate role of communications in supporting

policy objective delivery (outcome)Output: Final evaluation report

Key performance indicator basket C4

Activity Effectiveness Result

1. Strategic alignment – Ensure activity objectives are SMART, aligned and supports policy delivery.

2. Business impact – Always measure true business impact (outcome) rather than outputs and out-takes

3. Big IDIA – Try to adopt the suggested frameworks and follow the big IDIA stages.

4. Continuous improvement – Ensure results drive appropriate actions and any learnings inform future activities.

5. Best practice – Be objective, share results and make evaluation an integral part of your communications approach.

Top TipsC5

GCS Mandatory Evaluation Project

Progress update & Next steps

D

D1: Performance Frameworks & Dashboards

1. Performance frameworks /dashboards (Phase one: 37 departments /ALBs) 33 depts. /ALBs (89%) with complete or near complete frameworks. 27 depts./ALBs (73%) expected to have fully functional dashboards by April 14.

Headline measures Audience Team activity / Performance indicators Resource Risks / Concerns

Stakeholders

Partners

Staff

Claimants

Staff

People • FTE (actual vs. plan)• FTE (team and profiles)• Flexible resource stats

Financials • Allocations (Staff vs. Non-staff costs)• Spend vs. Forecast• ERF submissions / spend (teams)

1 2 3 4 5

Directorate Targets

• Sickness levels: AWDL (teams)• Directorate engagement score• HSE incident:• Compliance targets (e.g.) mandatory training performance discussions held Skills manager

Risk Impact Action Lead

1 H    

2 M    

3 L    

Risk register

No Raised by date Status

1    

2    

3    

3 C’s / Hot Topic register

Press office ‘Forward Look’ – up coming events

12 months rolling activity communications plan

Prioritisation activity list

Planning Section

New

Existing

6

SRP1: Welfare Reform

Business Objective: Introduce the Universal Credit and other reforms to simplify the welfare system; Ensure that the system always incentivises work and work always pays; Help to make the welfare system affordable longer term. Comms Objective / Outcome:

WSR

Team / Area Input / output / outcome indicators)

Universal Credit

• Milestones – performance up date• Nos. engaged / events held• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage

SRP2: Labour Market (GBW)

Business Objective: Implement and manage the Work Programme; Deliver personalised package of support to get people into work. Comms Objective / Outcome:

Stakeholders

Employers

Individuals

Staff GBW

Team / Area Input / output /Outcome (indicators)

Labour Market

• Milestones – performance up date• Nos. events held / attendance levels• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage

SRP3: Tackling Poverty

Business Objective: Develop a welfare system that recognises work as the primary route out of poverty; Implement a the Social Justice and Child Poverty Strategies that focuses on eradicating child poverty by 2020. Comms Objective / Outcome:

POVERTY

Team / Area Input / output metrics (indicators)

Social Justice

• Milestones – performance up date• Nos. events held / attendance levels• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage

Stakeholders

OGDs

General public

Staff

SRP4: Pensions Reform (PR)

Business Objective: Provide decent State Pensions; Encourage employers to provide high quality pensions and make automatic enrolment and higher pension saving a reality; Phase out the default retirement age

Comms Objective / Outcome: PR

Team / Area Input / output metrics (indicators)

Pensions

• Milestones – performance up date• Camp tracker data (Ind / employers)• No. of web visits / downloads. • No. of comms issued• % Net positive media coverage• Auto enrolment data

Stakeholders

OGDs

General public

SRP5: Disability Equality

Objective: Develop and implement a disability strategy to enable disable people fulfil their potential; Support more independent living for disable people through an effective strategy and evidence-based policies for disability benefits; Support disabled people take up employment opportunities through specialist programmes. Comms Objective / Outcome:

DISABILTY

Team / Area Input / output metrics (indicators)

A2 work

DLA

PIP

ESA

• Milestones – performance update• No. of events (dialogues) carried out• No. of partnerships established• No. of web visits / downloads. • No. of comms issued• % Net positive media coverage• % take up rates (PIP)

Stakeholders

OGDs

General public

Disabled people

Claimants

Staff

SRP6: Improving our services

Objectives

• Effective communications delivery

• Staff engagement improvements

• Public and Corporate info delivery

• Secure and effective digital comms

• Resource management / R&E

Outcomes:

People Survey – Headline scores

People survey scores for: B31,41,47,50-54, B06-08.

SERV I CES

Teams Input / output metrics (indicators)

Internal Comms

• Milestone – performance update• No of staff comms issued• No. of proactive briefings (SCS & Line mgrs)• No. of bright ideas (submitted vs. Implemented)• Intranet staff visits to: Home page / DWP Story pages Have your say / forum discussions• Wavelength scores (QB1, QB5,QC1,QE1, QE3)

Corp Info

• No. of leaflets / forms delivered or reduced• No. of evaluation reports requested / delivered• No. of jobs successfully delivered• L&D core training delivered / attendance levels• People survey scores for: B22, B25-26

R&E

Business MI*

Existing

D2: Mandatory Evaluation Training

2. Evaluation training - Free half-day courses: 34 training sessions delivered. 541GCS staff trained to date. 4,402 unique page views to our online evaluation guide.

Evaluation Training (400 free places)*

• Introduction to Evaluation for Marketing

• Introduction to Evaluation for Press & Media

• Guest speaker Events

* NB: The nature of the programme this year 2014/15 under review (TBC)

D3: Evaluation Standards & Capability

• New evaluation standards – Three levels

• Capability audit to start soon - April / May 2014

• Tailored package of support & training for each level

Advance

Practitioner

Foundation

D4: Evaluation ‘ centre of excellence’

• Established an Evaluation Council - Industry expert who meet regularly and act as a sounding board for new evaluation thinking as well as to review and validate government evaluation plans and outputs.

• Creating a repository for:

Case studiesEvaluation GuidePerformance Framework Top tipsStandard Metrics

What Next

• GCS Live - 1st April 2014

• Govt. Comms Plan launched – 12th May

• Capability Audit – April /May 2014

• Developing a tier package of support for each level

• Phase 2 roll-out across all departments / ALBs (330)

• Regular updates to GCS Programme board and Council Invite departments / ALB to share their experience & approach

• Lots more events planned – Register on GCS now!

24

Thank you!

Contact: Paul NjokuEmail : paul.njoku@cabinet-office.gsi.gov.ukWeb link to guide: https://gcn.civilservice.gov.uk/guidance/evaluation/

Appendix

ProductsCustomer Service

InvestmentsEmployment

BrandingPublic Relations

MarketingSocialResponsibility

MEDIA (Traditional, Social)

Topic Experts, Leaders,Friends/Family

Perceptions & expectations

Supportive Behaviour

Results

What your departmentSays/Does

Direct Experience

What Others Say

How reputation is created

The attributes and dimensions have different meanings and importance for different stakeholders. Beneath the 7 dimensions, 32 attributes underpin the individual dimension themes. Different stakeholder groups typically have unique attributes that are found more important than others (reputation drivers).

Reputation Attributes

Reputation Dimensions

The seven dimensions specify at a more operational level, which aspects are most important for stakeholders’ perceptions and expectations – i.e. what’s driving a company’s reputation

Reputation StrengthA measure of the emotional connection.

Reputation has a positive/negative impact on support. An increase in reputation = an increase in support. Support (such as buying products and services, saying something positive, giving the benefit of doubt in times of crisis (etc.) leads to increased business results

Supportive Behaviour

Reputation drivers & dimensions

Measuring outputs

28

Content metricsFavourability

MessagesThemes / Issues

CampaignsArticle type

Proactive / reactiveProminence

Calls to action

Delivery metricsVolume

Opportunities to seeReach and Frequency

Media typeKey publications

JournalistsSpokespeople

Regional breakdownImpact measures

Output Metrics

Are we saying the right thing... ....to the right audience?

Activity

Intermediary Effect

Audience Effect

‘Earned’ mediaModel

Amplification from social media

Government analysis audit

29

Are we saying the right thing...

....to the right audience

Metric% of projects

Favourability 100%Message delivery 91%Campaign tracking 48%Theme / issues tracking 57%Spokespeople / commentators 78%Prominence 57%Article type 43%Proactive / reactive 22%Calls to action 39%Volume (number of articles) 100%Opportunities to see (OTS) 74%Reach and frequency 78%Impact measures 26%Media type 83%Regional split 57%Publications 87%Journalists 87%Advertising value equivalent AVE) 57%Cost per thousand (CPM) 22%Linking ouput to outcomes 22%

Content Metrics

Delivery metrics

Efficiency / effectiveness

metrics