evaluation framework - safework.nsw.gov.au · evaluation plan the mechanics of how a particular...

EVALUATION FRAMEWORKWORK HEALTH AND SAFETY ROADMAP FOR NSW 2022

JULY 2017

DisclaimerThis publication may contain information about the regulation and enforcement of work health and safety in NSW. It may include some of your obligations under some of the legislation that SafeWork NSW administers. To ensure you comply with your legal obligations you must refer to the appropriate legislation.Information on the latest laws can be checked by visiting the NSW legislation website www.legislation.nsw.gov.auThis publication does not represent a comprehensive statement of the law as it applies to particular problems or to individuals or as a substitute for legal advice. You should seek independent legal advice if you need assistance on the application of the law to your situation.This material may be displayed, printed and reproduced without amendment for personal, in-house or non-commercial use.

Catalogue No. SW08641SafeWork NSW, 92–100 Donnison Street, Gosford, NSW 2250Locked Bag 2906, Lisarow, NSW 2252 | Customer Experience 13 10 50Website www.safework.nsw.gov.au© Copyright SafeWork NSW 0817

http://www.legislation.nsw.gov.au

http://www.safework.nsw.gov.au

ContentsABBREVIATIONS AND ACRONYMS 6

KEY EVALUATION TERMS AND DEFINITIONS 7

EXECUTIVE SUMMARY 9

1. INTRODUCTION 10

1.1. Overall objectives of evaluating the Roadmap 10

1.2. Purpose of the framework 10

1.3. Overview of the Roadmap 10

1.4. Alignment with the NSW Government Program Evaluation Guidelines (2016) 11

2. EVALUATION GUIDELINES 12

2.1. Principles of evaluation 12

2.2. Information management 13

2.3. Ethical conduct of evaluation activities 14

2.4. Risk management 14

2.5. Updating the evaluation framework 14

3. FRAMING THE ROADMAP EVALUATION ACTIVITIES 15

3.1. Theory of change – how the Roadmap is expected to work 15

3.2. Logic models and outcomes matrices frame the evaluation 16

3.2.1. Outcomes matrices and performance measures for Action Area 1: Embed the health and safety landscape in NSW workplaces 20

3.2.2. Outcomes matrices and performance measures for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist 25

3.2.3. Outcomes matrices and performance measures for Action Area 3: Build exemplar regulatory services 29

3.2.4. Outcomes matrix for long term outcomes 33

3.3. Roadmap evaluation and guiding questions 34

4. PERFORMANCE MONITORING 39

4.1. About performance monitoring 39

4.2. Collection of performance metrics 40

4.2.1. Reporting performance metrics 40

4.3. Accountability 40

5. EVALUATION GOVERNANCE, EVALUATION PLANNING AND ADVICE ON METHODS 41

5.1. Governance and management of evaluation studies 41

5.2. Directorate-level annual evaluation planning 42

5.3. Individual project evaluation plans 42

5.4. Decision support tool 42

5.5. Evaluation checklist 45

5.6. Methodological approaches 46

6. UNDERSTANDING, USING AND REPORTING EVALUATION FINDINGS 48

6.1. Using evaluation results for service and product improvement 48

6.2. Communication through evaluation reports 48

6.3. Analysis of attribution and causal contribution 49

6.4. Interpreting trends in lag indicators (long term policy outcomes) 50

Tables and figuresTABLES

Table 1: Action Area 1 Inputs Outcomes Matrix 20

Table 2: Activities, reach, short term outcomes and medium term outcomes 20





Table 7: Outcomes matrix for Roadmap long term outcomes 33

Table 8: The Roadmap Evaluation Priority Levels 43

Table 9: Commonly used methods in evaluation 47

FIGURES

Figure 1: Overarching WHS Roadmap for NSW program logic model 18

Figure 2: Logic model for Action Area 1: Embed the health and safety landscape in NSW workplaces 19

Figure 3: Logic model for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist 24

Figure 4: Logic model for Action Area 3: Build exemplar regulatory services 28

Figure 5: Evaluation checklist 45

6 SAFEWORK NSW

Abbreviations and acronyms

ACT Australian Capital Territory

AWU Australian Workers’ Union

CFMEU Construction, Forestry, Mining and Energy Union

NSW New South Wales

PCBU Persons conducting a business or undertaking. The Work Health and Safety Act 2011 places the primary duty of care and various other duties and obligations on a 'person conducting a business or undertaking'; the meanings are set out in Section 5 of the Act.

The Roadmap Work Health and Safety Roadmap for NSW 2022

SafeWork SafeWork New South Wales

SW NSW SafeWork New South Wales (used as abbreviation in tables)

WHS Work health and safety

WHS ROADMAP EVALUATION FRAMEWORK 7

Key evaluation terms and definitions

Evaluation A rigorous, systematic and objective process to make an evidence-based assessment of the effectiveness, efficiency, appropriateness and sustainability of an initiative.

Evaluation design How a particular evaluation is set up to provide the evidence to achieve its purpose (eg experimental design). The design takes account of the appropriateness of methods for the project’s scope and status, as well as ethical safeguards.

Evaluation framework A structure to guide evaluation and monitoring activities, based on systematic and conceptual thinking about what a program is setting out to achieve and how the anticipated outcomes can be assessed.

Evaluation plan The mechanics of how a particular evaluation will be carried out. It sets out the tasks and responsibilities to implement an evaluation project/study and the budget.

Evaluation strategy An overall approach to evaluation that reflects the strategic priorities and state of the project at the time. It starts with the purpose (why), focus (what), and stakeholder information needs (what evidence, by when). It references the project outcomes framework and links with other policy frameworks/strategies. It includes the main evaluation activities/studies, with an outline of designs and methods. It also sets out the timing, responsibilities and resources.

Inputs The resources (human and other) you use to do the work.

Outcome A result that can be measured or observed and a changed condition of people/organisations/systems.

Reach Used in program logic to show which group/s are being targeted by particular activities

Short term outcomes Changes expected to occur as a direct result of your work.

Theory of change These are the big research-based theories about how change occurs for individuals, groups, organisations and communities. Theories of change can provide a conceptual framework for developing or critiquing a program theory.

Medium term outcomes Changes your program is expected to contribute to, the next link in the causal chain of outcomes after short term outcomes. Generally, they relate to the strategic objectives of a program.

Long term outcomes Changes your program is expected to contribute to, the next link in the causal chain of outcomes after medium term outcomes.

Outputs The tangible products or services the activity will deliver.

Outcomes matrix A matrix of project outcomes at different levels, causally linked by a results logic diagram. For each outcome, it shows attributes of success and is usually used to frame an evaluation strategy and/or a monitoring system.

8 SAFEWORK NSW

Performance monitoring Provides information about whether or not an initiative is on track. That is, information about how much has been done to what quality and sometimes about the immediate outcomes achieved. Performance monitoring is one source of evidence for an evaluation study but monitoring by itself is not evaluation and is less in-depth than evaluation. Monitoring is ongoing while evaluation is typically periodic; mainly uses data captured as part of delivering services; informs quality assurance activities and is used to manage programs/businesses on an ongoing basis.

Program theory An explicit statement of how an intervention/program is understood to contribute to observed or intended outcomes.

Program logic model A diagram that shows how an intervention/program is assumed to work by causally linking program activities with intended outcomes.

Purposive sampling A sample of subjects selected deliberately (on purpose) because these subjects have particular characteristics of interest.


Executive summaryThis is the evaluation framework for the Work Health and Safety Roadmap for NSW 2022 (the Roadmap). The evaluation framework is intended to provide a blueprint for ongoing evaluation of the three action areas over the life of the Roadmap and it will be reviewed and updated in line with the two-yearly review of the Roadmap itself.

The development of the evaluation framework has been a collaborative process. Six key informants representing unions and business peak groups provided input into the framework through interviews. SafeWork Directors and members of the Business Performance team contributed via interviews and workshops. This consultative process, complemented by a literature review, Research into features of effective evaluation frameworks, was delivered in February 2017.

The framework has an overarching program logic and program logics for each of the three action areas under the Roadmap. The action area program logics have been used to specify outcomes matrices that represent the intended outcomes of the action areas at different levels, the attributes of success and potential data sources.

The framework identifies key evaluation and guiding questions (for each action area) and includes a decision support tool to identify priorities for evaluation. The framework also covers evaluation governance, advice on planning evaluations, and advice about reporting and using findings to inform and influence service improvement. In addition, evaluation guidelines outline principles for evaluation and identify ethical considerations.

10 SAFEWORK NSW

1. Introduction1.1. Overall objectives of evaluating the RoadmapEvaluation can be defined as a rigorous, systematic and objective process to assess a program’s effectiveness, efficiency, appropriateness and sustainability. An effective evaluation produces information (within ethical standards) that key stakeholders find credible, relevant and useful. Evaluation studies need to focus on the aspects of the program that stakeholders find most relevant at a particular time, and to function within the limits of resources and timelines.

The objectives of evaluating the Roadmap are to:

1. Measure whether there is a change in line with the outcomes identified in the program logics for each action area.

2. Assess whether the development, design and implementation of activities under the Roadmap are effective, and which activities need improvement.

3. Assess the extent to which activities under the Roadmap contribute to the achievement of outcomes shared with industry, government and community partners.

1.2. Purpose of the frameworkThis customised evaluation framework is intended to guide evaluation and monitoring for the Work Health and Safety Roadmap for NSW 2022 (the Roadmap) over the next six years. The evaluation framework will provide SafeWork NSW guidance about:

• when and how the Roadmap will be evaluated

• the development of sound evaluation plans and the implementation of monitoring and evaluation activities that ensure SafeWork can determine whether key objectives of the Roadmap have been met and to what extent

• the development of a risk-based operating model where evaluation effort is prioritised to the most critical area.

Formal project evaluation is used to provide a systematic assessment of a program’s efficiency and effectiveness in delivering intended outcomes. The key point is that evaluation is also about learning and program improvement. A high quality evaluation can provide information about the success of a program of work, the factors that have contributed to success and failure, and the challenges that have had to be overcome along the way. A solid evaluation informs ongoing improvement.

The evaluation framework was developed in consultation with SafeWork Directors and representatives of peak industry bodies and associations.

1.3. Overview of the RoadmapThe Roadmap is a project under the NSW Government’s Better Regulation Division (BRD) Strategic Plan 2016, and is the responsibility of SafeWork NSW. It is a six-year strategy that will drive state-wide activities for improvement in work health and safety; the BRD measure of success is to reduce the incidence rate of fatalities and serious injuries in NSW workplaces.

The Roadmap is an agreement aimed not only at the regulator, but at peak industry and employee bodies/associations, community leaders, and each employer and worker, all of whom have played a major part in contributing to the Roadmap through extensive consultation.

http://www.safework.nsw.gov.au/roadmap


The Roadmap’s vision is for healthy, safe and productive working lives, and it sets out targets to measure success. Its purpose is to ‘drive state-wide activities for improvement in work health and safety in NSW’. The Roadmap identifies three action areas:

1. Embed the health and safety landscape in NSW workplaces

2. Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist

3. Build exemplar regulatory services.

An underlying enabler is community commitment, which is seen as important to the achievement of the three action areas. Community means everyone – workers, employers, professional and community groups, and regulators. A key factor for the Roadmap is that SafeWork is a contributor to many workplace health and safety actions which they do not lead.

The Roadmap spans 2016–2022 but it will be refreshed and released every two years to ensure it stays relevant. The Roadmap combines existing workplace health and safety work with new actions. The first year of the Roadmap has been dedicated to planning the delivery of, and operationalising, the action areas.

1.4. Alignment with the NSW Government Program Evaluation Guidelines (2016)Across all agencies, the NSW Government is committed to using evaluation to inform policy, and achieve greater efficiency and transparency in government, through the use of digital services and data analytics.

The NSW Government Program Evaluation Guidelines (2016) outline the government’s approach to program and agency evaluations.

12 SAFEWORK NSW

2. Evaluation guidelinesThis section provides information about how best to conduct, commission and manage evaluation activities.

Using independent evaluators and research groups to carry out the different evaluation phases may be advantageous because their independence has the potential to enhance the credibility of the findings to program stakeholders at a strategic policy level. Evaluation activities might be able to be conducted internally depending on available internal resources and expertise.

2.1. Principles of evaluation1

The following evaluation principles have informed the design of the evaluation framework and aim to ensure that all evaluation activities of the Roadmap are of a high quality and will provide useful evidence. The principles are drawn from the NSW Government Program Evaluation Guidelines (2016), with additional SafeWork NSW principles to guide individual evaluation planning.

NSW Government Program Evaluation Guidelines Principles1. Build evaluation into your program design. Plan your evaluation before you implement a project

to ensure the project has clearly defined and measurable outcomes and sources of information have been identified and data collection systems are in place. This in turn increases the quality of the evaluation.

2. Base your evaluation on sound methodology. Use best practice methodologies to suit the program’s size, significance and risk.

3. Include resources and time to evaluate. Consider the required evaluation resources and timeframe when planning the project. Ensure evaluation findings will be available when needed to support decision-making.

4. Use the right mix of expertise and independence. Choose evaluators who are experienced and independent from program managers, but always include program managers in evaluation planning. An external evaluation can bring expertise, capacity and credibility based on greater independence. An internal evaluation can bring first-hand knowledge about a policy or program, and develop capacity for future evaluation work. A hybrid model – such as an evaluation conducted internally with an external evaluation partner or mentor – can be a cost-effective way of strengthening an internal evaluation.

5. Ensure proper governance and oversight. Use governance processes to ensure oversight of evaluation design, implementation and reporting.

6. Be ethical in design and conduct. Carefully consider the ethical implications of any evaluation activity, particularly collecting and using personal data, and any potential impacts on vulnerable groups. You may need formal review and approval from an ethics committee certified by the National Health and Medical Research Council.

7. Be informed and guided by relevant stakeholders. Listen to stakeholders, including program participants, government or non-government staff involved in managing and delivering the program, and senior decision makers.

8. Consider and use evaluation data meaningfully. Include clear statements of findings for consideration in evaluation reports. Use reports to inform any decisions about changes to programs.

9. Be transparent and open to scrutiny. Publicly release key information about all aspects of the evaluation unless there is an overriding public interest against disclosure. This could include methodologies, assumptions, analyses and findings.

1 NSW Government Program Evaluation Guidelines (2016).

http://www.dpc.nsw.gov.au/programs_and_services/policy_makers_toolkit/evaluation_in_the_nsw_government


10. Be clear about what success could look like. Successful evaluation is reliant on clear problem definition. You may either develop a program logic for your specific intervention, referencing the action area program logics, and/or identify attributes of success for your specific activity that fit under one or more specific outcomes in the three action area logics.

11. Balance the technical, strategic and political dimensions in planning evaluations. Evaluation is often seen as a technical exercise. But evaluation is also a strategic exercise that aims to deliver information to key stakeholders when they need to make decisions. For SafeWork, the (small p) political dimension is working with your industry partners who may have different interests in the program and the evaluation, including different ideas about evidence, methods, and how the evaluation is managed and reported.

12. Make use of lead indicator and other administrative data already collected through administrative data systems.

2.2. Information managementAll data, information and documentation provided or obtained during the course of any evaluation activities should be kept in strict confidence and in accordance with the Australian Privacy Principles effective 12 March 2014, which cover the collection and management of personal information.

• Information should be sought from informants lawfully, with their consent as appropriate, and used only for the purposes of the evaluation.

• Informants should be advised of the intended primary and secondary use for the collected information.

• Information and documents must be kept in secure places and not divulged to any party other than those involved in contracting or conducting the evaluation. All evaluation data should be kept on a secure, password-protected server and be protected by protocols to ensure the security of any data transmitted through the server to other locations. For example, auditable, granular data access for evaluation folders (limited to relevant persons), firewall protection and intrusion detection system, nightly backup to an offsite location and strong physical security (double locks and limited physical access to servers). All mobile devices used to access and transmit project data should be controlled centrally so these can be wiped remotely if necessary.

• Information must be kept only for the purposes of the evaluation, record-keeping, or research validation, and should be de-identified within the terms of the Australian Privacy Principles’ application to evaluation. Evaluation information/data should be kept for a minimum of ten years before being destroyed.

External contractors for evaluation activities should be asked to describe the following.

• How will data be stored?

• The format of stored data.

• Arrangements for security of stored data, including when and by whom that shared data can be accessed.

• The duration the data will be stored.

• The method of destruction of data.

14 SAFEWORK NSW

2.3. Ethical conduct of evaluation activitiesIt is important that SafeWork staff members including those commissioning evaluation studies and any evaluation contractors consider the ethics of collecting and storing evaluation data and be guided about ways to solve ethical dilemmas that may arise during the evaluation. In your evaluation planning you should consider how consent will be obtained.

The implementation of and commissioning of, and all SafeWork evaluation activities should be guided by the Australasian Evaluation Society (AES) Guidelines for the Ethical Conduct of Evaluations, the NHMRC National Statement on Ethical Conduct in Human Research and the NHMRC Ethical Considerations in Quality Assurance and Evaluation Activities (2014).

Triggers for ethical review

The NHMRC Ethical Considerations in Quality Assurance and Evaluation Activities (2014) provides advice about triggers for ethical review, and relevant triggers are:

• where the activity potentially infringes the privacy or professional reputation of participants, providers or organisations

• secondary use of data – using data or analysis from quality assurance or evaluation activities for another purpose

• comparison of cohorts

• targeted analysis of data involving minority or vulnerable groups, whose data is to be separated out of that data collected or analysed, as part of the main quality assurance or evaluation activity.

2.4. Risk managementEvaluation studies of Roadmap initiatives will need to deal with a number of issues and risks for implementation. These fall into two main areas.

1. Availability, reliability and usefulness of expected administrative data.

2. Gaining the desired level of participation in data collection activities across different sectors, types of businesses and stakeholder groups.

The extent that these issues will influence the capacity of the evaluation studies to adequately answer the evaluation questions depends on their resolution and on strategies to engage and communicate with stakeholders.

The evaluation plan for each phase should include a risk management strategy, categorisation of risks using a risk matrix, and risks and solutions should be reported against regularly in progress reports.

2.5. Updating the evaluation frameworkThe evaluation framework is based on consultations about how the Roadmap is expected to work (theory of change), be used, and produce outcomes. The framework may need to be refined when:

• two-year evaluation cycles show that the program works in ways not explained by the current theories of change

• the program processes and implementation processes are changed to increase the effectiveness of activities under the Roadmap

• the evaluation framework is not producing useful information.


3. Framing the Roadmap evaluation activities

This chapter describes an overarching program logic model for the Roadmap and logic models for each of the action areas. The action area logic models describe how these can be used to frame evaluations of the Roadmap over the six years, drawing upon the underlying theories of change for the work. The logic models were used to develop the key evaluation questions about the Roadmap.

To guide evaluation, outcomes matrices have been developed for each action area, which set out the measures and potential data sources for each level of the logic models.

3.1. Theory of change – how the Roadmap is expected to workTheories of change are useful to explain how and why a change is expected to happen in a particular context. Because the Roadmap is a multi-faceted intervention combining educative, environmental and regulatory approaches that have strategic, tactical and operational foci, it is useful to identify one or more broad theories of change that are built upon how similar interventions have worked in other circumstances. The outcomes logics reference some elements of each of these theories of change.2 In practice most of our programs have one or more elements of each of these theories. A typical approach would be to gain an understanding of the behaviours of a target group and then seek to change these behaviours using visits, doing compliance checks and providing guidance materials (theory of behaviour and theory of reasoned action). This approach is often supported by social media campaigns (diffusion theory). In short, most of our programs are based on the idea that if we make it easy for people to comply then they are more likely to do so.

1. Theory of behaviour influenced by social, cognitive and emotional influences. The NSW Department of Premier and Cabinet has invested in the development of a newer understanding of the way people behave, in an attempt to gain better compliance with a wide range of behaviours. Their work has ranged from changes in no-show rates at public hospitals and increasing quitting rates amongst smokers to increased compliance with motor vehicle laws. The principle behind the Behavioural Insights Unit’s work is that behavioural change is not exclusively influenced by rational choices; the Behavioural Insights Unit draws on research into behavioural economics and psychology to influence choices in decision-making. By focusing on the social, cognitive and emotional behaviour of individuals and institutions it suggests that subtle changes to the way decisions are framed and conveyed can have big impacts on behaviour.3

2. Diffusion theory holds that change occurs when an innovation (new practice/idea/approach) is diffused and adopted leading to certain consequences. Diffusion is a process in which an innovation is communicated and exchanged through social channels such as media campaigns and networks. Diffusion theory has four key concepts and related theories:

• Theory of perceived attributes, which states than an individual will adopt an innovation if they perceive it has a relative advantage over an existing approach; it is compatible with existing values and practices; it is not too complex; it can be tested for a limited time without committing to adoption and with observable results.

• Communication channels, that is, propositions about how messages are shared effectively. Mass media is useful for contacting large numbers of people and increasing awareness. Interpersonal contact is more effective for persuading people to accept a new idea. Effective communication can depend on engagement of peers and opinion leaders.

• Diffusion among members of a social system. Diffusion is affected by social structure, system norms, opinion leaders and change agents.

• A time dimension to diffusion. Innovation decision process theory postulates that the adoption process has five steps: 1) Knowledge 2) Persuasion 3) Decision 4) Implementation, and 5) Confirmation. Further, those individuals differ in when they will adopt an innovation in predictable ways.

2 Funnell SC and Rogers PJ (2011). Purposeful Program Theory. Jossey-Bass. 1st Edition.

3 http://bi.dpc.nsw.gov.au/

http://bi.dpc.nsw.gov.au/

16 SAFEWORK NSW

3. Theory of reasoned action and theory of planned behaviour. These theories are based on the notion that humans are rational and have control over what they do. Intentions predict behaviours, and intentions are influenced by a person’s beliefs about the likely consequences of the behaviour, their attitudes towards the behaviour and consequences and perceptions of norms, that is, other people’s opinions of the behaviour.

3.2. Logic models and outcomes matrices frame the evaluationThe Roadmap is intended to drive state-wide activities for improvement in work health and safety – so that NSW residents have healthy, safe and productive working lives. The Roadmap comprises three action areas to help achieve SafeWork’s targets.

The overarching program logic (Figure 1) shows how the three action areas are expected to contribute to medium and long term outcomes and provides a line of sight for the Roadmap implementation. Three pipeline logic models, one for each action area, are also shown with more detail about the activities and the causal links between activities and specific outcomes (Figures 2, 3 and 4). The action area logic models provide the structure to prescribe the scope of evaluation activities and focus the data collection methods.

Under Action Area 1, the problem being addressed by SafeWork is that businesses need assistance to embed the health and safety landscape. Under Action Area 2, the problem being addressed is that SafeWork needs to work with industry to identify and address high risk sectors and harms, protect at risk workers and ensure that high risk workplaces meet compliance standards. Under Action Area 3, the problem being addressed is that SafeWork needs to provide right touch regulatory approach so that customers can be confident that SafeWork will enforce WHS to protect workers, that SafeWork will use data and insights to make decisions and provide innovative services to meet the needs of business.

What is a program logic model?

A program logic model is an analysis of the aims, objectives and activities of an initiative (in this case the Roadmap) that represents ideal ‘outcomes’ at different levels and stages, and the causal links between them. It sets out the chain of expected outcomes in a diagram, starting with the inputs over which you have a high degree of control and progressing to the hoped for long term outcomes. It shows short term outcomes and medium term outcomes that logically, if these are achieved, should then lead to the longer term outcomes. A program logic diagram is not a process flow diagram or a systems map even though it can resemble one.

How to read the program logic model4

Certain resources are needed to deliver your projects and services

If you have access to them, then you can use them to accomplish your planned activities

If you accomplish your planned activities, then you will hopefully deliver the services/products (amount, quality) that you intended and reach your intended audience

If you accomplish your planned activities to the extent you intended, then your audience will benefit in certain ways

If these benefits to participants are achieved, then certain changes in workplaces/systems and impacts on individuals and the community might be expected to occur

Inputs

Your planned work Your intended results

Activities Reach Outcomes Long termoutcomes

1 2 3 4 5

4 W.K. Kellog Foundation Logic Model Development Guide


Assumptions

The Roadmap overarching program logic is underpinned by a number of assumptions about how the Roadmap will work.

• The communications campaign will result in increased engagement with SafeWork NSW.

• Interacting with SafeWork NSW will help businesses become more confident about, and capable of, managing WHS in NSW.

• SafeWork can work collaboratively with stakeholders in order to develop and deliver the right products and services.

• SafeWork NSW will have sufficient and consistent resources to deliver on the Roadmap.

• Engagement with industry partners (peak bodies, associations, employers and employees) will result in them taking actions in line with the Roadmap, and their actions will contribute to achieving the long term safety outcomes.

• SafeWork NSW will put systems and processes in place to deliver exemplar regulatory services.

• A risk-based regulatory approach will improve compliance and workplace practices in targeted industries and workplaces. Inspections improve compliance.

• A right touch regulatory approach leads to awareness and action.

The achievement of outcomes under the Roadmap will be influenced by factors external to SafeWork that are outside of the control of the agency. According to SafeWork stakeholders, external factors include the following.

• Change in government/change in priorities for the government

• Changes in legislation/regulatory framework

• Economic environment and activity. Industry/business activity trends with industries expanding or contracting. For example, work health and safety can be compromised when businesses are rapidly expanding.

• Changes in technology and other business operating conditions

• Changes in workforce structure and employment conditions. Two trends were raised: 1) the increase in casual or temporary employment 2) more people working as contractors instead of employees, for example in the road transport context.

• Changes in the nature of work (such as more automation of dangerous work), the mix of work (such as a shift from manufacturing to less dangerous service provision as the mainstay of the economy).

Part of the function of evaluation studies will be to test the assumptions and also take account of the influence of external factors when analysing and interpreting results.

Outcomes matrices show what information can be used to assess the achievement of outcomes

Outcomes matrices set out the full range of information needed to assess the achievement of outcomes and so answer the evaluation and guiding questions, and indicate where the information can be sourced from.

Outcomes matrices include:

• attributes of success – what outcomes should look like when achieved

• performance evaluation measures – metrics for attributes of success.

18 SAFEWORK NSW

Fig

ure

1: O

vera

rch

ing

WH

S R

oad

map

fo

r N

SW

pro

gra

m lo

gic

mo

del

Bus

ines

s p

rob

lem

Act

ivit

ies

Rea

chSh

ort

-ter

m o

utco

mes

Med

ium

-ter

m o

utco

mes

Long

-ter

m o

utco

mes

Targ

eted

sta

keho

lder

sar

e aw

are

of

WH

S is

sues

ac

coun

tab

iliti

esIn

crea

sed

will

ing

ness

to

take

ac

tion

Targ

eted

sta

keho

lder

s id

enti

fy

way

s to

imp

rove

WH

S in

the

ir

ind

ustr

ies,

wo

rkp

lace

sW

ork

pla

ces

und

erst

and

the

d

iffer

ent

elem

ents

of

H&

S

land

scap

e, r

esp

onsi

bili

ties

Targ

eted

bus

ines

ses

are

awar

e of

R

TW is

sues

/acc

oun

tab

iliti

es a

nd

iden

tify

way

s to

imp

rove

RT

W

out

com

es in

the

ir w

ork

pla

ceIn

crea

sed

aw

aren

ess

of

the

Safe

Wor

k b

rand

Incr

ease

d e

ngag

emen

t w

ith

Saf

eWo

rk

Bus

ines

ses

need

ass

ista

nce

to e

mb

ed t

he h

ealth

and

sa

fety

land

scap

e

Eff

ecti

ve d

eliv

ery

of

Act

ion

Are

a 1.

Em

bed

the

hea

lth

and

sa

fety

land

scap

e in

NSW

wor

kpla

ces

Pea

k in

dus

try

bo

die

s,

asso

ciat

ions

, co

mm

unit

y le

ader

s, N

SW g

ener

al

com

mun

ity,

em

plo

yers

and

w

ork

ers

NSW

wo

rkp

lace

s ha

ve

hig

h sa

fety

sta

ndar

ds

A w

ell i

nfo

rmed

NSW

co

mm

unit

y,

know

led

gea

ble

ab

out

W

HS

issu

esA

ll jo

bs

are

des

igne

d

wit

h sa

fe w

ork

p

ract

ices

and

co

ntro

lsE

ffec

tive

RT

W

pra

ctic

es a

re

emb

edd

ed in

NSW

w

ork

pla

ces

Few

er fa

talit

ies,

se

rio

us in

juri

es,

illne

ss a

nd

mus

culo

skel

etal

d

iso

rder

sat

wo

rk

Gen

eral

co

mm

unit

y, e

mp

loye

rs a

nd

wo

rker

s ac

cess

pro

duc

ts a

nd

serv

ices

Incr

ease

d c

onfi

den

ce, c

apab

ility

and

w

illin

gne

ss t

o t

ake

acti

on

Em

plo

yers

pro

acti

vely

imp

lem

ent

elem

ents

of

H&

S la

ndsc

ape

and

im

pro

ve t

heir

sys

tem

s an

d c

ont

rols

Wo

rker

s im

pro

ve t

heir

wo

rk p

ract

ices

Targ

eted

sta

keho

lder

s ad

voca

te fo

r im

pro

ved

WH

S w

ithi

n th

eir

own

sect

ors

Ind

ustr

ies

pro

acti

vely

imp

lem

ent

sect

or

wid

e W

HS

init

iati

ves,

red

uce

wor

kers

’ exp

osur

e to

ris

kTa

rget

ed e

mp

loye

rs a

nd w

ork

ers

acce

ss s

ervi

ces

Targ

eted

bus

ines

ses

imp

lem

ent

RT

W

pra

ctic

es in

the

ir w

ork

pla

ces

Imp

rove

d a

war

enes

s o

f hi

gh

imp

act

harm

s an

d W

HS

issu

es o

f ta

rget

ed in

dus

trie

s an

d w

ork

ers

Incr

ease

d c

onfi

den

ce, c

apab

ility

an

d w

illin

gne

ss t

o t

akes

act

ion

PC

BU

s aw

are

of

Saf

eWo

rk’s

re

gul

ato

ry, a

dvi

sory

and

en

forc

emen

t ap

pro

ach;

pro

duc

ts

and

ser

vice

s; la

test

dev

elo

pm

ent/

s ad

vanc

es in

WH

SP

CB

Us

und

erst

and

the

ir W

HS

re

spo

nsib

ilite

s an

d w

hat

com

plia

nce

loo

ks li

keP

CB

Us

and

key

sta

keho

lder

s re

cog

nise

the

ben

efits

of

wo

rkin

g

wit

h S

afeW

ork

to

dev

elo

p

inno

vati

ve s

olu

tio

nsS

afeW

ork

use

dat

a sy

stem

s an

d

evid

ence

to

info

rm

dec

isio

n-m

akin

gS

afeW

ork

pub

lish

info

rmat

ion/

stan

dar

ds/

po

licie

s In

tera

ctio

ns w

ith

cust

om

ers

mee

t ex

pec

ted

sta

ndar

ds

of

serv

ice

Saf

eWo

rk n

eed

s to

wo

rk

wit

h in

dus

try

to id

enti

fy

and

ad

dre

ss h

igh

risk

se

cto

rs a

nd h

arm

s,

pro

tect

at

risk

wo

rkp

lace

s m

eet

com

plia

nce

stan

dar

ds

Eff

ecti

ve d

eliv

ery

of

Act

ion

Are

a 2.

Pri

orit

ise

sect

ors,

har

ms,

w

orke

rs a

nd w

orkp

lace

s w

here

mos

t si

gni

fica

nt

WH

S ri

sks

exis

t

Pea

k in

dus

try

bo

die

s,

asso

ciat

ions

, co

mm

unit

y le

ader

s, m

anag

emen

t an

d

staf

f o

f ta

rget

ed N

SW

Gov

ernm

ent

agen

cies

, m

anag

emen

t an

d s

taff

of

sele

cted

hig

h ri

sk

wo

rkp

lace

s

Hig

h ri

sk in

dus

trie

s ar

e si

gni

fican

tly

safe

r an

d h

ealt

hier

Hig

h im

pac

t ha

rms

are

elim

inat

ed o

r si

gni

fican

tly

red

uced

A s

afe

and

hea

lthy

N

SW p

ublic

sec

tor

Saf

e an

d h

ealt

hy

wo

rker

sS

afe

and

hea

lthy

w

ork

pla

ces

Ind

ustr

ies

and

bus

ines

ses

and

NSW

G

over

nmen

t ag

enci

es im

ple

men

t ag

reed

WH

S in

itia

tive

s an

d a

dop

t hi

gh-

leve

l co

ntro

ls

Saf

eWo

rk n

eed

s to

p

rovi

de

rig

ht t

ouc

h re

gul

ato

ry a

pp

roac

h so

th

at c

usto

mer

s ca

n b

e co

nfid

ent

that

we

will

p

rote

ct w

ork

ers,

tha

t w

e w

ill u

se d

ata

and

insi

ght

s to

mak

e d

ecis

ions

and

p

rovi

de

inno

vati

ve

serv

ices

to

mee

t th

e ne

eds

of

bus

ines

s

Eff

ecti

ve d

eliv

ery

of

Act

ion

Are

a 3.

Bui

ld e

xem

pla

rre

gul

ator

y se

rvic

es

NSW

wo

rkp

lace

s an

d

asso

ciat

ed s

take

hold

ers,

N

SW c

om

mun

ity,

pea

k,

bo

die

s, a

sso

ciat

ions

, co

mm

unit

y le

ader

s, P

CB

Us

and

wo

rker

s S

afeW

ork

sta

ff

Saf

eWo

rk is

re

cog

nise

d a

s o

pen

an

d t

rans

par

ent;

d

rive

n by

evi

den

ce;

fost

erin

g h

ealt

h an

d

safe

ty in

nova

tio

nS

afeW

ork

is s

een

as a

tr

uste

d a

nd in

fluen

tial

re

gul

ato

r N

SW is

re

cog

nise

d a

s th

e b

est

pla

ce t

o s

tart

and

ru

n b

usin

ess

PC

BU

s ac

cess

Saf

eWo

rk’s

pro

duc

ts

and

ser

vice

s, s

atis

fied

wit

h th

ese

PC

BU

s re

view

and

imp

rove

the

ir

wo

rk p

ract

ices

, sys

tem

s an

d c

ont

rols

in

line

wit

h S

afeW

ork

’s r

egul

ato

ry,

advi

sory

and

enf

orc

men

t ap

pro

ach

and

late

st e

vid

ence

/po

licy/

stan

dar

ds

PC

BU

s/ke

y st

akeh

old

ers/

par

tner

s w

ork

colla

bor

ativ

ely

wit

h S

afeW

ork

to

imp

lem

ent

inno

vati

ve s

olu

tio

nsP

CB

Us

have

incr

ease

d c

apac

ity

to

man

age/

elim

inat

e W

HS

ris

ksS

afeW

ork

ser

vice

s, a

nd r

egul

ato

ry

reg

ime

adap

ted

acc

ord

ing

to

ev

iden

ce f

rom

dat

a sy

stem

s

Ext

erna

l fac

tors

imp

acti

ng o

n w

ork

hea

lth

and

saf

ety

(so

cial

, eco

nom

ic, n

ew t

echn

olo

gie

s, s

cien

tifi

c ad

vanc

es, n

atio

nal p

olic

y)


Fig

ure

2: L

og

ic m

od

el f

or

Act

ion

Are

a 1:

Em

bed

th

e h

ealt

h an

d s

afet

y la

nd

scap

e in

NS

W w

ork

pla

ces

Un

der

Act

ion

Are

a 1,

the

pro

ble

m b

ein

g a

dd

ress

ed is

th

at b

usi

nes

ses

nee

d a

ssis

tan

ce t

o e

mb

ed t

he

safe

ty la

nd

scap

e.

Inp

uts

Act

ivit

ies

Rea

chSh

ort

ter

m o

utco

mes

Med

ium

ter

m o

utco

me

Long

ter

m o

utco

mes

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Un

der

take

tar

get

ed

pro

du

ctiv

e en

gag

emen

t an

d c

olla

bo

rati

on

wit

h

stak

eho

lder

s.

Pea

k in

du

stry

bo

die

s,

asso

ciat

ion

s an

d

com

mu

nit

y le

ader

s.

• Ta

rget

ed s

take

ho

lder

s ar

e aw

are

of

WH

S is

sues

/ac

cou

nta

bili

ties

.

• In

crea

sed

will

ing

nes

s to

ta

ke a

ctio

n.

• Ta

rget

ed s

take

ho

lder

s id

enti

fy w

ays

to im

pro

ve

WH

S in

th

eir

ind

ust

ries

/w

ork

pla

ces.

• Ta

rget

ed s

take

ho

lder

s ad

voca

te f

or

imp

rove

d

WH

S w

ith

in t

hei

r o

wn

se

cto

rs.

• In

du

stri

es p

roac

tive

ly

imp

lem

ent

sect

or

wid

e W

HS

init

iati

ves

to r

edu

ce

wo

rker

s’ e

xpo

sure

to

ris

ks.

• N

SW

wo

rkp

lace

s h

ave

hig

h s

afet

y st

and

ard

s.

• F

ewer

fat

alit

ies,

ser

iou

s in

juri

es, i

llnes

ses

and

m

usc

ulo

skel

etal

dis

ord

ers

at w

ork

.

• A

wel

l-in

form

ed N

SW

co

mm

un

ity

kno

wle

dg

eab

le

abo

ut

WH

S is

sues

.

• A

ll jo

bs

are

des

ign

ed t

o

be

safe

wit

h s

afe

wo

rk

pra

ctic

es a

nd

co

ntr

ols

.

• N

SW

wo

rkp

lace

s h

ave

hig

h s

afet

y st

and

ard

s.

• E

ffec

tive

RT

W p

ract

ices

ar

e em

bed

ded

in N

SW

w

ork

pla

ces.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Lau

nch

co

mm

un

icat

ion

s an

d a

dve

rtis

ing

ca

mp

aig

ns.

NS

W g

ener

al

com

mu

nit

y,

emp

loye

rs a

nd

w

ork

ers.

• In

crea

sed

aw

aren

ess

of

Saf

eWo

rk N

SW

bra

nd

.

• In

crea

sed

leve

ls o

f en

gag

emen

t w

ith

S

afeW

ork

.

• G

ener

al c

om

mu

nit

y,

emp

loye

rs a

nd

wo

rker

s ac

cess

pro

du

cts

and

se

rvic

es.

• In

crea

sed

co

nfi

den

ce a

nd

w

illin

gn

ess

to t

ake

acti

on

.

• E

mp

loye

rs im

pro

ve t

hei

r sy

stem

s an

d c

on

tro

ls.

• W

ork

ers

imp

rove

th

eir

wo

rk

pra

ctic

es.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Dev

elo

p t

ailo

red

pro

du

cts

and

ser

vice

s fo

r sp

ecifi

c in

du

stri

es a

nd

diff

eren

t si

zed

bu

sin

esse

s.

Targ

eted

ind

ust

ries

an

d d

iffer

ent

size

d

bu

sin

esse

s.

• W

ork

pla

ces

un

der

stan

d

the

diff

eren

t el

emen

ts o

f th

e H

&S

lan

dsc

ape.

• W

ork

pla

ces

pro

acti

vely

im

ple

men

t el

emen

ts o

f th

e H

&S

lan

dsc

ape.

• Ta

rget

ed e

mp

loye

rs a

nd

w

ork

ers

acce

ss p

rod

uct

s an

d s

ervi

ces.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Wo

rk w

ith

SIR

A, d

evel

op

an

d d

eliv

er p

rog

ram

s to

im

pro

ve R

TW

pra

ctic

es.

Targ

eted

ind

ust

ries

an

d b

usi

nes

ses.

• Ta

rget

ed b

usi

nes

ses

are

awar

e o

f R

TW

issu

es/

acco

un

tab

iliti

es.

• Ta

rget

ed b

usi

nes

ses

iden

tify

way

s to

imp

rove

R

TW

ou

tco

mes

in t

hei

r w

ork

pla

ces.

• Ta

rget

ed b

usi

nes

ses

imp

lem

ent

RT

W p

ract

ices

in

th

eir

wo

rkp

lace

s.

20 SAFEWORK NSW

3.2.1. Outcomes matrices and performance measures for Action Area 1: Embed the health and safety landscape in NSW workplaces

Table 1: Action Area 1 Inputs Outcomes Matrix

Inputs (for each activity)

Attributes of success Performance measures

Time, Staff, Partners, Budget

Time, staff, budget are applied to the program as planned, for each year.

Proportion of planned time, staff, partners, budget applied, for each year.

Proportion within budget.

Appropriate partners identified. Stakeholders agree all relevant partners identified.

Table 2: Activities, reach, short term outcomes and medium term outcomes

Outcome level Outcomes and attributes of success Performance measures

Undertake targeted productive engagement and collaboration with stakeholders

Activities Priority peak industry bodies, associations and community leaders effectively engaged in collaboration.

Effective collaborative mechanisms in place allow all interest groups to be involved.

Number, type of stakeholders who agree to collaborate, compared with target.

Feedback on effectiveness of engagement strategies.

Reach All priority peak industry bodies, associations and community leaders are effectively reached.

Number, type of stakeholders who collaborate on specific issues.

Areas of collaboration.

Short term outcomes

Targeted priority peak industry bodies, associations and community leaders are:

• aware of WHS issues and accountabilities for these

• report an increased willingness to take action

• identify ways to improve WHS in their industries.

Number of priority peak industry bodies, associations and community leaders who can identify WHS issues and accountabilities relevant to their industry.

Number, type of stakeholder who report they are willing to act on WHS issues.

List of agreed identified actions to improve WHS.



Medium term outcomes

Targeted stakeholders advocate for improved WHS within their own sectors, demonstrate leadership using industry forums and levers.

Industries proactively implement sector wide WHS initiatives to reduce workers’ exposure to risks, commit resources.

Number of priority peak industry bodies, associations and community leaders who advocate for improved WHS, type of advocacy.

Evidence of industry driving change (eg – lobbying, audits, provision of information and training).

Kinds of actions being advocated (eg – changes in equipment design, complaints processes).

Number, type of initiatives by sector.

Feedback on barriers to implementation.

Launch communication and advertising campaigns

Activities Campaigns and advertising designed.

Messages developed and tested for audience appeal.

Number, type of campaigns, audience.

Extent each campaign is implemented, compared with plan.

Reach Target audience/s aware of campaign messages and services and products

For each campaign, recall rates of messages and services and products.

Short term outcomes

Target audience/s have increased awareness of SafeWork NSW brand.

Target audience have increased levels of engagement with SafeWork.

Proportion of target groups report being aware of SafeWork NSW brand.

Numbers of target audience/s seek further information through visits to website.


General community, employers and workers access products and services.

Increased confidence and willingness to take action.

Employers improve their systems and controls.

Workers improve their work practices.

Numbers of target audience/s access advertised services and products and ways services are accessed (business visits, phone calls).

Proportion of users of services and products satisfied, find these useful.

Number of employers in different types of industries who make changes to WHS systems and controls, kinds of changes made.

Proportion of workers who report changing work practices, nature of changes (drawn from a sample of businesses who have changed systems and controls).

Feedback on barriers to changing systems and controls and worker behaviour.

22 SAFEWORK NSW


Develop tailored products and services for specific industries and different sized businesses

Activities Specific industries and their WHS service and product needs identified.

Products and services developed, tailored for each industry and for different sized businesses within each industry, address the health and safety landscape.

Products and services effectively promoted to specified industries.

Number, type of industries and products and services implemented.

Extent each service and product is delivered as intended.

Match of products and services with industry needs.

Number and types of promotional activities.

Reach All targeted industries and different sized businesses are aware of new products and services and access these.

Numbers and sizes of businesses from targeted industries access new services and products.

Short term outcomes

Products and services useful, meet information needs in timely way.

Responsible staff in different size businesses understand elements of the H&S landscape and how these can be applied in their own workplace.

Proportion of responsible business staff who use services and products are satisfied, find these useful, get timely response.

Numbers of responsible staff in a sample of businesses of different sizes can describe relevant elements of the H&S landscape.


Workplaces proactively implement elements of the H&S landscape.

Numbers of businesses of different sizes have fully implemented relevant elements H&S landscape (new policy, guidelines, systems and controls).

Feedback on barriers to implementing H&S landscape elements.

Work with SIRA, develop and deliver programs to improve RTW practices

Activities SafeWork works with SIRA on RTW programs, contributes expertise on health and safety issues for returning workers.

Processes to support effective collaboration established.

Number, target group for and focus of programs developed, compared with target.

Nature of contribution by SafeWork.

Feedback from stakeholders.

Reach All targeted industries and businesses are effectively reached.

Number, type of industries and businesses participate in RTW programs.



Short term outcomes

Targeted businesses are aware of relevant RTW issues/accountabilities.

Targeted businesses identify ways to improve RTW outcomes in their workplaces.

Proportion of businesses find support or information offered through programs useful.

Number, type of businesses report being aware of RTW issues and accountabilities, can describe issues relevant to their operating context and actions to address these.

Numbers of businesses intending to change RTW practices.


Targeted businesses implement RTW practices in their workplaces.

Numbers of businesses that can demonstrate implementing relevant RTW practices, description of practices.

Rates of return to work and whether employment sustained after return.

Feedback on barriers to implementing RTW practices.

24 SAFEWORK NSW

Fig

ure

3: L

og

ic m

od

el f

or

Act

ion

Are

a 2:

Pri

ori

tise

sec

tors

, har

ms,

wo

rker

s an

d w

ork

pla

ces

wh

ere

the

mo

st s

ign

ifica

nt W

HS

ris

ks e

xist

Un

der

Act

ion

Are

a 2,

th

e p

rob

lem

bei

ng

ad

dre

ssed

is t

hat

hig

h r

isk

sect

ors

an

d h

arm

s n

eed

to

be

pri

ori

tise

d, s

o S

afeW

ork

nee

ds

to w

ork

wit

h in

du

stry

to

id

enti

fy a

nd

ad

dre

ss h

igh

ris

k se

cto

rs a

nd

har

ms,

pro

tect

th

e m

ost

at

risk

wo

rker

s an

d e

nsu

re t

hat

hig

h r

isk

wo

rkp

lace

s m

eet

com

plia

nce

sta

nd

ard

s.

Inp

uts

Act

ivit

ies

Rea

chSh

ort

ter

m o

utco

mes

Med

ium

ter

m o

utco

me

Long

ter

m o

utco

mes

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Wo

rk w

ith

rel

evan

t st

akeh

old

ers

to a

dd

ress

h

igh

imp

act

har

ms

and

W

HS

issu

es o

f ta

rget

ed

hig

h r

isk

ind

ust

ries

an

d a

t ri

sk w

ork

ers.

Pea

k in

du

stry

b

od

ies,

ass

oci

atio

ns,

co

mm

un

ity

lead

ers,

em

plo

yers

an

d

wo

rker

s.

• Im

pro

ved

aw

aren

ess

of

hig

h im

pac

t h

arm

s an

d

WH

S is

sues

of

targ

eted

in

du

stri

es a

nd

wo

rker

s.

• In

crea

sed

co

nfi

den

ce,

cap

abili

ty a

nd

will

ing

nes

s to

tak

e ac

tio

n.

• In

du

stri

es a

nd

bu

sin

esse

s im

ple

men

t W

HS

init

iati

ves

and

ad

op

t h

igh

leve

l co

ntr

ols

.

• H

igh

ris

k in

du

stri

es a

re

sig

nifi

can

tly

safe

r an

d

hea

lth

ier.

• H

igh

imp

act

har

ms

are

elim

inat

ed o

r si

gn

ifica

ntl

y re

du

ced

.

• F

ewer

fat

alit

ies,

ser

iou

s in

juri

es, i

llnes

ses

and

m

usc

ulo

skel

etal

dis

ord

ers

at w

ork

.

• A

saf

e an

d h

ealt

hy N

SW

p

ub

lic s

ecto

r.

• S

afe

and

hea

lthy

w

ork

pla

ces.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Wo

rk w

ith

th

e N

SW

G

ove

rnm

ent

sect

or

to

dev

elo

p a

nd

del

iver

WH

S

init

iati

ves.

Man

agem

ent

and

st

aff o

f ta

rget

ed

NS

W G

ove

rnm

ent

dep

artm

ents

.

• Im

pro

ved

aw

aren

ess

of

WH

S is

sues

.

• In

crea

sed

co

nfi

den

ce,

cap

abili

ty a

nd

will

ing

nes

s to

tak

e ac

tio

n.

• N

SW

Go

vern

men

t d

epar

tmen

ts im

ple

men

t W

HS

init

iati

ves.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Wo

rk w

ith

hig

h r

isk

wo

rkp

lace

s to

imp

rove

W

HS

.

Man

agem

ent

and

st

aff o

f se

lect

ed h

igh

ri

sk w

ork

pla

ces.

• Im

pro

ved

aw

aren

ess

of

WH

S is

sues

.

• In

crea

sed

co

nfi

den

ce,

cap

abili

ty a

nd

will

ing

nes

s to

tak

e ac

tio

n.

• Ta

rget

ed b

usi

nes

ses

imp

lem

ent

WH

S in

itia

tive

s

• Ta

rget

ed b

usi

nes

ses

ado

pt

hig

h le

vel c

on

tro

ls.


3.2.2. Outcomes matrices and performance measures for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist






Priority harm prevention projects identified.

Proportion of planned time, staff, budget applied, for each year.


Number, type of projects selected, coverage of projects against identified risks.




Work with relevant stakeholders to address high impact harms and WHS issues of targeted high risk industries and at risk workers

Activities Relevant stakeholders effectively engaged in collaborative projects on ways to address high risk harms.

Effective collaborative mechanisms in place allowing all interest groups to be involved.

Number, type of stakeholders who agree to collaborate, compared with targets identified in project plans.

Areas of collaboration, and which risks addressed (high impact harms being addressed, WHS issue).


Projects delivered as intended.

Reach All stakeholders (priority peak industry bodies, associations, community leaders, employers and workers) are effectively reached.

Number, type of stakeholders who collaborate on specific high impact harm projects.

Nature of collaborative initiatives.

Short term outcomes

All stakeholders are more aware of high impact harms and WHS issues.

All stakeholders are more confident in their ability to address high impact harms in their industry or workplace.

All stakeholders are capable of and willing to take action.

Number of stakeholders who can identify high impact harms and WHS issues relevant to their industry.

Number, type of stakeholder who report they are willing to act on high impact harms and WHS issues, able to do so.

List of agreed identified initiatives to address high impact harms.

26 SAFEWORK NSW



Industries and businesses implement agreed WHS initiatives, address high impact harms, industry-wide implementation.

Evidence of industry driving change (eg – provision of information and training, funding).

Number, type of initiatives by sector.

Evidence of reduced exposure to risk for workers in targeted industries.

Evidence of high-level controls being implemented.

Fewer complaints about WHS from workers.

Feedback on barriers to implementation

Extent of implementation by industry and sector.

Work with the NSW Government sector to develop and deliver WHS initiatives

Activities All targeted departments effectively engaged in collaboration to develop and deliver suitable WHS initiatives.

Effective collaborative mechanisms in place.

Number of departments who agree to collaborate, compared with targets

Areas of collaboration (description of initiatives)

Reach All responsible managers and staff members of targeted NSW Government departments are effectively reached.

Number of departments who collaborate on WHS initiatives, positions of staff involved.

Nature of joint initiatives.

Short term outcomes

All responsible managers and staff have improved awareness of WHS issues, increased confidence, capability and willingness to take action.

Number of responsible managers and staff members who can identify WHS issues relevant to their work.

Number of responsible managers and staff members who report they are willing to act on WHS issues, able to do so.

List of agreed identified actions to address WHS issues across departments.


All targeted NSW Government departments implement agreed WHS initiatives, department leaders drive implementation.

Evidence of department managers driving change (eg – provision of information and training, guidelines).

Number, type of actions taken by departments.


Extent of implementation across targeted departments.

Evidence of reduced exposure to risk for workers in NSW Government departments

Fewer complaints about WHS from workers in Government departments.



Work with high risk workplaces to improve WHS

Activities All selected high risk workplaces’ managers and staff effectively engaged in collaborative initiatives to improve WHS.

Effective collaborative mechanisms in place.

Number of selected workplaces that agree to collaborate, compared with targets.

Areas of collaboration (description of initiatives).

Reach All relevant managers and staff in selected workplaces are effectively reached.

Number, type of staff who collaborate on initiatives to improve WHS.

Nature of initiatives.

Short term outcomes

All relevant managers and staff are more aware of how WHS issues are addressed.

All relevant managers and staff are more confident that WHS issues can be addressed in their workplace.

All relevant managers and staff are capable of and willing to take action.

Number of relevant managers and staff who can identify WHS issues relevant to their workplace.

number of relevant managers and staff who report they are willing to act on high risk WHS issues, able to do so.

List of agreed identified actions to address WHS issues.


Targeted businesses implement agreed WHS initiatives.

Evidence of managers driving change (eg – provision of information and training, funding).

Number, type of actions taken by selected businesses.

Evidence of high-level controls being implemented.

Evidence of reduced exposure to risk for workers in targeted industries.

Fewer complaints about WHS from workers.


Extent of implementation by selected businesses.

28 SAFEWORK NSW

Fig

ure

4: L

og

ic m

od

el f

or

Act

ion

Are

a 3:

Bu

ild e

xem

pla

r re

gu

lato

ry s

ervi

ces

Un

der

Act

ion

Are

a 3

, th

e p

rob

lem

bei

ng

ad

dre

ssed

is t

hat

Saf

eWo

rk n

eed

s to

pro

vid

e ri

gh

t to

uch

reg

ula

tory

ap

pro

ach

so

th

at c

ust

om

ers

can

be

con

fid

ent

that

S

afeW

ork

will

en

forc

e W

HS

law

s to

pro

tect

wo

rker

s, t

hat

Saf

eWo

rk w

ill u

se d

ata

and

insi

gh

ts t

o m

ake

dec

isio

ns

and

pro

vid

e in

no

vati

ve s

ervi

ces

to m

eet

the

nee

ds

of

bu

sin

ess.

Inp

uts

Act

ivit

ies

Rea

chSh

ort

ter

m o

utco

mes

Med

ium

ter

m o

utco

me

Long

ter

m o

utco

mes

Tim

e, S

taff

, P

artn

ers,

B

ud

get

En

able

an

d e

mp

ow

er

wo

rkp

lace

s to

man

age

thei

r h

ealt

h a

nd

saf

ety.

NS

W w

ork

pla

ces

and

ass

oci

ated

st

akeh

old

ers.

• P

CB

Us

are

awar

e o

f S

afeW

ork

’s p

rod

uct

s an

d

serv

ices

.

• P

CB

Us

un

der

stan

d t

hei

r W

HS

res

po

nsi

bili

ties

.

• P

CB

Us

acce

ss S

afeW

ork

’s p

rod

ucts

an

d s

ervi

ces,

sat

isfie

d w

ith t

hes

e.

• P

CB

Us

revi

ew a

nd im

pro

ve t

hei

r w

ork

pra

ctic

es, s

yste

ms

and

co

ntro

ls.

• In

tera

ctio

ns w

ith c

usto

mer

s m

eet

exp

ecte

d s

tan

dar

ds

of

serv

ice.

• N

SW

is r

eco

gn

ised

as

th

e b

est

pla

ce

to s

tart

an

d r

un

b

usi

nes

ses.

• S

W N

SW

is s

een

as

a t

rust

ed a

nd

in

flu

enti

al r

egu

lato

r.

• S

W N

SW

is

reco

gn

ised

as

op

en

and

tra

nsp

aren

t,

and

dri

ven

by

evid

ence

.

• S

W N

SW

is

reco

gn

ised

as

fost

erin

g h

ealt

h a

nd

sa

fety

inn

ova

tio

n.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Cla

rify

an

d p

rom

ote

S

afeW

ork

NS

W’s

re

gu

lato

ry, a

dvi

sory

an

d

enfo

rcem

ent

app

roac

h.

NS

W w

ork

pla

ces

Saf

eWo

rk s

taff

.

• P

CB

Us

are

awar

e o

f S

afeW

ork

’s r

egu

lato

ry,

advi

sory

an

d e

nfo

rcem

ent

app

roac

h.

• P

CB

Us

und

erst

and

th

eir

WH

S r

esp

on

sib

iliti

es a

nd

w

hat

co

mp

lian

ce lo

oks

like

.

• P

CB

Us

acce

ss S

afeW

ork

’s p

rod

ucts

an

d s

ervi

ces,

sat

isfie

d w

ith t

hes

e.

• P

CB

Us

revi

ew a

nd im

pro

ve t

hei

r w

ork

pra

ctic

es, s

yste

ms

and

co

ntro

ls in

lin

e w

ith S

afeW

ork

’s

reg

ulat

ory

, ad

viso

ry a

nd

en

forc

emen

t ap

pro

ach

.

• In

tera

ctio

ns w

ith c

usto

mer

s m

eet

exp

ecte

d s

tan

dar

ds

of

serv

ice.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

En

able

th

e u

se o

f d

ata

and

evi

den

ce t

o in

form

W

HS

po

licie

s an

d

pro

gra

ms.

NS

W w

ork

pla

ces

and

ass

oci

ated

st

akeh

old

ers.

Saf

eWo

rk s

taff

.

• P

CB

Us

are

awar

e o

f la

test

d

evel

op

men

ts/a

dva

nce

s in

WH

S.

• S

afeW

ork

use

dat

a sy

stem

s an

d e

vid

ence

to

in

form

dec

isio

n-m

akin

g.

• P

CB

Us

revi

ew a

nd im

pro

ve t

hei

r w

ork

pra

ctic

es, s

yste

ms

and

co

ntro

ls in

lin

e w

ith t

he

late

st

evid

ence

an

d s

tan

dar

ds.

• S

afeW

ork

ser

vice

s an

d r

egul

ato

ry

reg

ime

adap

ted

acc

ord

ing

to

ev

iden

ce f

rom

dat

a sy

stem

s.

Tim

e, S

taff

, P

artn

ers,

B

ud

get

Dev

elo

p a

nd

pro

mo

te

inn

ova

tive

ap

pro

ach

es t

o

wo

rk h

ealt

h a

nd

saf

ety

issu

es.

Pea

k b

od

ies,

as

soci

atio

ns,

co

mm

un

ity

lead

ers,

P

CB

Us

and

wo

rker

s.

Saf

eWo

rk s

taff

.

• P

CB

Us

and

key

st

akeh

old

ers

reco

gn

ise

the

ben

efits

of

wo

rkin

g

wit

h S

afeW

ork

to

dev

elo

p

inn

ova

tive

so

luti

on

s.

• S

afeW

ork

use

hu

man

ce

ntre

d d

esig

n p

rin

cip

les

to

dev

elo

p e

ffec

tive

so

luti

ons

.

• P

CB

Us,

key

sta

keho

lder

s an

d

par

tner

s w

ork

co

llab

ora

tive

ly w

ith

Saf

eWo

rk t

o im

ple

men

t in

nova

tive

solu

tio

ns.

• P

CB

Us

and

key

sta

keho

lder

s ha

ve

incr

ease

d c

apac

ity

to m

anag

e o

r el

imin

ate

WH

S r

isks

.


3.2.3. Outcomes matrices and performance measures for Action Area 3: Build exemplar regulatory services






Proportion of planned time, staff, budget applied, for each year.





Enable and empower workplaces to manage their health and safety

Activities Products and services for workplaces/PCBUs provide clear and appropriate advice about WHS issues and how to manage these.

Products and services effectively promoted to PCBUs.

Number, type of products and services implemented.

Projects implemented as intended.

List of SafeWork information products and audience for these, gaps in information.

Extent information meets SafeWork communication protocols/guidelines.

Extent to which products and services meet the needs of customers.

Number, type of promotional activities.

Reach All workplaces and associated stakeholders are effectively reached.

Number of PCBUs who seek information about services and products, when needed.

Short term outcomes

All PCBUs are aware of products and services.

All PCBUs understand their WHS responsibilities, relevant to their business.

Number of PCBUs who can identify WHS responsibilities relevant to their business.

Number of PCBUs who are aware of products and services.


PCBUs access products and services when needed.

PCBUs review and improve their work practices, systems and controls in response to advice received.

SafeWork staff interactions with PCBUs meet expected standards of service.

Number of PCBUs who access services and products.

Proportion of PCBUs who use services and products are satisfied, find these useful, get timely response.

Number of PCBUs review and improve work practices, systems and controls.

Number of complaints about services.

30 SAFEWORK NSW


Clarify and promote SafeWork NSW’s regulatory, advisory and enforcement approach

Activities Appropriate products and services for PCBUs provide clear information on SafeWork’s regulatory, advisory and enforcement approach.

Information effectively promoted to PCBUs.

Number, type of products and services implemented.

Projects implemented as intended.

Number, type of promotional activities.

Reach All NSW workplaces are effectively reached.

Number of PCBUs who seek information about services and products related to compliance and enforcement.

Short term outcomes

PCBUs are aware of SafeWork’s regulatory, advisory and enforcement approach.

PCBUs understand their WHS responsibilities, what compliance looks like in their type of business and understand penalties for non-compliance.

Relevant information developed, is clear and consistent across the agency.

Number of PCBUs who are aware of products and services related to compliance and enforcement.

Number of PCBUs who can accurately describe SafeWork’s regulatory, advisory and enforcement approach and what compliance looks like in their business.

SafeWork interactions with PCBUs.

List of SafeWork information products and audience for these, gaps in information.

Extent information meets SafeWork communication protocols/guidelines.


PCBUs access SafeWork’s regulatory, advisory and enforcement services.

PCBUs review and improve their work practices, systems and controls in line with SafeWork’s regulatory, advisory and enforcement approach.

Number of PCBUs who access services and products related to compliance and enforcement.

Proportion of PCBUs who use services and products are satisfied, find these useful, get timely response.

Number of PCBUs who review and improve work practices, systems and controls, extent these are in line with regulatory, advisory and enforcement approach.



Enable the use of data and evidence to inform WHS policies and programs

Activities SafeWork makes data and evidence available to PCBUs about WHS developments and advances.

Evidence is translated and easy to understand for different types of industries and businesses, readily accessible.

SafeWork establishes data systems to collect evidence and support risk-based regulatory approach.

Number and topics of papers published/presented/advice provided about WHS developments and advances (amount of dissemination).

Extent evidence is produced and disseminated compared to plan.

Data systems functioning according to specifications and requirements, gaps in functions.

Reach All NSW workplaces and associated stakeholders are effectively reached.

Number of PCBUs who report published information is useful and relevant.

Number of PCBUs who seek and find information about data and evidence on WHS developments and advances.

Data analytics meet SafeWork information needs.

Short term outcomes

PCBUs are aware of latest developments/advances in WHS and how these apply to their business.

SafeWork’s managers and executive seek out and use evidence to inform decision-making.

Number of PCBUs who are aware of WHS developments and advances.

Number of PCBUs who can accurately describe how the evidence can be applied to their business.

Examples of how data has informed decision-making.


PCBUs review and improve their work practices, systems and controls in line with latest evidence about effective WHS practices, systems and controls.

SafeWork systems, services and policies adapted according to evidence base.

Number of PCBUs who access information products about latest WHS developments and advances.

Proportion of PCBUs who access information or advice are satisfied, find these useful.

Number of PCBUs who review and improve work practices, systems and controls, extent these are in line with latest WHS developments and advances.

Description of adaptations made to SafeWork systems, services and policies, rationale for change.

32 SAFEWORK NSW


Develop and promote innovative approaches to work health and safety issues

Activities Peak bodies, associations, community leaders, PCBUs and workers effectively engaged in collaboration.

Effective collaborative mechanisms in place allow all interest groups to be involved.

SafeWork actively promotes innovation internally and within the business community.

SafeWork use human centred design principles to develop effective solutions.

Number, type of stakeholders who agree to collaborate, compared with target.


Change requests, number of innovative projects.

Reach Peak bodies, associations, community leaders, PCBUs and workers are effectively reached, agree on issues to be addressed.

Type of joint initiatives and innovations agreed, WHS issues being addressed.

Short term outcomes

PCBUs and key stakeholders recognise the benefits of working with SafeWork to develop innovative solutions.

Number of stakeholders who agree that working with SafeWork is beneficial, believe that joint working can address WHS issues.


PCBUs, key stakeholders and partners work collaboratively with SafeWork to implement innovative solutions.

PCBUs and key stakeholders have increased capacity to manage or eliminate WHS risks.

Number, type of stakeholders who collaborate on specific issues and innovative approaches.

Number of PCBUs who implement innovations.

Description of innovation and how it impacts on business capacity to manage and eliminate risks.


3.2.4. Outcomes matrix for long term outcomes

Long term outcomes are separated because the achievement of these depends on the combined successful implementation of all three action areas and on actions taken by industry and government partners, national efforts to address work health and safety and on external social influences.

Table 7: Outcomes matrix for Roadmap long term outcomes

Outcomes Performance measures

NSW workplaces have high safety standards

Proportion of workplaces that meet safety standards.

Fewer fatalities, serious injuries, illnesses and musculoskeletal disorders at work

Achievement against Fatality Target – 20% decline (1 Quarter Lag).

Number of NSW Fatalities.

Fatality Rate.

Achievement against Serious Injury and Illness Target – 30% decline (1 Quarter Lag).

Number of NSW Serious Injuries and Illnesses.

Serious Injuries and Illnesses Rate.

% to Serious Musculoskeletal Injuries and Illnesses Target – 30% decline (1 Quarter Lag).

Number of NSW Serious Musculoskeletal Injuries and Illnesses.

Serious Musculoskeletal Injuries and Illnesses Rate.

Sustained reduction in Fatalities and Serious Injury and Illness Rates.

A well-informed NSW community knowledgeable about WHS issues

Proportion of the community who understand key work health and safety issues.

Fewer adverse social and economic impacts on families.

All jobs are designed to be safe with safe work practices and controls

Number and proportion of businesses where jobs are designed with safe work practices and controls by industry category.

Number and proportion of government agencies where jobs are designed with safe work practices and controls by industry category.

Effective RTW practices are embedded in NSW workplaces

Proportion of workplaces with appropriate return to work processes that are consistently applied.

NSW recognised as the best place to run a business

Lower cost of running a business compared to baseline (and adjusted for CPI increases).

Reduced regulatory impost on business.

SafeWork seen as a trusted and influential regulator

Number of stakeholders who agree that SafeWork is a credible, trustworthy and responsive organisation.

SafeWork is recognised as open and transparent and driven by evidence

Number of stakeholders who agree that SafeWork is an open and transparent organisation, driven by evidence.

SafeWork is recognised as fostering health and safety innovation

Number of stakeholders who agree that SafeWork fosters health and safety innovation.

34 SAFEWORK NSW

3.3. Roadmap evaluation and guiding questions

Evaluation questions support reality-testing, which is finding out what is actually going on in a program and its effects. This can be compared to what was intended and hoped for as stated in the WHS Roadmap for NSW 2022. Evaluation questions also shape the focus of the evaluation efforts.

Guiding evaluation questions point towards additional information needed to answer the evaluation questions for each action area. The outcome matrices prescribe the full set of information that can potentially be collected to answer the evaluation questions, and likely sources of data. What is actually collected then depends on decisions about identified priorities, the available resources, and the feasibility and ethical safeguards for collecting these data.

Evaluation studies will address one or more of the key evaluation questions, which span the logic model (Figure 1) and the three action areas from bottom to top. Which evaluation questions you address will depend on the purpose and scope of your evaluation study. A bank of guiding questions is proposed for each evaluation question – these may be specific to action areas, and have informed the identification of outcomes matrix measures. You can choose from the bank of guiding questions or add guiding questions to clarify what specific aspects of a project you are interested in assessing.

SafeWork core services are assistance services, public awareness (excluding major media campaigns), consultative mechanisms, prevention programs, information and advice services, financial incentives, sanctions, transparency and accountability, regulatory response, authorisations, verifications and audits.

Evaluation questions

Inputs

1. Were adequate resources allocated (staff, time and budget) to research, develop and deliver initiatives under each of the Roadmap action areas and for each Directorate?

2. How consistent are project communication materials and protocols with the right touch regulatory approach?

3. Were initiatives selected based on the best evidence available?

4. Has input from partners shaped the design of initiatives and in what ways?

Activities

5. To what extent were initiatives implemented as intended?

6. How well did SafeWork manage risks and address barriers to implementation?

7. How appropriate were project activities for the task/s?

8. How appropriate were tailored products and services for engaging stakeholders and for meeting the information needs of different audiences?

9. To what extent were communication materials and protocols consistent across the organisation?

10. What unexpected or emerging factors influenced the implementation of initiatives?

11. What consultations have taken place, who participated and have changes been made to improve SafeWork services and products as a result?


Reach

12. How effective were the actions in reaching their target audiences? For each target audience, what was the pattern of reach (number, proportion, characteristics of those reached)?

13. What methods were more or less effective in reaching their target audiences? What success factors and barriers were identified?

Short term outcomes

14. Are SafeWork initiatives under each action area achieving their intended short term outcomes?

Guiding questions for Action Area 1: Embed the health and safety landscape in NSW workplaces

• To what extent are the general community, employers and workers who were reached by SafeWork initiatives:

a. aware of the SafeWork brand, and has the level of awareness changed?

b. engaged with SafeWork; what is the nature of the engagement and how has it changed over time?

• Which engagement strategies with small, medium and large businesses and peak groups of employers and employees have been more or less successful?

• To what extent are peak industry bodies, associations and community leaders

a. aware of their WHS issues and accountabilities and has their level of awareness changed?

b. able to identify ways to improve work health and safety in their industries?

• To what extent do businesses of all sizes better understand the relevant elements of the health and safety landscape?

• Are targeted businesses more aware of return to work issues and accountabilities and able to identify ways to improve their processes?

Guiding questions for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist

• What has been done to elevate work health and safety in high risk workplaces?

• To what extent are targeted industries (peaks, employers and workers) aware of high impact harms and health and safety risks and has their level of awareness changed?

• To what extent are managers in NSW Government agencies aware of WHS issues and has their level of awareness changed?

• To what extent are targeted industries, management and staff in selected high risk workplaces and managers in NSW Government agencies more confident, capable and willing to act on health and safety issues in the workplace?

• To what extent are targeted industries aware of relevant SafeWork products and services?

36 SAFEWORK NSW

Guiding questions for Action Area 3: Build exemplar regulatory services

• What has been done to elevate work health and safety in workplaces?

• Is the compliance regime being consistently delivered under the legislation?

• What kinds of impacts have compliance activities had on industry attitudes to health and safety?

• What actions have non-compliant businesses taken to address work health and safety issues identified by inspectors?

• What has been done to elevate work health and safety in workplaces and how does this translate to contracts that are won?

• To what extent are persons conducting businesses and undertakings:

a. aware of relevant SafeWork products and services and has their level of awareness changed?

b. aware of SafeWork’s regulatory, advisory and enforcement approach?

c. understanding of their work health and safety responsibilities and what compliance looks like?

d. aware of SafeWork products and services?

e. aware of latest developments and advances in work health and safety?

• Do persons conducting businesses and undertakings, peak bodies and associations, workers and community leaders recognise the benefits of working with SafeWork to address work health and safety issues?

15. What factors influenced the achievement of short term outcomes?

16. What changes to services, products and projects are needed to better achieve the intended outcomes, and can lessons be applied to other work?


17. Are SafeWork initiatives under each action area achieving their intended intermediate outcomes?


• How many businesses of all sizes have implemented or upgraded work health and safety landscapes as a result of SafeWork NSW’s intervention and improved their safety systems and controls; what kinds of businesses and industries?

• How many businesses have implemented a strategy to improve or work towards a mentally healthy workplace?

• How many businesses have implemented a strategy to reduce the risk of musculoskeletal disorders and what types of controls are in place?

• In what ways have Roadmap partners (peak industry bodies, associations and community leaders) advocated to encourage workplaces to improve health and safety and reduce risks?

• To what extent are general members of the community, employers and workers more confident, capable and willing to act on health and safety issues in the workplace?

• To what extent have workers changed their practices to work in a healthier and safer way?

• How many businesses of all sizes have a return to work plan in place as a result of SafeWork’s intervention?

• What products and services have the community and workers accessed; who accessed which services/products and how many?



• Has targeting workplaces in high risk sectors and sectors where workers are at risk of high impact harms improved compliance and identification of safety risks in these industries?

• What kinds of actions to address health and safety issues have been taken by targeted industries, high risk workplaces and government agencies?

• In the chemicals area, what behaviors and practices have changed in targeted industries to targeted controls? Have these changes been sustained? Do industries have the capability to sustain these changes going forward?

• To what extent has workers’ exposure to hazards been reduced in workplaces targeted by SafeWork NSW?

• How many targeted employers have improved work practices and changed behaviours to better protect workers’ health and safety and specifically, those businesses in sectors where workers at high risk of harm and/or in high risk industries or government agencies?


• What products and services have persons conducting businesses and undertakings accessed; who accessed which services/products and how many?

• Do compliance and enforcement activities influence behaviour and what activities are most effective?

• What kinds of actions to identify and then address health and safety issues have been taken by persons conducting businesses and undertakings? Are the actions in line with the latest evidence?

• Are businesses of all sizes better equipped and more able to manage health and safety issues?

• In what areas of health and safety has SafeWork collaborated with peak bodies, associations and community leaders?

• What kinds of health and safety innovations have been introduced as a result of collaboration with industry?

• Has the Roadmap delivered a level playing field in terms of competitiveness within industries?

18. Were there any unintended consequences with respect to the successful implementation of the three actions areas for the community, industry, employees and workers?

19. What external factors influenced the achievement of medium term outcomes?

20. Have the lessons learnt been applied across the organisation to better achieve outcomes?

Long term outcomes

21. What impact has the Roadmap had on trends in serious workplace injuries, illnesses, musculoskeletal disorders and fatalities in different sectors?

22. What external factors are influencing the observed trends in serious workplace injuries, illnesses, musculoskeletal disorders and fatalities in different sectors?

23. In what ways has the Roadmap contributed to achieving SafeWork’s vision of workers in NSW leading healthy, safe and productive working lives?

38 SAFEWORK NSW


• What impact has the Roadmap had on how SafeWork is perceived by industry, workers and the community?

• What impact has the Roadmap had on the NSW community’s understanding about work health and safety issues?

• What impact has the Roadmap had on physical and mental environments in the workplace and on supply chains?

• What impact has the Roadmap had on making safe return to work practices part of business as usual in NSW workplaces?


• What impact has the Roadmap had on health and safety standards in government and the private sectors?

• What impact has the Roadmap had on health and safety in high risk industries?

• What impact has the Roadmap had on the incidence and prevalence of high impact harms?


• What impact has the Roadmap had on business perceptions about NSW as the best place to start and run a business?

• What impact has the Roadmap had on perceptions about SafeWork’s reputation as an agency that fosters health and safety innovation?

24. What external factors are influencing the observed trends in action area policy outcomes?


4. Performance monitoringSafeWork has performance monitoring systems in place and is further refining these to collect performance information.

4.1. About performance monitoringPerformance monitoring system is the provision of outcome information about whether or not a project/program is on track, specifically about how much has been done to what quality and sometimes, the short term outcomes achieved. Performance monitoring:

• is often a part of evaluation, but monitoring by itself is not evaluation and is less in-depth than evaluation

• is ongoing while evaluation is typically periodic

• mainly uses data captured as part of delivering services

• informs quality assurance activities, and

• is used to manage programs/businesses on an ongoing basis.

A good monitoring system provides the required monitoring as performance indicators and also provides those who submit data with reports that explain how their information has been used to motivate them to continue to provide good quality data.

What monitoring systems are

Feature Purpose

Logic Needs to be clear about what success looks like, used to identify indicators.

Measurement Outlines what is feasible to measure and source of data.

Reporting Communicates progress and achievements (eg – dashboard reports).

Improvement Indicates how the information will be used to improve things (quality assurance activities).

Accountability Identifies appropriate checks and balances (eg – data quality checks).

40 SAFEWORK NSW

How monitoring relates to program logic and evaluation

Program logic Domain Potential evaluation methods

Long term outcomes

Evaluation/research

ABS Work-related Injuries Survey

Traumatic Fatalities Database

Workers Compensation Claims Database

Longitudinal studies


Evaluation Post surveys, interviews and observation, case studies

Longitudinal studies

Short term outcomes

Monitoring Lead indicator data collected via surveys and web analytics to assess reach, administrative databases

Reach Monitoring Administrative databases

Surveys

Interviews

Domain of monitoring

Activities Monitoring Document reviews

Interviews

Inputs Evaluation Review of initiative and project plans, other documents

4.2. Collection of performance metricsThe evaluation framework should be complemented by a monitoring plan to progressively collect performance data that will be needed for the evaluation activities.

Organisational-wide and directorate-level performance or lead indicators and metrics are being developed in 2017 and systems for measuring these are being identified or existing systems adapted. Six lead indicators and metrics will track the achievement of immediate outcomes that will be shared externally, quarterly, to track the progress of the Roadmap. These metrics have been included where known in the outcomes matrices in chapter 3.

4.2.1. Reporting performance metrics

SafeWork is designing a balanced score card to report performance information to potential internal and external users of the information.

4.3. AccountabilitySafeWork is currently reviewing and refining data quality checking processes.


5. Evaluation governance, evaluation planning and advice on methods

This section suggests governance structures for evaluation activity, a decision support tool and describes the evaluation planning cycle.

5.1. Governance and management of evaluation studiesStrong and effective governance and management structures for the evaluation of Roadmap initiatives are necessary to ensure that appropriate evaluation studies are conducted and stakeholders are engaged in the evaluation. Governance structures also provide a mechanism to inform program improvement and policy decisions. For the Roadmap, it is particularly important to have governance structures in place that facilitate decision-making and allow advice to be sourced when partners are involved in an initiative, and when an external team is conducting the evaluation. One overarching principle is that the governance arrangements for the evaluation of the Roadmap should be aligned with the governance structures for the delivery of the Roadmap.

The Executive (or relevant governance committee) will have oversight for evaluation and for decision-making about funding of individual evaluation studies. The Committee will be guided by the decision support tool shown in section 5.4. Business Performance will provide advice to the relevant governance committee (as requested), manage the commissioning of external evaluation studies and be responsible for ensuring that evaluation activities dovetail with the regular monitoring of Roadmap outputs.

Role and functions of evaluation steering committees

A steering committee is appropriate when a new program of work is planned and where this work involves multiple sectors and stakeholders; and where decisions about program improvement and policy are expected to be influenced directly by the findings of the evaluation.

The relevant governances committee’s role in regard to evaluation governance could encompass:

• supporting the implementation of robust and credible evaluation studies across each action area

• making decisions about which initiatives should be evaluated, based on the application of the decision support tool and on advice from Business Performance (as requested)

• making decisions about funding amounts and mix for evaluation studies

• making decisions about the procurement of external evaluation services

• approving detailed evaluation plans for Tier 3 and Tier 4 evaluations

• accepting evaluation reports

• identifying key policy and program implications

• supporting the dissemination of findings to key decision makers, operational officers.

Role and functions of evaluation advisory or working groups

Evaluation advisory groups are time limited and formed to provide specific advice for an evaluation study. Having an evaluation advisory or working group is important where there are multiple agencies and stakeholders involved and when sector specific advice will be needed to refine data collection methods and engage stakeholders in the evaluation. Advisory groups are also useful in these circumstances because they are able to provide in-depth knowledge about the program or policy being implemented. During the implementation of an evaluation the usual roles and functions of evaluation advisory groups encompass providing advice about:

• planning and implementing specific evaluation activities

• how, and by whom, issues impacting on the implementation of the evaluation should be raised, escalated and resolved

42 SAFEWORK NSW

• where or from whom to get sector specific advice where the expertise is not available from within the advisory group

• how to communicate with stakeholders about the evaluation and protocols for doing so

• the accuracy of findings, evaluation reports and other products

• how to interpret and contextualise findings

• how to disseminate the evaluation findings for program or policy improvement and accountability.

Recommendation for the Roadmap: Evaluation advisory groups are established for Tier 3 and Tier 4 level evaluation studies (section 5.4) with terms of reference that clearly communicate the role of the group. These advisory groups would be drawn from and sit under the relevant program governance committees. Additional members could include relevant SafeWork operational level staff and representatives of key peak bodies, associations and community leaders. Alternatively, input into evaluation design and considerations of the implications of findings be sought from external stakeholders via existing engagement structures.

5.2. Directorate-level annual evaluation planningEach Directorate should plan evaluation studies as part of the annual cycle of business planning. Such planning would involve identifying and proposing evaluation studies for the following year and how the results will be used to inform decision-making about service improvement and expansion and/or continuation of initiatives.

Directorate annual evaluation planning should also identify which external partners will be involved in the planned evaluations and the manner of that involvement. Ways that external partners could get involved include, as:

• members of advisory groups for evaluation studies and

• key informants about the implementation of initiatives.

5.3. Individual project evaluation plansAt the individual project level, all registered projects plans should be inclusive of evaluation; that is the project plan should identify the evaluation tier, methods and oversight.

5.4. Decision support toolIt is not feasible, cost effective or appropriate to fully evaluate all initiatives under the Roadmap. In recognition of this, a decision tool has been developed, ‘Roadmap Evaluation Priority Levels’ (Table 8). This tool is intended to assist SafeWork to make decisions about what kind of evaluation is needed and how to prioritise budget resources. The tool describes four tiers of evaluation with 1 being a very simple program or change in service and 4 representing a substantial investment in an action area under the Roadmap. The approach to evaluation is described for each level. The four tiers are a guide to assist SafeWork directorates to scale evaluation planning and for SafeWork executive to allocate resources for evaluation.

Each Tier 2, 3 and 4 evaluation study must have an evaluation and data collection plan that sets the purpose, stages, methods and timeline for the evaluation, and who will be responsible for specific tasks. Project managers will develop evaluation plans in conjunction with or using guidance from Business Performance staff.


Tab

le 8

: Th

e R

oad

map

Eva

luat

ion

Pri

ori

ty L

evel

s5

Tier

Typ

e o

f p

roje

ctE

valu

atio

n ap

pro

ach

Ove

rsig

htE

valu

atio

n au

die

nce

and

co

mm

unic

atio

n

1M

ino

r to

mo

der

ate

enh

ance

men

ts

to c

ore

ser

vice

wit

h a

fo

cus

on

se

rvic

e im

pro

vem

ent*

.

No

or

very

sm

all b

ud

get

ary

inve

stm

ent

in c

han

ge.

Inte

rnal

sta

keh

old

ers

on

ly.

No

str

ateg

ic o

r ac

cou

nta

bili

ty

imp

licat

ion

s u

nd

er t

he

Ro

adm

ap.

• Id

enti

fied

in a

nn

ual

bu

sin

ess

pla

n a

nd

in a

pp

rove

d p

roje

ct p

lan

s.

• P

roce

ss e

valu

atio

n, l

imit

ed s

cop

e.

• R

eso

urc

ed f

rom

wit

hin

bu

sin

ess

bu

dg

et.

Pro

gra

m m

anag

er a

nd

/o

r ev

alu

ato

r.R

epo

rtin

g w

ith

in

Dir

ecto

rate

.

2S

ub

stan

tive

ch

ang

e to

co

re s

ervi

ce

un

der

th

e R

oad

map

or

pro

ject

.

New

ap

pro

ach

or

pilo

t in

itia

tive

.

Rel

ativ

ely

low

inve

stm

ent

nee

ded

fo

r in

itia

tive

to

be

del

iver

ed.

Ext

ern

al s

take

ho

lder

s co

nsu

lted

.

Co

ntr

ibu

tes

to a

per

form

ance

m

easu

re.

• Id

enti

fied

in a

nn

ual

pla

n a

nd

in a

pp

rove

d p

roje

ct p

lan

s.

• E

valu

atio

n f

ocu

s: p

roce

ss, r

each

an

d im

med

iate

ou

tco

mes

.

• R

eso

urc

ed f

rom

wit

hin

th

e b

usi

nes

s b

ud

get

.

• P

roce

ss e

valu

atio

n fi

nd

ing

s av

aila

ble

wit

hin

12

mo

nth

s an

d s

tud

y is

co

mp

lete

d w

ith

in t

wo

-yea

r ev

alu

atio

n c

ycle

.

Rel

evan

t D

irec

tor.

Co

nsi

der

ad

vice

fro

m

Bu

sin

ess

Per

form

ance

o

n e

valu

atio

n d

esig

n.

Rep

ort

ing

wit

hin

D

irec

tora

te.

Rep

ort

may

be

circ

ula

ted

to

inte

rnal

an

d r

elev

ant

exte

rnal

st

akeh

old

ers.

3S

trat

egic

ally

sig

nifi

can

t p

roje

cts.

Mo

der

ate

inve

stm

ent.

Ext

ern

al s

take

ho

lder

s co

nsu

lted

or

par

tner

s in

th

e in

itia

tive

.

• S

tatu

s id

enti

fied

in a

nn

ual

eva

luat

ion

pla

n a

nd

in a

pp

rove

d

pro

ject

pla

ns.

• P

roje

ct p

lan

incl

usi

ve o

f ev

alu

atio

n p

lan

; sta

ged

imp

lem

enta

tio

n.

• E

valu

atio

n f

ocu

s o

n p

roce

ss, r

each

, im

med

iate

ou

tco

mes

an

d

inte

rmed

iate

ou

tco

mes

.

• M

eth

od

s m

ay in

clu

de

colle

ctio

n o

f b

asel

ine

dat

a an

d o

utc

om

es

ove

r ti

me.

• F

ind

ing

s av

aila

ble

aft

er e

ach

sta

ge

of

eval

uat

ion

.

• A

n a

lloca

ted

bu

dg

et/r

eso

urc

es t

o c

om

ple

te e

valu

atio

n.

• C

on

sid

er in

dep

end

ent

eval

uat

or

or

pee

r re

view

.

• F

ull

eval

uat

ion

rep

ort

.

Saf

eWo

rk e

xecu

tive

or

rele

van

t G

ove

rnan

ce

Co

mm

itte

e m

ust

en

do

rse

eval

uat

ion

pla

n.

Pro

gra

m g

ove

rnan

ce

com

mit

tee

advi

sory

ro

le.

See

k ad

vice

fro

m

Bu

sin

ess

Per

form

ance

o

n e

valu

atio

n d

esig

n.

Rep

ort

pu

blic

ly

avai

lab

le.

5 T

iers

dev

elo

pe

d f

rom

NS

W G

ove

rnm

ent

Eva

luat

ion

Fra

mew

ork

, 20

16.

http://www.dpc.nsw.gov.au/programs_and_services/policy_makers_toolkit/evaluation_in_the_nsw_government

44 SAFEWORK NSW

Tier

Typ

e o

f p

roje

ctE

valu

atio

n ap

pro

ach

Ove

rsig

htE

valu

atio

n au

die

nce

and

co

mm

unic

atio

n

4S

ign

ifica

nt

inve

stm

ent

in a

ctio

n a

rea

(rel

ativ

e to

SW

NS

W).

Dir

ect

invo

lvem

ent

of

par

tner

s b

oth

g

ove

rnm

ent

and

ind

ust

ry.

Hig

h p

rofi

le.

Fin

anci

al o

r o

rgan

isat

ion

al

imp

licat

ion

s.

• C

om

pre

hen

sive

eva

luat

ion

str

ateg

y, f

ocu

s o

n o

utc

om

es a

nd

im

pac

ts.

• M

eth

od

s m

ust

incl

ud

e co

llect

ion

of

bas

elin

e d

ata

and

ou

tco

mes

o

ver

tim

e.

• S

tag

ed d

eliv

ery

of

fin

din

gs.

• In

dep

end

ent

eval

uat

or

or

pee

r re

view

.

• A

n a

lloca

ted

bu

dg

et/r

eso

urc

es t

o c

om

ple

te e

valu

atio

n.

• F

ull

eval

uat

ion

rep

ort

.

Saf

eWo

rk e

xecu

tive

or

rele

van

t G

ove

rnan

ce

Co

mm

itte

e m

ust

en

do

rse

eval

uat

ion

pla

n.

Pro

gra

m g

ove

rnan

ce

com

mit

tee

advi

sory

ro

le.

Rep

ort

pu

blic

ly

avai

lab

le a

nd

sen

t to

M

inis

ter.

No

tes:

*C

ore

ser

vice

s ar

e as

sist

ance

ser

vice

s, p

ub

lic a

war

enes

s (e

xclu

din

g m

ajo

r m

edia

cam

pai

gn

s), c

on

sult

ativ

e m

ech

anis

ms,

pre

ven

tio

n p

rog

ram

s, in

form

atio

n

and

ad

vice

ser

vice

s, fi

nan

cial

ince

nti

ves,

san

ctio

ns,

tra

nsp

aren

cy a

nd

acc

ou

nta

bili

ty, r

egu

lato

ry r

esp

on

se, a

uth

ori

sati

on

s, v

erifi

cati

on

s an

d a

ud

its.


Relationship to SafeWork’s Project Prioritisation Tool

To assist in decision-making during the business planning phase as to which projects shall proceed, SafeWork has developed a Project Prioritisation Tool. The tool comprises six criteria that are applied to each potential project to generate a rating against each criteria and an overall rating score. The criteria are harm prevention (frequency of incidents); harm prevention (injury severity); strategic alignment; economic impact; customer reach; and operational excellence. The projects will then be given a ranking that corresponds to the relative scores of all projects.

The Project Prioritisation Tool and Evaluation Priority Tiers have different purposes. The Project Prioritisation Tool is used to select from all potential projects. The Evaluation Priority Tiers consider the evaluability of a project, the nature of the service innovation or project and the size of investment in innovation.

5.5. Evaluation checklistThe checklist below (Figure 5) has been designed to assist SafeWork staff members when planning evaluations.

Figure 5: Evaluation checklist

Project or initiative name

Purpose statement of project/program

Why do you want to do an evaluation

What questions are you seeking answers to

Proposed methodology and data sources, consider feasibility and cost

Proposed target group

Source of budget

Proposed budget

Involvement of external partners and who

46 SAFEWORK NSW

5.6. Methodological approachesAny particular approach to an evaluation will depend on what is considered credible, feasible, useful and possible, given the nature of the initiative being evaluated, the information needs of the project team, senior managers and corporate, and the available skills and resources. The methodological approach chosen will coincide with the evaluation tier the study has been placed in.

Being explicit about the reason/s for evaluating your initiative helps decide what data to collect and when. What are you seeking to do?

• Monitor how much is being done and how well

• Improve your service/product/project

• Assess outcomes/benefits/impacts.

One of the biggest risks in impact evaluation is getting a precise but inaccurate answer about the impact of an initiative or project. This often leads to decisions based on a false sense of certainty where a less ambitious approach may lead to more accurate answers about a more limited range of impacts. This risk is highest when only one method is used, even if this method is considered robust. Mixed methods are the best way to reduce this risk. For example, professionally gathered stories can assist in explanations of what hard data means and help understand the contexts in which a particular initiative was effective and through what mechanisms, such that past successes may be replicated in future contexts.

For many Roadmap evaluations, the use of mixed methods will be suitable, drawing on both qualitative and quantitative data. Evidence from different sources can be triangulated to support or qualify findings. Evaluation plans should ensure the use of the most robust and credible evidence that is available.

Appropriate and feasible data collection methods should be used to collect monitoring and evaluation data that is not available through existing sources. This level of detail, along with sampling frames and timelines, will need to be developed in the evaluation plans for each project, based on the program logic developed for that project.

The table on page 47 provides an overview of key data collection methods that are commonly used in a mixed methods evaluation.


Tab

le 9

: Co

mm

on

ly u

sed

met

ho

ds

in e

valu

atio

n6

Met

hod

Stre

ngth

sW

eakn

esse

sC

ons

ider

atio

ns

Sur

vey

Can

gen

erat

e a

larg

e am

oun

t o

f q

uant

itativ

e an

d

qua

litat

ive

dat

a.D

oes

no

t al

low

fo

r in

tera

ctio

n o

r p

rob

ing

of

resp

ons

es. S

urve

ys a

re b

eco

min

g u

biq

uito

us

and

may

no

t b

e su

ffici

ently

sal

ient

to

ob

tain

su

ffici

ent

resp

ons

e ra

tes

to b

e us

eful

. S

elf-

rep

ort

ed c

hang

es.

Mus

t b

e d

evel

op

ed a

nd t

este

d t

o

ensu

re q

uest

ions

are

co

mp

rehe

nsiv

e an

d u

nam

big

uous

and

sam

ple

is

rep

rese

ntat

ive

of

the

po

pul

atio

n

of

inte

rest

.

Fo

cus

gro

ups

Allo

w f

or

exp

lora

tion

of

key

issu

es w

hen

they

are

re

lativ

ely

com

ple

x o

r no

t to

p-o

f-m

ind

fo

r re

spo

nden

ts.

Pro

vid

e re

spo

nden

ts w

ith o

pp

ort

unit

y fo

r le

arni

ng f

rom

o

ther

s fir

st-h

and

.

Diffi

cult

to a

naly

se in

div

idua

l res

po

nses

and

no

t su

ited

to

qua

ntita

tive

anal

ysis

.R

equi

re a

ski

lled

fac

ilita

tor

and

the

ab

ility

to

bri

ng p

artic

ipan

ts t

og

ethe

r.

Wo

rksh

op

sC

an d

raw

on

colle

ctiv

e kn

owle

dg

e o

f d

ecis

ion

mak

ers

to in

terp

ret

evid

ence

and

gen

erat

e re

leva

nt a

nd u

sefu

l im

plic

atio

ns a

nd fi

ndin

gs.

The

util

isat

ion

focu

s m

eans

the

y m

ay n

ot

wel

l su

ited

fo

r p

rim

ary

dat

a co

llect

ion

.T

he im

po

rtan

t d

ecis

ion

mak

ers

or

thei

r vi

ews

mus

t b

e re

pre

sent

ed in

th

e w

ork

sho

p.

Inte

rvie

ws

Go

od

fo

r g

athe

ring

in-d

epth

info

rmat

ion

fr

om

res

po

nden

ts w

ho u

nder

stan

d a

n is

sue

wel

l. C

an in

clud

e b

oth

qua

ntita

tive

and

qua

litat

ive

dat

a.

May

no

t d

raw

out

new

idea

s o

r b

e fe

asib

le t

o

cond

uct

eno

ugh

for

a re

pre

sent

ativ

e sa

mp

le.

Can

be

cond

ucte

d f

ace-

to-f

ace

or

on

th

e p

hone

.

Cas

e st

udie

sC

an e

ffec

tivel

y ill

ustr

ate

an is

sue,

pro

mo

te a

pra

ctic

e, o

r ex

pla

in a

co

mp

lex

phe

nom

eno

n in

a s

pec

ific

cont

ext.

Cas

es m

ay n

ot

be

rep

rese

ntat

ive

even

if t

he

find

ing

s m

ay b

e us

eful

in a

ran

ge

of

cont

exts

.O

ften

use

d a

s p

art

of

a b

road

er

eval

uatio

n.

Med

ia a

naly

sis

May

pro

vid

e p

olic

y m

aker

s w

ith in

sig

ht in

to m

edia

co

vera

ge

of

po

litic

ally

sen

sitiv

e is

sues

.M

ay r

eflec

t m

edia

rat

her

than

key

sta

keho

lder

vi

ews

on

an is

sue.

New

s m

edia

is r

elat

ivel

y ea

sy a

nd

chea

p t

o o

bta

in.

Long

itud

inal

stu

die

sW

here

ind

ivid

uals

are

fo

llow

ed o

ver

time

to d

isco

ver

whe

ther

cha

nges

in, f

or

exam

ple

exp

osu

re t

o r

isk,

are

d

ue t

o a

sp

ecifi

c in

terv

entio

n.

Can

but

gen

eral

ly d

o n

ot

cons

ider

the

co

unte

r-fa

ctua

l, ie

– d

o n

ot

follo

w a

co

ntro

l g

roup

to

ass

ess

whe

ther

the

re a

re o

ther

ex

pla

natio

ns f

or

chan

ge.

Req

uire

bas

elin

e d

ata

to b

e co

llect

ed

bef

ore

pro

ject

is im

ple

men

ted

.

Qua

si-e

xper

imen

tal

des

igns

In

volv

e a

com

par

iso

n g

roup

. Are

use

d t

o m

easu

re

imp

act

whe

n ra

ndo

m a

lloca

tion

to t

reat

men

t o

r co

ntro

l g

roup

s is

no

t p

oss

ible

.

Sub

ject

s se

lf-se

lect

into

gro

ups,

whi

ch c

an

intr

od

uce

bia

s.R

equi

re p

roce

sses

fo

r ch

ecki

ng

inte

rven

tion

and

co

ntro

l gro

ups

are

equi

vale

nt p

rio

r to

an

inte

rven

tion

.

Two

use

ful r

efer

ence

s th

at d

etai

l po

ten

tial

met

ho

do

log

ies

and

wh

en t

o a

pp

ly t

hem

are

:

• R

efer

ence

Gu

ide

to E

valu

atin

g t

he

Eff

ecti

ven

ess

of

Str

ateg

ies

for

Pre

vent

ing

Wo

rk In

juri

es: H

ow

to

Sh

ow

Wh

eth

er a

Saf

ety

Inte

rven

tio

n R

eally

Wo

rks.

Ly

nd

a S

. Ro

bso

n, H

arry

S. S

han

no

n, L

ind

a M

. Go

lden

har

, An

dre

w R

. Hal

e. D

epar

tmen

t O

f H

ealt

h a

nd

Hu

man

Ser

vice

s, P

ub

lic H

ealt

h S

ervi

ce, C

ente

r fo

r D

isea

se C

on

tro

l an

d P

reve

nti

on

, Nat

ion

al In

stit

ute

fo

r O

ccu

pat

ion

al S

afet

y an

d H

ealt

h. A

pri

l 20

00

.

• A

RT

D R

epo

rt f

or

Wo

rk H

ealt

h a

nd

Saf

ety

Div

isio

n, W

ork

Co

ver

NS

W. C

ont

emp

ora

ry a

pp

roac

hes

an

d m

eth

od

s fo

r co

st e

ffec

tive

eva

luat

ion

. 20

15.

6

So

urc

es:

AR

TD

Re

po

rt f

or

Wo

rk H

eal

th a

nd

Saf

ety

Div

isio

n, W

ork

Co

ver

NS

W. C

on

tem

po

rary

ap

pro

ach

es a

nd

met

ho

ds

for

cost

eff

ect

ive

eval

uat

ion.

13

Fe

bru

ary

20

15. S

criv

en

, Mic

hae

l (19

91)

. Eva

luat

ion

Th

esau

rus.

4

th E

dit

ion

. Sag

e P

ub

licat

ion

s.

48 SAFEWORK NSW

6. Understanding, using and reporting evaluation findings

The main product of an evaluation is some form of report that is communicated to the intended audiences and used in line with the purpose. Likely audiences are NSW Government; SafeWork Executive, managers, operational staff; and external partners, for example regulatory agencies and industry partners.

6.1. Using evaluation results for service and product improvementOne of the fundamental reasons for evaluation is for improvement purposes. The results of evaluation studies should be used to improve services, products, harm prevention approaches and regulatory initiatives. As shown in the decision support tool, planning an evaluation also includes being explicit about how the evaluation will inform strategic and operational considerations about where directorates should direct efforts and resources in the future. Having appropriate governance structures in place will ensure that evaluation findings reach key decision makers and feed into other governance arrangements supporting the implementation of the Roadmap.

When planning how evaluation findings will be used, such a plan could include the following elements.7

• Plan to form an evaluation advisory group (see section 5.1), which includes both producers of evaluation findings and potential users of the findings, for example relevant operational staff and their managers, the evaluators and key external partners. The group should be used to discuss the implications of the interim findings for service improvement and produce recommendations to inform the development of annual business plans.

• Plan to present findings to the appropriate program governance committee to ensure the findings reach key decision makers and feed into other governance arrangements supporting the implementation of the Roadmap.

• Plan the timing of reporting of evaluation findings to coincide with business planning cycles and with the two-year review of the Roadmap.

• Identify how the report’s findings will be communicated to different stakeholders/audiences (see section 6.2).

6.2. Communication through evaluation reportsTo produce useful evaluation reports you should start planning the report early, including consulting communications professionals within SafeWork to seek advice on how to promote the findings to the different audiences.

The usefulness of a report depends upon its perceived credibility and how it is promoted. When promoting the report consideration should be given to the ‘What’s in it for me?’ principle from the perspective of the intended audience. This is particularly important for external partners who want reports that use plain English and that can be readily understood and distributed to their peers or members.8

Evaluation is a cycle, and it is important to complete the cycle of evaluation by reporting and using findings. This will keep intended audiences informed, assist managers to discuss findings with staff and consider the implications of the findings. Reports will also inform decision-making about the direction projects should take, for example about the:

• effectiveness of a risk mitigation or compliance strategy

• how to improve the effectiveness of delivery of your activities in order to improve results

• how to modify the project to more accurately target desired outcomes.

7 Grimshaw, Jeremy M., Eccles, Martin P., Lavis, John N., Hill, Sophie J., and Squires, Janet E. (2012). Knowledge translation of research findings. Implementation Science 127:50. and, Green, L.W., Ottoson, J.M., García, C., Hiatt, R.A., and Roditis, M.L.(2014). Diffusion theory and knowledge dissemination, utilization and integration. Frontiers in Public Health Services and Systems Research. 3(1).

8 Feedback from consultation interviews conducted by ARTD in February 2017.


6.3. Analysis of attribution and causal contributionOne of the things to consider in impact evaluations is how what you have implemented has caused things to change.

An outcomes evaluation is generally concerned not just with measured changes but ways in which a given program has contributed to those changes, and the influence of other factors. For the Roadmap, the achievement of shared outcomes is expected to be through the contribution of multiple interventions (such as those described in sector plans) across the organisation. Other factors could be the influence of social and economic changes and the direct actions of social partners.

The most rigorous methods for demonstrating program impact involve a counterfactual, but in the right circumstances – when it is feasible and ethical to have some form of valid comparison.

Contribution analysis (Mayne, 2012) explores causality in situations where experimental designs are not feasible, based on the program’s proposed theory of change/program logic and an analysis of this theory on the available evidence, including other influencing factors. There are six steps in this process.

1. Set out the cause-effect issue to be addressed.

2. Develop the postulated theory of change and risks to it, including rival explanations.

3. Gather the existing evidence on the theory of change.

4. Assemble and assess the contribution claim, and challenges to it.

5. Seek out additional evidence.

6. Revise and strengthen the contribution story.

Contribution analysis offers a practical approach to help assess whether a given intervention has contributed to an observed result.9 From a process of reasoned analysis, it is possible to infer whether a project is the most likely contributor to observed changes.

Causality is inferred from a four-step logic and evidence,10 outlined as follows:

• Step 1 – The intervention is based on a reasoned theory of change: the program logic and assumptions behind it are plausible, sound, informed by the literature and supported by key stakeholders.

• Step 2 – The activities of the intervention were implemented as outlined in the program logic.

• Step 3 – The theory of change is verified by evidence: the chain of expected results occurred and assumptions held.

• Step 4 – External factors (context and rival explanations) are assessed and shown not to have made a significant contribution or, if they did, their relative contribution is recognised.

Contribution analysis enables investigators to reach credible conclusions about the causal relationship between a project and its outcomes.

9 Better Evaluation, 2013, Approaches, http://betterevaluation.org/approaches

10 Mayne, J. 2012, Contribution analysis: Coming of Age? http://evi.sagepub.com/content/18/3/270

http://betterevaluation.org/approaches

50 SAFEWORK NSW

6.4. Interpreting trends in lag indicators (long term policy outcomes)Any long term trends that are found in the lag indicators in the data (for example, reductions in injury rates, reductions in total losses paid or reductions in premiums charged based upon prior loss experience) must be interpreted with great sensitivity for the possibility of independent intervening variables that impact upon the outcome.

First, Australia specifically, and much of the rest of the developed world, is experiencing reductions in injury rates independent of the initiatives of this project.11 There has been much speculation about the cause of this reduction in reported accidents. Changes in the nature of work (such as more automation of dangerous work), the mix of work (such as a shift from manufacturing to less dangerous service provision as the mainstay of the economy), the nature of the workforce (such as more people working as contractors instead of employees, resulting in accidents often not being reported), and overall safety/liability consciousness on the part of tool and equipment manufacturers are some of the reason that are often cited.

Second, it is extraordinarily difficult to tease out the relative contributions to data trends created by one program in an environment where many programs are aimed at the reduction of injury. For instance, Safe Work Australia has made headway with public awareness of its national safety standard initiatives. Similarly, some impact has been felt from changes in director liability laws that require corporate boards to be more proactive and responsive to safety issues. Unions have also been active in the safety space, with at least one national union airing a series of television advertisements that feature the union's role in safety.

Third, individual compliance rates with any governmental initiative are influenced by a wide variety of factors that are less attributable to the content of the program than the manner in which the program is presented to the public.12 It is very difficult to separate the program content from the presentation of the content in ascertaining the impact of a program. That being said, the difference may not be one that is important from the perspective of achievement of the overall intention of the initiative, even if it impacts the assessment of the effectiveness of initiative components. For example, compliance activity may be the primary cause of behaviour change, or increased safety awareness brought about in part by compliance efforts may cause the observed change.

Finally, lag indicator trends must be viewed in the context of the larger environment. For instance, while injury rates in Australia are falling in many categories, the frequency of reported mental/psychological claims is up and the total expenditures for claims losses are also up.13 This presents a challenge for the delineation of the proper industry and harm categories for focus during the initiative.

Accurate and credible interpretation of trends in lag indicators can be advanced, although not entirely perfected, by analysis of all major known factors in the environment. The analysis, to be credible, must be transparent and as thorough as practicable, given resource and information limitations.

11 http://www.safeworkaustralia.gov.au/sites/SWA/Statistics/Documents/Table-1-1-number-frequency-rate-and-incidence-rate-of-serious-claims-by-occupation-2009-10-2013-14.pdf

12 Thaler, R. and Sunstein, C., Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin Group, N.Y., N.Y. 2009.

13 Comparison of Safe Work Australia reports for 2012–2013 (total cost $61.8 Billion) with the same report for 2008-2009 (total cost $34.1 Billion).

http://www.safeworkaustralia.gov.au/sites/SWA/Statistics/Documents/Table-1-1-number-frequency-rate-a

http://www.safeworkaustralia.gov.au/sites/SWA/Statistics/Documents/Table-1-1-number-frequency-rate-a

SW08641 0817