How do you solve a problem like Impact? Summary of survey findings

Paul Redmond

Head of the Careers & Employability Service, University of Liverpool

Overview

• 46 completers (82% heads of service with between 4 – 7 years’ service).

• Majority (85%) located in student service-related department.

• 11% in a teaching-related department• 3.8% in an external-relations department• Spread of universities (post-1992, RG, etc.)

Reporting structure

• 70% of heads report to Head of Student Services

• 20% report to PVC• Academic deans / heads of T&L = 6.8%

respectively

Defining your Service’s purpose

• Providing services to students – nearly all cite ‘employability’ services to students as the primary goal, e.g.

• “Improving the employability of students and graduates and providing a central point of contact to support CEIAG across the campus …”

Defining your Service’s purpose

• Providing services to students – nearly all cite ‘employability’ services to students as the primary goal, e.g.

• “Improving the employability of students and graduates and providing a central point of contact to support CEIAG across the campus as well as delivering centrally …”

Rank your main stakeholders

1st 2nd 3rd 4th 5th

Students 93% (53) 7% (4) 0 0 0

Graduates 0 57.4% (31) 9.3% (5) 14.8% (8) 18.5% (10)

Employers 0 19.6% (11) 44.6% (25) 28.6% (16) 7.1% (4)

Academics 0 9.1% (5) 36.4% (20) 29.1% (16) 25.5% (14)

University managers

7.1% 8.9% (5) 10.7% (6) 28.6% (16) 44.7% (25)

How do you or your HEI measure the responsiveness, profile and visibility of your service with the above groups?

• Mixture of internal and external feedback measures• Student numbers / record of users• Referrals from academics• Profile with senior managers• DLHE – institutional comparators (ePI)• Student satisfaction / user surveys (various)• League tables• Involvement of CES in key projects /initiatives• Access to ‘prized’ networks• Employer evaluations• Hits on website• ‘Profile and visibility’ …• Matrix

How do you record and track usage of your Service?

• Central recording systems• On-line systems (e.g. Interfase)• ‘Numbers attending events’ records• Mixture of qualitative and quantitative research –

headcounts to ‘mystery shoppers’• We don’t!• Basic headcounts• Monthly MIS surveys / reports• Interview stats• Hits on website• Twitter traffic …

How do you measure your Service’s impact on students?

Numbers Satisfaction Impact Don’t measure

Curriculum based group work

82.1% (46) 85.7% (48) 28.6% (16) 1.8% (1)

Centrally based group work

81.8% (45) 90.9% (50) 20% (11) 1.8% (1)

One to one discussions

84.2% (48) 89.5% (51) 36.8%(21) 0

Online / self help 72.2% (39) 63% (34) 18.5% (10) 13% (7)

How do you measure your impact with employers?

Numbers Satisfaction Impact Don’t measure

Employer vacancy handling

86.8% (46) 62.3% (33) 30.2% (16) 7.5% (4)

Employer fairs / presentations / events on and off campus

86.8% (46) 94.3% (50) 18.9% (10) 1.9% (1)

Employer input into group work with students

66.7% (34) 64.7% (33) 13.7% (7) 19.6% (10)

Facilitating employer links with academic departments

41.2% (21) 39.2% (2) 23.5% (12) 43.1% (22)

Consultancy and other tailored services for employers

34% (16) 36.2% (17) 21.3% (10) 55.3% (26)

Other ways in which you measure your impact on employers

• None (several)• Numbers of vacancies filled / quantity and quality of

applications• Employer advisory board• Repeat business• Direct feedback from employers in relation to vacancies

and internships

Do you or your institution measure the impact of your service or your INSTITUTION’S REPUTATION AND SUCCESS on:

0.0% 20.0% 40.0% 60.0% 80.0%

Return on investment and value added

Income generation

Efficient use of resources, including staff time (VFM)

Contribution to professional development of academic staff

Contribution to student retention / progression

Contribution to improving graduate destinations

66.7%

35.3%

23.5%

52.0%

44.0%

27.1%

31.4%

54.9%

68.6%

40.0%

50.0%

50.0%

2.0%

9.8%

7.8%

8.0%

6.0%

22.9%

Yes

No

Don't know

How your careers service impacts on your institution’s reputation and success

• Provides comparative data on graduate employment (shows where HEI stands in league tables)

• Success in generating funded projects• Income generation• League Tables / DLHE – “Seen as our most important

function by senior management, sadly.”)• KPI’s in reports to management• Head produces full-cost recovery study to gauge VfM• Contribution to student retention• “Return on Investment … not sure about how this might

be specifically measured …”• “Nothing too explicit ..” … “None.”

How highly are you perceived in your HEI?

What KPIs and / or targets does your Service have?

• DLHE – top listed KPI• Employability P.I.• No’s of finalists / graduates seen• Increased numbers using the Service• Student satisfaction surveys• Listed in ‘Top 10’ surveys• Personal recommendation• No. of placements achieved• Matrix• Mixture of externally imposed and internally verified

Final comments …

• “I am still at a loss as to how to establish meaningful and real impact measures for most of the work we are engaged in, despite working at the problem for many years! Fortunately, I am rarely required to justify our impact.”

• “Be careful of using DLHE as a KPI or measure of impact.”

• “Our work does not necessarily have immediate effect – it may take many weeks, months or years for a client to really take on advice. How do we measure the impact of guidance?”

• “Collective approach from AGCAS in addressing this is very much welcomed.”

Reflection

• The Impact Paradox: By prioritising overwhelmingly services to students, which produce intangible outcomes, careers services find it difficult to measure impact. Those services which can be measured are often not those viewed as key priorities by services.

• Location, Location, Location: How crucial is institutional positioning?

• Across the sector, wide variations exist (and varying levels of measurement sophistication).

• But impact measurement is becoming a widely accepted issue of importance for most careers services. Nevertheless, methodologies remain limited.