practice placement quality assurance (ppqa) website ... student evaluation... · practice placement...
TRANSCRIPT
Practice Placement Quality Assurance (PPQA) website: Student Evaluation Questions
A review update >>>>>>> Fiona Bates Quality Assurance Systems Project Manager 7th October 2013
PPQA: Student evaluation review Page 2 of 17 7th October 2013
Contents
Please click on the section title to access the information required
Section Page
Management Summary 3
Background 4
Outcome of the 19th June workshop 5
Student Survey: Data Analysis 7
Recommendations 11
Appendices
Appendix 1: Proposed Student Evaluation Questions 12
Appendix 2: Invitation to Students 14
Appendix 3: Student Survey Data 15
Acknowledgements 17
PPQA: Student evaluation review Page 3 of 17 7th October 2013
Management Summary
Background
The Practice Placement Quality Assurance (PPQA) website www.healthcareplacements.co.uk was conceived initially in 2004 as an evaluation tool for healthcare students to complete an online evaluation following each practice placement. The website now includes a lot more quality assurance functionality, including placement profile and audit. Requests to review the student evaluations questions have been expressed this year by stakeholders across the region. A workshop was arranged for the morning of 19
th June to review the student
evaluation questions, and responses, and propose alternatives. The outcome of the workshop and the subsequent consultation activities, including a student online survey are included in this report. At the workshop the purpose and format of the questionnaire was agreed. Linking student names to their comments was also confirmed. The proposed questions were refined via email and consultation during June and July and an online survey was developed to obtain student feedback on the proposed evaluation questions (see Appendix 1). An invitation to the students was issued in August for return by 30
th September (see Appendix 2).
Student Survey: Data Analysis 294 students complete the online survey. During the survey they were asked two questions about each of the proposed questions in Appendix 1 as follows:
A. Do you think Question X is clear and unambiguous? B. Do you think a Yes, No (and in some case N/A (see Appendix 1)) response is appropriate for
question X? C.
The responses were rated using a red, amber, green scale (see Appendix 3). The comments for the 11 proposed questions with an amber or red rating for question A are summarised in the report and included points about multiple questions and suggestions for alternatives. Comments specifically related to the actual responses (Y,N,N/A) in Question B were not significant. A number of students requested the ability to leave a comment, which is already included in the software, and others referred to the actual question (similar responses to Question A). The students were also asked a final question to establish if there was anything not covered by the 25 questions.
Recommendations
The recommendations are as follows:
The purpose of the questionnaire should be advertised to students by HEI staff and also on the PPQA website. Timescales to be decided by the Practice Placement Quality Working Group (PPQWG).
The proposed student questions with a green rating in Appendix 3 for Question A are agreed by the PPQWG as valid and will be included on the PPQA website when the student evaluation area is updated.
The proposed student questions which are listed in “Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A” should be reviewed to take into account the comments received by the students on each question and also the final question comments. A half day workshop is suggested, involving the individuals who contributed to the 19
th June workshop.
Contents
PPQA: Student evaluation review Page 4 of 17 7th October 2013
Background
The Practice Placement Quality Assurance (PPQA) website www.healthcareplacements.co.uk was conceived initially in 2004 as an evaluation tool for healthcare students to complete an online evaluation following each practice placement. The website now includes the following functionality:
Student and Mentor Evaluations
Mentor/Practice Educator Register
Practice Placement Profile and Audit
Student Capacity information
Documentation area
The student evaluation questions were reviewed by the Regional Educational Audit Tool (REAT) sub group in 2010 and the mentor / practice educator (PE) evaluation questions were also developed at this point. The software was updated in January 2011 to include the mentor evaluation functionality and also the ability for students and mentors to leave comments about each question they answered. It was compulsory to leave a comment if the response was negative and the names were linked to the comments. The actual questions for the students changed at the end of 2010/11 academic year as this was felt easier to manage.
Requests to review each of the student evaluation questions have been expressed this year by members of the Practice Placement Quality Working Group (PPQWG), Higher Education Institution (HEI) staff, Practice Learning Facilitators and via the online surveys undertaken as part of the PPQA review (reported in June 2013). The issue of comments being attributable to individual students was also being raised.
A workshop was arranged for the morning of 19th June to review student anonymity, student
evaluation questions, and responses, and propose alternatives. The outcome of the workshop, including the purpose of the student evaluations, and the subsequent consultation activities, including a student online survey, are contained within this report.
Contents
PPQA: Student evaluation review Page 5 of 17 7th October 2013
Outcome of the 19th June workshop
There were 22 delegates at the workshop. The issue of anonymity was discussed prior to reviewing the questions and it was agreed that the student names should be linked to the student comments because of the following:
The responses have never been completely anonymous and it is therefore unfair on students to give the impression they are anonymous
Good practice indicates follow up to the responses with the students, lecturers or mentors; particularly if there are professional conduct issues
Students should be encouraged to take a professional responsibility for their comments
It is important to promote a culture of transparency with the students, particularly in light of the Francis Report.
Purpose of the student questionnaire The purpose of the questionnaire was agreed at the commencement of the workshop (see below). It is recommended this should be advertised to the students on the PPQA website in the area where they complete the questionnaire and by the Higher Education Institutions when preparing the students for placement.
“Feedback on your learning experience will contribute to the on-going evaluation of education within
the placement setting and support from your education provider.
You have a responsibility as a healthcare practitioner of the future to participate in the quality
assurance processes in an open and transparent manner.”
Sections and Responses
Sections It was agreed that although the students should not see the section titles the questions need to follow a logical sequence from start to finish of the placement. The following sections were agreed:
Preparation for practice experience.
Orientation
Learning Opportunities/Resources
Supervision/Support
Feedback/Assessment
Recommendations/Concerns
Responses It was agreed that the responses should be changed from the current “strongly agree” through to “strongly disagree”. The responses should be as follows:
Yes/No, and not applicable available for identified questions (see Appendix 1)
Comment compulsory for “No”.
No check box required re terms and conditions
Additional comment box required at the end of the questionnaire
“There is a responsibility on healthcare providers to deliver high quality
education and training, not just for their students, but for all their staff in order to
ensure high quality and safe patient care.” A mandate from the government to
Health Education England (2013)
PPQA: Student evaluation review Page 6 of 17 7th October 2013
Consultation
The PPQA Project Manager collated the questions developed during the workshop by the 5 work groups under the section headings above, avoiding duplication and ensuring the questions were generic and suitable for any placement or placement period. The results were circulated to the attendees of the workshop for comment and onward circulation as appropriate. A draft student online survey, using the Bristol Online Surveys Tool (BOS), was also circulated at the same time for comment and amendment. The feedback received from the workshop group, and their colleagues, was then used to amend the questions. All comments and changes were circulated again to the workshop group with the rationale for changes and omissions. As agreed at the workshop the proposed questions (see Appendix 1) were then included in the online survey to obtain the student feedback.
Online Survey to the Students An invitation (see Appendix 2) to complete the evaluation by 30
th September was issued on the 6
th
August to the PPQWG HEI representatives who then contacted the students. 294 students had completed the survey by the closing date. The results of the survey responses are included in the section Student Feedback: Data Analysis below.
Contents
PPQA: Student evaluation review Page 7 of 17 7th October 2013
Student Survey: Data Analysis
There were 294 responses to the online survey including students from 7 HEIs, plus the Yorkshire Ambulance Service, and 11 professions were represented (See Questions 1 and 2 below).
Question 1. Please select your education provider from the list below.*
University of Bradford:
25.2% 74
University of Huddersfield:
21.1% 62
Leeds Metropolitan University:
20.7% 61
University of Leeds:
19.4% 57
The Open University:
6.5% 19
York St John University:
4.8% 14
Yorkshire Ambulance Service:
2.4% 7
*No Sheffield Hallam University, University of Sheffield and University of York students completed the
survey.
Question 2. Please select your profession from the list below.**
Nursing:
55.1% 162
Dietetics:
11.2% 33
Podiatry:
6.8% 20
Midwifery:
6.5% 19
Physiotherapy:
5.8% 17
Occupational Therapy:
5.4% 16
Radiography:
3.7% 11
Paramedics:
2.4% 7
Speech and Language Therapy:
2.0% 6
Clinical Language Sciences:
1.0% 3
Speech and Language Therapy:
2.0% 6
**No audiology, operating department practitioners or radiotherapy students completed the survey.
The students were asked if the abbreviation of N/A for “Not Applicable” is universally understood. 263 (90%) felt it was whereas 31(10%) said no. Two comments really sum up why we should not use an abbreviation without explaining it first.
“All abbreviations
should be avoided
as in practice”
“Might it not be a good idea to
write Not Applicable (N/A)
once at the beginning to
facilitate both international
students and understanding
and use of the abbreviation.”
PPQA: Student evaluation review Page 8 of 17 7th October 2013
On subsequent pages the students were presented with each of the proposed 25 questions and asked the following two questions:
D. Do you think Question X is clear and unambiguous? E. Do you think a Yes, No (and in some case N/A (see Appendix 1)) response is appropriate for
question X? There were options of either a Yes/No response to both questions and if the response was “No” a comment was compulsory)
It had been suggested to the students in the introductory text that they should consider their most recent placement when responding to the survey. Although it was a minority, there were a few comments where the students answered the actual question under discussion rather than the two questions A and B above. This was clearly a fault of the introductory text and numbers have been identified in the table below. The data for the responses are included as Appendix 3 with the responses categorised using a Red, Amber, Green (RAG) rating, for example the red rating indicates over 10% of the students answered the question negatively (Amber = 5-10%), Green = <5%). The results can be summarised as follows:
Question A 3 8 14
Question B 6 16 3
Question A
The comments for Question A resulting in a red or amber rating are summarised in the following table:
Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A
No. Proposed Student Evaluation Question
Qu.A Comments summarised
N
3. The placement profile was informative.
44 15%
Don’t know what a profile is / don’t know where to find it /not sure what is being asked (17)
What is meant by informative? (4)
Comments re profiles not being helpful / limited information / out of date / incorrect / relevant reading not available / profile not available for placement (11)
Suggestions for alternatives (9)
Other (3)
23. I had concerns about patient care on my placement but these were dealt with appropriately
34 12%
Two questions in one i.e. concerns (y/n), dealt with appropriately (y/n) (8)
“No concerns” was the student response (this could be related to their placement or a suggestion for a change to the wording). (9)
Student referred to their actual placement (5)
“N/A” was the student response(4)
What does the question mean? / needs rewriting (6)
First year students may not have confidence to answer, responses should be anonymous (2)
24. I had concerns about my educational experience on my placement but these were dealt with appropriately
32 11%
Very similar responses to Question 23 above
7. There was sufficient access to internet IT facilities
23 8%
Access for student research or patient care? (3)
Suggestions for alternatives (7)
Difficulties accessing internet whilst on placement (13)
5. I had somewhere safe to store my belongings.
22 7%
Suggestions for alternatives (3)
Student referred to their actual placement (18)
PPQA: Student evaluation review Page 9 of 17 7th October 2013
Question irrelevant (1)
4. I had a planned, timely and comprehensive induction which orientated me to my environment and relevant policies and procedures, including health and safety information
17 6%
Too many areas covered in the question (10)
Induction on placement or by the university? (2)
Suggestions for alternatives (1)
N/A, member of the trust, other (4)
1. I received information and support from my education provider, which was appropriate for the stage of my programme, so that I was prepared for my placement.
19 6%
Three questions in one (6)
Badly written, muddled, doesn’t make sense had to read it twice (6)
Should be “..this stage…” (1)
Please define “support” is it learning or practical support (1)
Suggestions for alternatives (3)
Student referred to their actual placement (2) 12. I received timely support from
my course/programme team whilst on placement when I needed it
16 5%
Suggestions for alternatives (5)
Ambiguity / vague (7)
Student referred to their actual placement (4)
13. The learning needs appropriate to the stage of my programme were discussed with my mentor(s) / practice educator(s) early in the placement.
16 5%
Suggestions for alternatives (3)
Needs rewording / clumsy (10)
Student referred to their actual placement (3)
14. My named mentor(s) / practice educator(s) and the wider team were able to facilitate my development and recognised the importance of my learning needs
14 5%
More than one question (4)
Needs rewording (7)
Student referred to their actual placement (2)
Other (1)
20. Where applicable, the placement encouraged service users/carers to contribute to the assessment of my performance in practice.
14 5%
Question confusing (2)
Suggested terminology 'provide performance feedback' (1)
N/A, other (11)
Question B
The comments for Question B were reviewed and the highest number of responses was requesting the ability to leave a qualitative comment. This functionality is currently available on the PPQA software. For example, for Question 1 which received the greatest number of negative responses (47, 16%) there were 24 students asking to leave a comment, 5 asking for a range of responses, e.g. a Likert scale and 18 commenting further on the actual question (similar to Question A). The number of students actually requesting a change to the Y/N/NA responses was very low.
Final Question As a final question the students were asked the following:
“Is there anything we have not covered in the 25 questions? Please let us know of any aspect of your placements which you feel you would not have had the opportunity to comment on.”
49 of the students left comments. 11 students requested a final optional comment box after the evaluations questions had been completed, which was also suggested at the 19
th June workshop (see
PPQA: Student evaluation review Page 10 of 17 7th October 2013
Consultation: Responses section above). 11 students wanted to be able to comment on the standard of the mentorship received i.e. in terms of time and quality and a review of the comments alongside the proposed questions 14 to 16 is therefore indicated (see Appendix 1). Comments and suggested questions included the following:
The following comment was left by one student praising the new questions but also expressing their
view regarding anonymity.
“Do you think your skills
and confidence in practice
have grown due to this
practice placement?”
“Do you feel more confident
in your abilities to practice
after completing the
placement?”
“….Support offered around
accommodation and travel
…”
“….relevance of coursework
to practice…”
“Were mentors afforded
sufficient time (by
management) in their
schedule to facilitate, support
and reflect on learning and
practice” “There should include a
question about having an
allocated mentor and being
able to work with them
often enough.” “I believe that what other
students have written should
be available to the students
that are next visiting to give
them ideas of how to get the
best out of their placement
and….”
“Were you provided with the
appropriate workwear/PPE
in order for you to carry out
your work in the placement
area.”
“Very good selection of questions - much
better improved than previous questions,
they deal with more relevant and realistic
situations. The only improvement I would
make is to make the questionnaire
anonymous as this can cause students to
feel scared to voice their true opinions on
all matters for fear of being reprimanded
by mentors, tutors etc.”
Contents
PPQA: Student evaluation review Page 11 of 17 7th October 2013
Recommendations
The recommendations are as follows:
The purpose of the questionnaire should be advertised to students by HEI staff and also on the PPQA website. Timescales to be decided by the PPQWG.
The proposed student questions with a green rating in Appendix 3 for Question A are agreed by the PPQWG as valid and will be included on the PPQA website when the student evaluation area is updated.
The proposed student questions which are listed above in “Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A” should be reviewed to take into account the comments received by the students on each question and also the final question comments. A half day workshop is suggested, involving the individuals who contributed to the 19
th June workshop.
Contents
PPQA: Student evaluation review Page 12 of 17 7th October 2013
Appendix 1: Proposed Student Evaluation Questions
Preparation for Practice Experience Response Y/N/NA
1. I received information and support from my education provider, which was appropriate for the stage of my programme, so that I was prepared for my placement.
Y/N
2. I was informed of my placement allocation with sufficient time for me to make contact with the placement.
Y/N/NA
3. The placement profile was informative. Y/N/NA
Orientation
4. I had a planned, timely and comprehensive induction which orientated me to my environment and relevant policies and procedures, including health and safety information
Y/N
5. I had somewhere safe to store my belongings. Y/N/NA
Learning Opportunities/Resources
6. The placement provided me with access to resources which helped me with my learning
Y/N
7. There was sufficient access to internet IT facilities Y/N
8. The learning opportunities available in this placement enabled me to achieve my learning outcomes
Y/N
9. I was encouraged to take part in practice activities appropriate to my level of experience.
Y/N
10. I had sufficient opportunities to receive feedback after being observed in practice.
Y/N
11. The placement provided me with the experience of working inter-professionally
Y/N/NA
Supervision/Support
12. I received timely support from my course/programme team whilst on placement when I needed it
Y/N/NA
13. The learning needs appropriate to the stage of my programme were discussed with my mentor(s) / practice educator(s) early in the placement.
Y/N
14. My named mentor(s) / practice educator(s) and the wider team were able to facilitate my development and recognised the importance of my learning needs
Y/N
15. With the support of my mentor(s) / practice educator(s) I was able to work towards my placement learning outcomes
Y/N
16. Staff in the practice area were friendly, helpful, included me as part of the team and had time to answer my questions
Y/N
17. I understood the process for raising concerns within the placement Y/N
Feedback/Assessment
18. My mentor(s) / practice educator(s) understood and followed the student assessment process.
Y/N
19. My development in practice was informed by on-going constructive feedback Y/N
20. Where applicable, the placement encouraged service users/carers to contribute to the assessment of my performance in practice.
Y/N/NA
21. My final assessment was fairly assessed and focused on feedback about my performance across the whole placement.
Y/N
Recommendations/Concerns
22. I would recommend the placement setting to friends and family if they needed similar care or treatment.
Y/N
PPQA: Student evaluation review Page 13 of 17 7th October 2013
23. I had concerns about patient care on my placement but these were dealt with appropriately
Y/N/NA
24. I had concerns about my educational experience on my placement but these were dealt with appropriately
Y/N/NA
25. I would recommend this placement as a valuable learning experience Y/N
Contents
PPQA: Student evaluation review Page 14 of 17 7th October 2013
Appendix 2: Invitation to students
Contents
PPQA: Student evaluation review Page 15 of 17 7th October 2013
Appendix 3: Student Survey Data
A. Do you think Question X is clear and unambiguous?
B. Do you think a Yes, No (and in some case N/A see Appendix 1)) response is appropriate for
question X?
Key
Over 10% of the students answered “No” to the question
5 to 10 % of the students answered “No” to the question
Less than 5 % of the students answered “No” to the question
No. Proposed Student Evaluation Question Question A. Question B.
Y N Y N
3. The placement profile was informative. 250 85%
44 15%
254 86%
40 14%
23. I had concerns about patient care on my placement but these were dealt with appropriately
260 88%
34 12%
249 85%
45 15%
24. I had concerns about my educational experience on my placement but these were dealt with appropriately
262 89%
32 11%
255 87%
39 13%
7. There was sufficient access to internet IT facilities 271 92%
23 8%
273 93%
21 7%
5. I had somewhere safe to store my belongings. 272 93%
22 7%
275 94%
19 6%
4. I had a planned, timely and comprehensive induction which orientated me to my environment and relevant policies and procedures, including health and safety information
277 94%
17 6%
262 89%
32 11%
1. I received information and support from my education provider, which was appropriate for the stage of my programme, so that I was prepared for my placement.
275 94%
19 6%
247 84%
47 16%
12. I received timely support from my course/programme team whilst on placement when I needed it
278 95%
16 5%
273 93%
21 7%
13. The learning needs appropriate to the stage of my programme were discussed with my mentor(s) / practice educator(s) early in the placement.
278 95%
16 5%
287 98%
7 2%
14. My named mentor(s) / practice educator(s) and the wider team were able to facilitate my development and recognised the importance of my learning needs
280 95%
14 5%
275 94%
19 6%
20. Where applicable, the placement encouraged service users/carers to contribute to the assessment of my performance in practice.
280 95%
14 5%
278 95%
16 5%
9. I was encouraged to take part in practice activities appropriate to my level of experience.
283 96%
11 4%
265 90%
29 10%
6. The placement provided me with access to resources which helped me with my learning
283 96%
11 4%
269 92%
25 8%
16. Staff in the practice area were friendly, helpful, included me as part of the team and had time to answer my questions
283 96%
11 4%
268 91%
26 9%
22. I would recommend the placement setting to friends and family if they needed similar care or treatment.
283 96%
11 4%
275 94%
19 6%
8. The learning opportunities available in this placement enabled me to achieve my learning outcomes
284 97%
10 3%
250 85%
44 15%
19. My development in practice was informed by on-going constructive feedback
285 97%
9 3%
284 97%
10 3%
11. The placement provided me with the experience of working inter- 286 8 273 21
PPQA: Student evaluation review Page 16 of 17 7th October 2013
professionally 97% 3% 93% 7% 18. My mentor(s) / practice educator(s) understood and followed the
student assessment process. 287 98%
7 2%
276 94%
18 6%
21. My final assessment was fairly assessed and focused on feedback about my performance across the whole placement.
287 98%
7 2%
272 93%
22 7%
10. I had sufficient opportunities to receive feedback after being observed in practice.
288 98%
6 2%
272 93%
22 7%
17. I understood the process for raising concerns within the placement
288 98%
6 2%
290 99%
4 1%
15. With the support of my mentor(s) / practice educator(s) I was able to work towards my placement learning outcomes
289 98%
5 2%
280 95%
14 5%
25. I would recommend this placement as a valuable learning experience
289 98%
5 2%
271 92%
23 8%
2. I was informed of my placement allocation with sufficient time for me to make contact with the placement.
290 99%
4 1%
277 94%
17 6%
Contents
PPQA: Student evaluation review Page 17 of 17 7th October 2013
Acknowledgements
Thank you to all partners in the region who contributed to the consultation undertaken with the
outcome of the 19th June workshop.
Thank you to all the students who completed the online survey and the members of the PPQWG who
liaised with them.
Thank you to the colleagues below who were involved in the June 19th workshop.
Name Organisation
Anne Brown PLF Bradford and Airedale
Catherine Coates Leeds Metropolitan University
Steve Himsworth University of Hull
Corinna Klaentschi York St John University
Migaela Scorah Leeds Metropolitan University
Fiona McInnes University of York
Gillian Swan Leeds Institute of Medical Education
Heidi Cheung Sheffield Hallam University
Helen Green University of Leeds
Isabel Brislen Sheffield Health and Social Care NHS Foundation Trust
Jenny Shaw PLF Leeds
Kath Bowles University of Leeds
Katie Cook PLF North Humber
Sharon Parker PLF Bradford and Airedale
Linda Jackson Sheffield Teaching Hospital NHS Foundation Trust
Sally Underwood University of Sheffield
Niall Dew University of Huddersfield
Nicky Hughes The Open University
Samantha Smithson Yorkshire Ambulance Service
Sherree Hamburg PLF Bradford and Airedale
Valerie Hales PLF South Yorkshire
Janice High University of Bradford
Contents