care quality commission (cqc) widget user testing february 2015 version 1.1

Care Quality Commission (CQC) Widget User Testing February 2015 Version 1.1

Introduction These are the findings and recommendations from a day of testing the Widget, February 2015. The individual moderated sessions (one user / one facilitator) were conducted with a total of FIVE users who had been recruited to an agreed specification in particular having themselves or a relative spent time in hospital within the last two years. Each session covered a subset of specified tasks as set out in the approved discussion guide previously circulated. Sessions lasted for 1 hour. Sessions were conducted using a Windows PC running Tobii eye tracking and Morae session recording with a remote observer adding notes in real time. They were remotely observed live locally and via web streaming, and recorded to enable subsequent further analysis.

The severity assigned to the issues listed below has been agreed by both of the consultants involved in the testing. There were no serious issues to report. Serious Major user experience issues that stop users from completing tasks and would likely lead them to abandon the site. Significant Issues that interrupt task progress or are detrimental to the users experience. Minor Smaller issues that irritate or distract users but do not impact progress. Positive something of note Severity Ratings

Summary of Results 1 Description of issueRelevant section Severity Widget Further work required to reduce widget appearing (and being overlooked) like an Ad 14 Significant Improve the prominence of this KEY.22 Significant Define your criteria and ensure it is clearly visible to those that wish to read it. 23 Significant To recommend that all institutions required to display the widget do so above the fold to improve the prominence of the CQC widget. 9 Significant Progress the development of the longer widget with domain field rankings suggested as it looks less like an advertisement. 27 Significant To suggest that providers display the widget on other key pages of their site associated with performance (especially one labeled performance). 10 Minor If possible, the widget should be included on the pages relevant to service level performance and rating. 12 Minor

Summary of Results 2 Description of issueRelevant section Severity Widget contd This reinforces the efforts already under way by the CQC to raise its public profile and increase general awareness of the official role that it has. 13 Minor Include Report Summaries wherever possible. Highlight to those involved in report generation of their usefulness. 20 Minor Users wanted to see what the Hospital response was, what are they doing about it? What is their Action Plan? If possible, these should be included. 20 Minor Draw together some best practice for widget display. This would include, positioning, and avoiding contrasting against other boxes on the side (with images for example). 25 Minor Provide some best practice examples of how the widget should be displayed on hospital websites in order to encourage hospitals to improve the visibility of the performance widget. 17 Minor

Participant Profiles ParticipantNameAgeGender P1C66M P2S42F P3M20F P4K55M P5M K45F

Task 1 Introduction. Looking for health info P1 Spent a long time working in the Middle East, was thinking about retiring to the Philippines but was worried about health care provision. Ended up back in the UK and looking at quality of care services, having to use them quickly on his return for prostate issues and suffered tuberculosis. P2 - she has bad experiences observing her parents care and was due for an operation herself, and so she wanted to check the level of service in advance. Says she wanted to be prepared for what to expect rather than finding out on the spot. P3 - has previously looked for herself and family regarding health care, Google is her first port of call and then usually ends up on a Hospital website. P4 - explains he researches for his mother; care homes near where he lives, all private no NHS (Google was his starting point). P5 She trusts the system, but would read things like OfStead reports if the school was bad and would do the same for healthhad a previous incident where she had to force the hospital to take action.

WIDGET

Task 2 Provider Widget (Trust) The red line marks the bottom of the screen. Participants were looking for information to help them rate the Trust and were sidetracked by the information in front of them. Most not scrolling down until prompted. Recommendation To recommend that all institutions required to display the widget do so above the fold to improve the prominence of the CQC widget. If the site is going to be honest, that should be higher up, its good for them but not good for the patient. Kirti

Task 2 Provider Widget (Trust) cont. The green boxes highlight the areas that were distracting participants from their task. In particular the section on the right that takes the user to the page offering an overview of their performance. Areas on the right-hand menu included Case Studies and Improving the Patient Experience. Recommendation To suggest that providers display the widget on other key pages of their site associated with performance (especially one labeled performance).

Quotes It feels that the site seems to try to sell itself. I would expect to see statistics such as number of operations, number of patients admitted, survivals etc. within the our performance, I would also expect to see death rate, who sets the targets. Presumably the trust itself has written the performance page? C I would immediately look for a yearly performance report, also would look for forum/feedback and minimum standards and objectives, the 'our performance' and 'case studies' are standing out for me. S This Our Performance is the page is based on the Trust, they would be trying to bury anything bad. I guess its from managers, I would hope there is something from a third party. (She does not scroll down) S I would read the report to find out more, but I would want a second opinion as I have not heard of CQC before. M. I would look at patients and visitors as well as our services, I think it is lower down because it is negative! K

If possible, the widget should be included on the pages relevant to service level performance and rating.

Task 2 Provider Widget (Trust) Three of the participants who did not notice the widget described glancing over it and assuming it was an advertisement. This will be heavily influenced by none of the participants knowing the role of the CQC has in determining service level quality. Recommendation This reinforces the efforts already under way by the CQC to raise its public profile and increase general awareness of the official role that it has.

The Widget looks rather like an Ad. It looks like an intrusion, like Google showing ads. I tend to not pay close attention to it... it looks like an advertisement, the requires improvement stands out in orange, too much so and that it's like an ad. C I think this is a really good idea but I wouldnt have clicked there it didnt catch my eye. M. It feels that the positioning of it is not too good. I would also expect to find more information on other pages. M. The smaller one, I could easily go past itit looks like an advert. Looking at a service, I would want to see it there regardless as it's helpful. M. It looks like an advert.. its like one of them popups, the orange colour is not as vibrant as the red, which is a lot more popup-like. MK. Recommendation Further consideration required to reduce the chances of being overlooked as an Ad.

Task 2 Provider Only the first two participants took the Provider task and it was decided to skip over and straight into the Hospital example instead. The overview of the hospitals within the Trust, as seen here. Was favorably received.

Task 2 Provider (smaller) Widget for Trust example As we dropped the Trust example we did not explore the preferences for the different widgets within this specific context, however this was picked up when looking at the Hospital and Service examples.

Task 3 Hospital Widget Again, the widget falls below the fold proving difficult to spot (apart from the first two participants). Recommendation Provide some best practice examples of how the widget should be displayed on hospital websites in order to encourage hospitals to improve the visibility of the performance widget.

Task 3 Hospital Widget The whole widget is one link that surprised participants expecting to be taken to the sub-section they are interested in. So, they would be interested in seeing why well-led is considered inadequate. There was some confusion as to what the criteria actually meant e.g. safe and how they would find out how it was being defined. Recommendation Having discussed this with the team present on the day, I dont think a specific recommendation can come from this as it is not advisable to bury someone deep into the report. Criteria descriptions are best dealt with in situ within the report rather than on the hospital pages.

Task 3 Hospital Widget (REPORT) These reports were favorably received by participants. They are clearly set out at first glance and not too overwhelming. However, there were a few issues to report.

Task 3 Hospital Widget (REPORT) Looking for why the Hospital was considered inadequately led, users expect to see a summary under summary not a link to the full report. Recommendations I understand there is a gradual move away from report summaries due to the time taken to draft them, but this is an example of where they are highly valuable (on the website). This should be highlighted to those involved in report generation. Users wanted to see what the Hospital response was, what are they doing about it? What is their Action Plan? If possible, these should be included.

Quotes What does well-led mean? S I want to see a very brief summary [tagline] to know information before diving in. S Wo! The definition of inadequate, what it says, Im very surprised and worried to see that inadequate means what it means inadequate is not a strong enough word! S Im not sure what the expressions mean and guess I might have to download the inspection report? M. I dont know why, but its the actual light system that caught my attention first, Im not sure what safe means, is it secure when youre there? Or is it your treatments are safe with them? M K. Where's cleanliness [within the ratings]? M K.

Task 3 Hospital Widget (REPORT) The Key The Click for Key went unnoticed in these sessions and was only spotted after heavy prompting. The key only describes the grades not the criteria. Recommendations Improve the prominence of this key. Each grade can have a roll-over linking to the key within the report that follows in order to reinforce the definitions being used. Define your criteria and ensure it is clearly visible to those that wish to read it.

Quotes I'd like to know what 'requires improvement' means [within report]. C It [widget] should be more prominent and I would also want to see a response from the organisation itself. C (Do the organisations tell you what their action plan is?) I didnt notice that at that time, I guess it gives you a bit more in depth information as to what the icons mean it needs to be a bit more user friendly. K Im clicking on the word safe, it doesnt tell me what does it mean by safe? M K. Okay thats interesting but it [the Key] was too small for me to see it.. I didnt see it. M K.

Task 4 Hospital Widget (Small) Again, it is difficult to consider the best widget design from the preferences of the five test participants (that did not offer much in terms of preference). The widgets are considered after the next example looking at a Service example.

Task 4 Service Widget This widget in particular looked like an advertisement to most of the participants. This is because it is on the side, brightly coloured and it is not helped by the image above it. Recommendation Draw together some best practice for widget display. This would include, positioning, and avoiding contrasting against other boxes on the side (with images for example). I want to see patient reviews on the website for reassurance. M.

Awareness of CQC and role very low The CQC have made an assumption that people know who they are! C I dont know if there are any externals involved in auditing. C I have heard of the CQC but dont know what they do. S

Widget Considerations Opinions on the widget were mixed between the small sample (5 participants). However, as there was a familiar theme emerging, warning that the widget could be mistaken for an advert the following considerations are offered based on the comments of the participants and a common sense approach moving forwards. The smaller widget without the breakdown across domain fields loses the context of the rating (and is more likely to be overlooked as an advert). The larger widget, containing the traffic light system for the relevant domain fields, gives a better indication that it is a considered (official) rating. The traffic light system presented here, reinforces the ratings used across the CQC landscape. It is recommended that the longer widget be the design developed moving forwards.

Final Reflections The system is a good idea and would help patients. S I think the explanations are really good.. and the way they present their information is clear and straightforward. M I think it is relevant to me - it's quite clear for most people its like the Food Standards Agency the CQC, I think it is clear that CQC are official. M. I would trust CQC... especially coming from the NHS site. K I prefer the small one [widget], it doesnt give you enough detail so Im enticed to click on read report, I dont want to be too put off at the start! K Would I look for the ratings? How would someone like my parents know to look for this stuff? For me, theres not enough information out there telling me that this information is at our hands. I didnt know who the Care Quality Commission were. M K.

REDEYE 29 Dr Jon Rimmer Senior Research Consultant RedEye [email protected]

care quality commission (cqc) widget user testing february 2015 version 1.1

Documents