How to understand the National Training Survey

The main problems interpreting the GMC Survey apply to any large dataset

Deciding what you want to find out is the first part of the battle!

Select one area of interest and focus on it.

Leave the other data to one side and concentrate on one theme at a time.

Later, you may need to cross-compare to see broader themes

Decide what you wish to investigate. Decide what you want to discover.

Follow the path! If you’re interested in ophthalmologists at Sunderland Eye Infirmary, ignore paediatricians at James Cook!

Concentrating on one area at a time will allow you to see the key themes for that group. Once you have seen those themes you can look at other data to ascertain whether it’s a problem to just that group or more widespread.

Lies, damn lies!The phrase “Lies, damn lies and statistics” isn’t to suggest that statisticians deliberately mislead, more that the person using the data must appreciate that what is shown (or not shown) can greatly affect the interpretation of the information. If an indicator score is not what you expected or hoped, check to see how many respondents were involved and how cohesive those responses are.

Read the instructionsThe GMC have given detailed guidance on how to interpret the survey results:http://gmc-onlineeducationreports.org/ReportsGuide.aspx

The box chart

The box chart represents a number of statistical techniques graphically.

It need not be scary. Just break them down into the pieces.

Two types of average are show – Mean and Median

Interquartile Ranges are represented by different shades of green

Confidence Intervals (Error bars) show how cohesive the responses are

The box chart

These are the “interquartile ranges”This is the MEAN score for the group

This is the MEDIAN score for all trainees nationally

These are the ERROR BARS (confidence interval)

This score is 0-100

Unless specified, it’s not a percentage!

Mean and Median

The mean is all the scores added up, then divided by the number of scores

The median is the middle value when all scores are arranged in order

2, 2, 8, 2, 7, 9, 2, 6, 3, 8, 1

Mean = 2 + 2 + 8 + 2 + 7 + 9 + 2 + 6 + 3 + 8 + 1 = 50.

50 ÷ 11 = 4.5

Median - sort the numbers into ascending order

1, 2, 2, 2, 2, 3, 6, 7, 8, 8, 9

Interquartile ranges

1, 2, 2, 2, 2, 3, 6, 7, 8, 8, 9

Bottom quartile Interquartile Range Top quartile

The quartile ranges are 25% (a quarter!) of the NUMBER of scores when arranged in ascending order.

The MEDIAN score is precisely in the middle of the 2nd and 3rd quartiles.

Error Bars

1, 2, 2, 2, 2, 3, 2, 2, 1, 3, 79

The mean score for the above series is 9.

Is that a true average?

The confidence bars allow you to see if everyone feels the same

If there are long error bars, there is a wide range of opinion

If there are short (or no) error bars, it shows everyone thinks the same

Outliers

A red or a green triangle to the right of the box chart denotes an outlier

Essentially, an outlier is a MEAN score for the group which sits in either the top or the bottom quartiles and the error bars do not extend into the interquartile range.

The easiest way to look at these is to regard green triangles as showing good practice and red triangles as showing areas of concern.

It’s not fool proof – if there’s little confidence (long error bars) in the results, a very high or very low score may not be flagged.

This could suggest that some very satisfied (or dissatisfied) doctors are being “lost” amongst the data.

Outliers

A red or a green triangle to the right of the box chart denotes an outlier

Essentially, an outlier is a MEAN score for the group which sits in either the top or the bottom quartiles and the error bars do not extend into the interquartile range.

The easiest way to look at these is to regard green triangles as showing good practice and red triangles as showing areas of concern.

It’s not fool proof – if there’s little confidence (long error bars) in the results, a very high or very low score may not be flagged.

This could suggest that some very satisfied (or dissatisfied) doctors are being “lost” amongst the data.

Indicators

Now we understand how to use a box chart, we can read the indicator scores

The trainee survey uses 22 indicator scores

Each indicator score is “fed” by the responses to a series of questions

The “Overall Satisfaction” indicator (for example) is fed by the questions:

How would you rate the quality of teaching [informal (and bedside teaching) as well as formal and organised sessions] in this post?

How would you rate the quality of supervision in this post?

How would you rate the quality of experience in this post?

How would you describe this post to a friend who was thinking of applying for it?

Groups

All the results for the indicators are collated nationally (hence the national average)

Scores are generated for smaller groups:

All trainees at a given deanery

All trainees at a given hospital, trust or GP training scheme

All trainees in a given specialty

All trainees in a given specialty at a given hospital

Groups and Indicators are the criteria that should be used to “follow the path”

It’s useful to compare different groups

How does ACCS at the RVI compare to ACCS at UHND?

How does UHND compare to the RVI?

It’s useful to compare different indicators

Overall satisfaction is low

Workload scores badly

Following the path – tell a story

When you follow the path, a story can emerge.

What do you think these results tell you about the unit?

Overall satisfaction

Clinical supervision

Work load

Work intensity(appropriateness)

“Damn Lies”

This is the caveat – although the results of the survey are statistically valid, that does not necessarily mean that they are important

Always remember the error bars – they show how useful the data is

Bear in mind that the smallest reporting unit for the GMC Survey is three.

From a group of 100 trainees, if three are very unhappy it doesn’t look like a big problem. If it’s a group of five trainees, it looks like a huge problem.

Do not forget that even from 100, three unhappy trainees is three people’s working lives, and the lives of the hundreds of patients they treat.