Field test optionsinstrument, instructions,

non-response/ refusals, and Interviewer debriefing

Washington Group Regional Training Workshop

Rio de Janeiro19 – 20 September 2005

Margie SchneiderHSRC, South Africa

Recap What we have covered so far:

– Purpose of the questions and where they came from

– Core set of questions and extended set and Q X Q specifications

– Objectives of the test– Translation protocol

What we still need to go through: – How to design the test and related issues– Enumerator training– Plan for analysis and report writing

Different components of the testing process

• Pre-testing and expert review • Testing internal validity on people with

known disability status (quota sampling and linked to cognitive testing)

• Pilot testing – as for full Census or survey

• Full field test

Different sampling approaches will be discussed in relation to different testing protocols

Field Test options• 2 questions to be answered:

1. What set of questions should be used in field test?

2. What field test approach should be used?

Consider your own context, need, funding, skills, etc. in deciding

Question sets1. Core set – 4 + 2 (6 domains of

functioning): Compulsory2. Core + extended set on core domains :

highly recommended3. Core + extended set + further

additional questions: nice to have for analysis• 2 additional domains (learning and

interpersonal interactions)• Psychological distress

4. Country specific question set : nice to have for comparison with prior data

What approach to testing?• What factors to consider in

deciding: –Country needs–Capacity to undertake test–Funds available–Time available–Planned activities, e.g. other

surveys– ?

Census or national survey• Use of question set in planned

Census or national survey• Advantages:

– Large sample (5000 – 10 000 or whole pop)

– Minimal additional cost for disability questions

• Disadvantages:– Not sufficient space for extended set– Need to administer extended set and

cognitive test to sub-sample (costs of time and additional training)

Special study – small sample• Select sample on basis of known

‘disability’ status • Allows for construction of 2X2 table for

sensitivity and specificity calculations and identifying true and false positives and negatives – use of ‘gold standard’

• Suggest around 200 true positives and 200 true negatives

• Interviewer does not know status of respondent (unless visible and obvious)

Special study (contd.) • Advantages:

– Larger set of questions– Close observation of interview– Cost is not exorbitant

• Disadvantages:– Might not get full population

representation– Cost of establishing true status is high– What counts as the true positive?

(beyond the easily observable) – What is the ‘gold standard’?

Special study – large sample• Uses population based sample• Sample size based on expected

prevalence of the different types of activity limitations

• Prevalence rates are usually low and so need large sample to yield enough ‘disabled’ respondents

Special study – large sample (contd)

• Advantages:– Large data set– Detailed set of questions - more than

extended set– Provides an indication of prevalence

using core and extended sets• Disadvantages:

– Costly and time consuming– Requires capacity to run and analyse

Field test and cognitive test• All respondents have core set • One sub sample of respondents

have cognitive test (core + extended are embedded)

• A second sub-sample of respondents do – Core + extended– Core + extended + further Qs– Core + extended + country specific

questions• Some do all ?

Non-responses and refusals•How do you define these?

•How do you deal with these?

•Aim to: –Reduce non-response and

refusals–Manage them when they do

arise

Reducing NR and refusals• Use up to date sampling frame• Clearly defined reasons for data

collection• Preparing the way – prior contact,

letters, etc. • Adequate interviewer training• Allow budget for call-backs• Plan for follow up of NR and refusals• Separate refusals, part refusals, non-

contacts and sample loss (e.g. vacant dwellings)

Effect of each on analysis• Sample loss: does not create bias but

reduces sample size; need large enough initial sample to take these into account

• Refusals and non-contact: bias where these respondents might be different to those respondents reached

• Need to understand reasons for refusals and non-contacts – during pre-testing and pilot stage

Interviewer debriefing• Interviewers are well placed to

evaluate process and content at an early stage

• Interviewers should note comments in margins of questionnaires

• Different approaches: – Group discussion (focus group technique)– Interviewer rating forms– Standardised interviewer questionnaires– Combination of all three

Enumerator training

Washington Group Regional Training Workshop

Rio de Janeiro19 – 20 September 2005

Margie SchneiderHSRC, South Africa

General points• Select some disabled interviewers• Importance of all interviewers

having a good understanding of what disability is and is not and how it relates to the questions asked

• Avoid using term ‘disabled’ or ‘with disabilities’

• All interviews are face to face• Translation training

General points (contd.)• Confidentiality and understanding what

this means• Getting informed consent (ethics

clearance?)• Interviewers must be able to explain

purpose of survey• Read questions in set order and with set

wording• Editing in field of completed questionnaires• Submitting of completed interviews to

head office

Types of interviews• Direct: respondent answers for him or

herself• Interpreted: an interpreter

‘translates’ and respondent answers directly

• Facilitated: a third party assists in explaining (e.g. intellectually disabled person)

• Proxy: a person responds for another (e.g. child)

Interviewing disabled people

• Show respect and treat the person like anyone else

• Don’t use first names unless permitted

• Address the person directly (not their attendant)

• Ask how you can adapt your presentation to make it easier (no need to ask what is wrong with person)

Hearing difficulties• Lip reading• Lighting• Face the person• Get person’s attention before

speaking• Reduce background noise• Set the context – especially when

changing topics• Use written communication (literate)

Physical difficulties • Accessibility of building where

conducting interviews• Presence of attendant and

confidentiality issues• Get to same level (e.g. sitting for person

using wheelchair)• Person to be seated comfortably• Address person directly• Pointing may be difficult • Person may need breaks to move around

Visual difficulties• Large print and small print for

cue cards• Braille versions of cue cards• Good contrast printing for

pictures and print (black on white or yellow)

• Identify yourself and others in the room verbally

Communication difficulties • Clarify preferred mode of

communication• Repeat what you think was said

to clarify unclear speech• Limit to yes / no questions

Specific learning difficulties• Manage problems in spatial

orientation, hand-eye coordination

• Limit auditory, visual and tactile distractions

• Avoid written text • Explain carefully (if verbal

language skills are affected)

Intellectual difficulties • Be careful with informed consent• Explain terms simply• Listen carefully• Have familiar person (friend or

relative) close by• Use pictures or role play with

little human or animal figures

Emotional or mental health difficulties

• Side effects of medication• Break up interview if too fatigued • Give encouragement and support• Manage expressions of

frustration• Manage stress

Hidden difficulties• Might not come forward with

information because of fear of stigma

• Effect of medication• May need to break up interview