questionnaire design, pretesting and quality control in clinical research celia p. kaplan division...
TRANSCRIPT
![Page 1: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/1.jpg)
Questionnaire Design, Pretesting and Quality Control in Clinical Research
Celia P. KaplanDivision of General Internal Medicine
August 24th 2009
![Page 2: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/2.jpg)
Good Clinical Practice (GCP)
“A standard for the design, conduct, performance, monitoring, auditing, recording, analysis, and reporting of clinical trials that provides assurances that the data and reported results are credible and accurate, and that the rights, integrity, and confidentiality of trial subjects are protected”
WHO good clinical practice standards
![Page 3: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/3.jpg)
Quality Control
![Page 4: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/4.jpg)
Quality Control
Minimizes the chance of obtaining faulty data and drawing erroneous conclusions
![Page 5: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/5.jpg)
Quality Control
Minimizes the chance of obtaining faulty data and drawing erroneous conclusions
Fulfills administrative and regulatory obligations
![Page 6: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/6.jpg)
Sources of Data Errors Data acquisition
![Page 7: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/7.jpg)
Sources of Data Errors Data acquisition
instrument problems
![Page 8: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/8.jpg)
Sources of Data Errors Data acquisition
problems instrument lost data
![Page 9: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/9.jpg)
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data, changed )
![Page 10: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/10.jpg)
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data, changed)
Data recording
![Page 11: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/11.jpg)
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data or changed)
Data recording Data entry
![Page 12: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/12.jpg)
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data or changed)
Data recording Data entry Data management
![Page 13: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/13.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
![Page 14: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/14.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
![Page 15: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/15.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project
![Page 16: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/16.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project Develop an Operations Manual
![Page 17: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/17.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system
![Page 18: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/18.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
![Page 19: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/19.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation
![Page 20: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/20.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring
reports
![Page 21: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/21.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings
![Page 22: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/22.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and
performance reviews
![Page 23: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/23.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
![Page 24: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/24.jpg)
Questionnaires/interviews Much of the data in clinical and
epidemiological research is gathered using questionnaires and interviews
![Page 25: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/25.jpg)
Questionnaire Measures
Desired qualities Reliability – the extent to which the
measures give consistent results A reliable measure is consistent, responses do not
change Test-retest reliability Statistical techniques
Validity – the extent to which the measures reflect the “truth”
![Page 26: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/26.jpg)
Survey measures: Validity Face validity: the researcher’s best judgment as
to whether the item is measuring what it is supposed to measure
Predictive validity: the item accurately predicts an outcome (e.g., college entrance exams predicting college performance)
Concurrent validity: the item is validated by comparing it against a known valid measure (e.g., a measure of job satisfaction might be correlated with work performance)
![Page 27: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/27.jpg)
Survey measures: Validity Content validity: the items are a representative
sample of the characteristics that comprise the area they intend to measure. Determination of content validity usually relies on experts Example: Patient decision making style
Construct validity: the item demonstrates that it accurately measures a psychological construct like depression or satisfaction
![Page 28: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/28.jpg)
Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”
Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”
Questionnaire Measures Desired qualities
Accommodate participants’ age, literacy,
cultural, linguistic, and SES characteristics
Yield good variability in answers
Result in low percent of missing data
![Page 29: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/29.jpg)
Questionnaire Development: Pitfalls
Double-barreled questions – Avoid “and/or” statements in questions e.g., “Does your department have a special
recruitment policy for ethnic minorities or women?”
Long questions and/or questionnaires
Use of “other” as a response category
![Page 30: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/30.jpg)
Use of Other as Response Category
Example
10. Are you Latino/a or Hispanic? 1 Yes
0 No
11. What is your race/ethnicity? Please check one answer only. 1 Black or African American 2 Asian, Asian American or Pacific Islander 3 White, European American or Caucasian 4 American Indian or Alaska Native 5 Other Please specify ____________________
![Page 31: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/31.jpg)
Use of Other as Response Category
Example1 Ameriasian 19 INDIAN (EAST) AMERICAN
2 American 20 Italian
3 Arab 21 MIDDLE EAST
4 Ashkenazi 22 Middle Eastern/ Indian Subcontinent
5 Asian and Caucasian 23 Mixed
6 Brazilian 24 NATIVE AMERICAN
7 Cuban 25 Oriental
8 East Indian 26 PERSIAN
9 Egyptian 27 POLISH AMERICAN
10 EGYPTIAN 28 South Asian
11 egyptian american 29 Southern Asian
12 From India 30 West Indian
13 half Latin, half black 31 White, Arab
14 Indain American 32
15 India- Indian 33
16 Indian 34
17 INDIAN 35
18 Indian (east) 36
![Page 32: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/32.jpg)
Questionnaire Development: Order
First question should be a fact
Beginning questions should stimulate interest
Questions should be in logical order and groupings
If ask attitude questions first and then behavior questions, may inflate reports of behavior
Sensitive questions and open-ended questions should be at the end of the survey
e.g., sexual behavior, religion, income
![Page 33: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/33.jpg)
Questionnaire Development: Language Use simple, clear, common, and
natural language Avoid jargon, complex terms, clichés,
colloquialisms, e.g., “down in the dumps” Check number of syllables in words Check reading level: aim for < 8th grade Define ambiguous terms, e.g., “family” or
“neighborhood” Keep sentences short
![Page 34: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/34.jpg)
Questionnaire Development: Recall
Solutions: Narrow the reference period Average the response: ask about a typical
day or in general (e.g., hours of sleep) Use landmark events or milestones to
anchor time frame: “Since you moved to the US,” or use calendar with major holidays to establish major life events
Use cues to generate associations to jog memory: describe procedure, setting
![Page 35: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/35.jpg)
Questionnaire Design
1Select Interview Methodology
Interviewmethodology
![Page 36: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/36.jpg)
Questionnaire Design
2Review existing
questionnaires and measures
Interviewmethodology
Reviewexisting
measures
![Page 37: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/37.jpg)
Questionnaire Design
3Conduct exploratory
qualitative work
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
![Page 38: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/38.jpg)
Questionnaire Design
4Develop a draft survey
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
![Page 39: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/39.jpg)
Questionnaire Design
5Translation and back-translation
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
![Page 40: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/40.jpg)
Questionnaire Design
6Pre-testing
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
![Page 41: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/41.jpg)
Questionnaire Design
6Pre-testing
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
![Page 42: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/42.jpg)
Questionnaire Design
7Finalize the Questionnaire
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
Finalizedinstrument
![Page 43: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/43.jpg)
Questionnaire design
Selection of interview methodology
Face-to-face or in-person interview
Telephone
Self-administered, paper
Self-administered, computer/web/e-mail
![Page 44: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/44.jpg)
Questionnaire design
Interview methodology: Face-to-Face
Advantages
Enables the interviewer to establish rapport with respondent
Permits more complex questions Permits the use of visual aids Longer interviews are possible
Disadvantages Cost per interview Quality of answers too personal (sensitive
questions)
![Page 45: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/45.jpg)
Questionnaire design
Interview methodology: Telephone Advantages
Faster contact of participants Better to elicit personal/sensitive information Results are available right away if using
computer-assisted interviewing (CAI) Can do random selection of telephone numbers
Disadvantages Certain sectors of the population are switching to
cell phones only More expensive than mail surveys Difficulties in reaching participants during the day
![Page 46: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/46.jpg)
Questionnaire design
Interview methodology: Self-administered and mail surveys (Paper)
Advantages One of the least expensive methodologies Needs addresses of potential participants Allows participants to answer at their leisure Less intrusive Elimination of interviewer bias
Disadvantages Response rate is low Difficulties to develop clear skip patterns It may be answered by a family member Very difficult to implement in low literacy populations
![Page 47: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/47.jpg)
Questionnaire design
Example: Self-administered
& Mail Surveys
![Page 48: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/48.jpg)
Questionnaire design
Interview methodology: Self-administered, web-based surveys (Computer)
Advantages Elimination of data entry and editing costs Get better answers to sensitive questions Elimination of interviewer bias Ensure that skip patterns are accurately followed Allows participants to answer at their leisure
Disadvantages Interviewees must have access to a computer Must possess (or purchase) a list of email addresses Lower response rate People may quit in the middle of a questionnaire
![Page 49: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/49.jpg)
Questionnaire designExample: Web-based survey
![Page 50: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/50.jpg)
Questionnaire design
Survey modalities: Answer types
Numeric open-endedHow old are you? ____
Text open ended How can we improve our services? ______________
Multiple Choice--CategoricalWhat is your marital status
Single 1 Married 2
etc.
![Page 51: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/51.jpg)
Survey modalities: Answer types Multiple choice-- Ordinal
In general, would you say your health is…?1 Excellent2 Very good3 Good4 Fair5 Poor77 DK
Scales On a scale where “10” means you have a great amount
of interest in a topic and “1” means you have none at all, how would you rate your interest in each of the following specialties?
Cardiology ___ Oncology __ Dermatology__
Questionnaire design
![Page 52: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/52.jpg)
Survey modalities: Answer types Multiple choice-- Ordinal
In general, would you say your health is…?1 Excellent2 Very good3 Good4 Fair5 Poor77 DK
Scales On a scale where “10” means you have a great amount
of interest in a topic and “1” means you have none at all, how would you rate your interest in each of the following specialties?
Cardiology ___ Oncology __ Dermatology__
Questionnaire design
![Page 53: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/53.jpg)
Survey modalities: Answer types Likert-Type scales
Used to know respondents' feelings or attitudes The respondent indicates how closely their feelings
match the question or statement on a rating scale. Response categories: “Strongly Disagree" (least
agreement) to or "Strongly Agree." (most
agreement)
Example: In general, to what degree is each of these factors a
barrier for you in referring a patient to a clinical trial? Not a barrier A major barrier
1 □ 2 □ 3 □ 4 □ 5 □
Questionnaire design
![Page 54: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/54.jpg)
Questionnaire design
Review Existing Questionnaires & Measures
Reference databases Medline, Pubmed, Psychinfo, others
Compendia of measures Books that compile various measures
and review their characteristics
Request questionnaires from other investigators
![Page 55: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/55.jpg)
Questionnaire design
Review Existing Questionnaires & Measures
Is there evidence the instrument works in your target population?
Examine validity and reliability of measures when used in populations similar to your study population
When gold standard does not exist, for construct validity, look at relationship of measure to other measures to see if in hypothesized direction
![Page 56: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/56.jpg)
Questionnaire design
Conduct Qualitative Exploratory Work
Focus groups or semi-structured interviews Explores a topic before constructing formal
questions Rationale:
Determine what is important Discover how respondents think about the topic Assess better language/words to use Identify response categories
![Page 57: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/57.jpg)
Questionnaire design
Create a Draft Survey Instrument
Write initial draft Process of revision and refinement through
cycles of translation/back-translation, reviews and pre-tests (Steps 5 and 6)
![Page 58: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/58.jpg)
Questionnaire design
Translation/Back-translation Procedures
Both versions should have the same meaning Translation and back translation Review by a group of native speakers Challenging when more than one language
involved in the study
![Page 59: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/59.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
![Page 60: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/60.jpg)
Quality Control
Pre-testing the Questionnaire: Overview
Refines Instruments and questionnaires
Evaluates Recruitment methods
Interventions
Data entry and management system
Protocols
Assists with training of personnel
![Page 61: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/61.jpg)
Quality Control
Pre-testing the Questionnaire
Good to test: Sensitive questions Complex or poorly defined topics Questions with terms respondents may not
understand Translations Layout and instructions
![Page 62: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/62.jpg)
Quality Control
Pre-testing the Questionnaire:Field Testing Techniques
Expert reviews Cognitive interviews Full pretest
![Page 63: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/63.jpg)
Quality Control
Pre-testing the Questionnaire:Expert Reviews Formal and systematic examination of
a questionnaire by experts in the field Can cover question wording, layout,
and skip patterns Fast method prior to formal testing May not be a complete review
![Page 64: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/64.jpg)
Quality Control
Pre-testing the Questionnaire:Cognitive Interview: Overview Diagnostic tool for pretesting survey
questions derived from social and cognitive psychology
Interview about an “interview” Explores the processes by which
respondents reach answers Based on structured questionnaire and
protocol
![Page 65: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/65.jpg)
Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Concurrent proving
Ask probes immediately after respondent has given answer to survey item
Advantage: information is fresh in respondent’s mind Disadvantage: Disrupts interview flow and relationship
between questions Retrospective probing
Ask probes after entire interview or block of questions Advantage: able to assess standard administration of
items Disadvantage: Participant may not remember thought
process
![Page 66: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/66.jpg)
Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Probing questions Think aloud interviews
Interviewer asks respondent to think aloud as they answer question
Paraphrasing Respondents rephrase the question in their own words
![Page 67: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/67.jpg)
Quality Control
Pre-testing the Questionnaire:Cognitive Interview: Key Features Comprehension of the question
Do participants understand words and phrases as intended by the researchers (meaning)
“How did you arrive at your estimate of risk?” “What did you understand by the word ‘risk’?”
Retrieve of information Identify process respondents use to answer questions Examine strategies used to access memory
Edit responses: Decide what to report Are items unacceptable? Is answer embarrassing,
socially undesirable?
![Page 68: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/68.jpg)
Quality Control
Pre-Test
Pretest with a sample similar to the population to be studied
Simultaneously pretest recruitment procedures and questionnaire implementation
![Page 69: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/69.jpg)
Quality Control
Pre-Test: Behavior Coding
Code interaction between interviewer and respondent Examples
R answers before hearing whole question R looks puzzled R asks for clarification I repeats question
![Page 70: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/70.jpg)
Quality Control
Pre-Test: Respondent Debriefing
Structured follow-up questions at the end of the interview to assess: Whether questions were clear Ease of completing the questionnaires If two versions are tested, which is
preferable Whether an important question was
missed
![Page 71: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/71.jpg)
Quality Control
Pre-Test: Interviewer Debriefing Interviewers implement pretest with
participants Interviewers detect problems with
questions, response categories, and skip patterns
Interviewers report issues to investigators
![Page 72: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/72.jpg)
Quality Control
Pre-Test Questionnaire: Summary
Testing takes time and resources Some procedures take place late in the
developmental cycle, limiting the amount of change possible
Danger of making changes that are not further tested
Not testing leads to questionnaires that are potentially unreliable or will give you invalid results
![Page 73: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/73.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
![Page 74: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/74.jpg)
Quality Control
Operations Manual Aim is to ensure high quality data Clearly written, detailed written instructions to obtain
uniformity across sites, staff, and procedures Standardizes procedures for all aspects research project.
Defines how to… Recruit participants Consent participants Measure variables Transport clinical samples Enter data Monitor progress Conduct data analysis
Requires updates, clarifications, and recording of decisions made
![Page 75: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/75.jpg)
Quality Control
Data Management System Desirable features
Range and field type checks Ease of screen set up and use Double data entry (if possible)
Security features protection of human subjects’ rights (privacy)
For Web-based systems Virus protection to monitor and eliminate security
threats Database server behind firewall All data backed up regularly
![Page 76: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/76.jpg)
Quality Control
Staff Training Training and certification of the research team members
Rules to conduct interviews Rules for recruiting participants, including how to
obtain consent Methods
“Train the trainer” model Audio-visual techniques
Certification/recertification to maintain skills
![Page 77: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/77.jpg)
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring
reports Conduct periodic project meetings Complete interim staff training and
performance reviews
![Page 78: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/78.jpg)
Quality Control
Data Monitoring Reports
Standard procedures for Identifying missing, incomplete data Identifying incorrect skip patterns Identifying inconsistent and erroneous data Verifying correction
![Page 79: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/79.jpg)
Quality Control
Interim Staff Training and Performance Reviews
Provide continuous supervision Hold frequent meetings of the study team Review all aspects related to implementation of
the study, including Interviewers activities
Listen to interviews Compare measures between interviewers Compare refusal rates between interviewers
![Page 80: Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 24 th 2009](https://reader036.vdocument.in/reader036/viewer/2022062517/56649ed05503460f94bde2c1/html5/thumbnails/80.jpg)
Good Clinical Practice
Good Instruments Complete and clear protocol manuals Extensive pretesting Ongoing monitoring procedures