Download - How to assess questionnaires
![Page 1: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/1.jpg)
QUESTIONNAIRE
EVALUATION Dr. Ahmed-Refat AG Refat December 12,
2014
1
![Page 2: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/2.jpg)
Contents
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
2
Main Concepts & terms ( Pre-test)
Sequence of questionnaire development &
testing
Questionnaire Testing Methods
Errors Detection
Questionnaire Evaluation Criteria
Criteria for a Good Questionnaire
Using Questionnaire Appraisal System (QAS)-99
![Page 3: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/3.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
3
Main Concepts
& Terms ( Pre-test :10 terms)
Define the underlined terms in the following slide…….
![Page 4: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/4.jpg)
It has to be established that the Qes. is valid and reproducible in the
context in which it is going to be employed.
The validity of the questionnaire should have been investigated;
The questionnaire are a correct and comprehensive reflection of the
concept the questionnaire is intended to measure (content validity).
the dimensionality (factor structure) and internal consistency of the
questionnaire should have been investigated;
the test-retest reliability of the questionnaire should have been
investigated;
Responsiveness will need to be determined for questionnaires being
to be used to measure changes within individuals .
Information regarding the interpretation of questionnaire scores should
be available
Floor effects or ceiling effects less than 15%
Accepted score 7+ December 12, 2014
4
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 5: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/5.jpg)
Recommended Source-1
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
5
![Page 6: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/6.jpg)
Recommended Source-2
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
6
![Page 7: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/7.jpg)
Recommended Source-3
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
7
Questionnaire Appraisal System
(QAS-99)
QAS-99 is based on a system that developed for
Behavioral Surveillance Branch of the Centers for
Disease Control and Prevention for use in evaluating
questions for the Behavioral Risk Factor Surveillance
System (BRFSS).
http://www.cdc.gov/nccdphp/brfss
![Page 8: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/8.jpg)
Questionnaire development approach
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
8
I. Determine Analytic Objectives
• What types of data will answer the research question?
II. Develop general concepts to be covered
• List areas to be covered by questions
III. Translate concepts into questions
IV. “Appraise” questions for common pitfalls
V. Evaluate questions empirically
![Page 9: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/9.jpg)
December 12, 2014
9
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 10: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/10.jpg)
Sequence of questionnaire development & testing
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
10
![Page 11: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/11.jpg)
1- Conceptualization
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
11
Conceptual frame of the Questionnaire is more
important for new surveys, whereas in existing
surveys concepts might be already well established.
An integral part in every change in a questionnaire.
The main output are :
An entities/relationships scheme,
An area tree about the structure of the targeted
questionnaire,
List of target variables
![Page 12: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/12.jpg)
2- Questionnaire design.
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
12
First draft of a questionnaire
Appropriate wording,
Order of questions and
Definition of answering categories
![Page 13: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/13.jpg)
3- Testing.
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
13
The questionnaire needs to be tested regarding:
Wording of questions/answers, order and
structure of the questionnaire;
Problems related to translation, and cultural
background ;and
Data collection mode and the involvement of an
interviewer ……….V&R
![Page 14: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/14.jpg)
Testing the Questionnaire-Cont’
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
14
Pre-field and field methods.
It is recommended to involve respondents.
A combination of different methods is
advisable
![Page 15: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/15.jpg)
4- Revision.
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
15
After testing, making revisions to the
questionnaire
Afterwards, a new round of testing is often
essential.
This process may be repeated through two,
three or even more phases of testing.
The aim is to check if the changes are really
resulting in a higher validity and reliability of
data in relation to the specific objectives of
the survey.
![Page 16: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/16.jpg)
5- Data collection
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
16
With the implementation of the survey (either by
a pilot study or as the real survey) the process of
development and revision is terminated,
but the process of observation should be
continued via
monitoring the interviewers
![Page 17: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/17.jpg)
5-Data collection-cont’
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
17
Monitoring can be regarded as :
Continuous tool of evaluation .
The monitoring of fieldwork can be essential
for the further phase of post survey
evaluation when conducting ongoing surveys
or having implemented a new survey on full
scale.
![Page 18: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/18.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
18 ?
Questionnaire Testing
Methods
![Page 19: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/19.jpg)
Questionnaire Testing Methods
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
19
Aims Phase of testing Methods
1- Focus
groups
(FGs)
(respondent
group
discussion)
![Page 20: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/20.jpg)
Questionnaire Testing Methods
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
20
Aims Phase of testing Methods
Early stage of
questionnaire
design
1- Focus
groups
(FGs)
(respondent
group
discussion)
![Page 21: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/21.jpg)
Questionnaire Testing Methods
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
21
Aims Phase of testing Methods
To gain a
reflection of
the target
population
perspective
• To check
terms
Early stage of
questionnaire
design
1- Focus
groups
(FGs)
(respondent
group
discussion)
![Page 22: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/22.jpg)
Questionnaire Testing Methods -Cont
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
22
Aims Phase of testing Methods
2- Informal
test
(evaluation by
colleagues)
![Page 23: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/23.jpg)
Questionnaire Testing Methods -Cont
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
23
Aims Phase of testing Methods
Possible in
each
phase,
preferably at
the beginning
2- Informal
test
(evaluation by
colleagues)
![Page 24: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/24.jpg)
Questionnaire Testing Methods -Cont
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
24
Aims Phase of testing Methods
• To detect all
kinds of
mistakes:
wording,
layout, skips
etc.
Possible in
each
phase,
preferably at
the beginning
2- Informal
test
(evaluation by
colleagues)
![Page 25: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/25.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
25
Aims Phase of testing Methods
3- Expert group
Group
discussion among
design and matter
experts,
sometimes
with users
********************
********************
Questionnaire Testing Methods -Cont
![Page 26: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/26.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
26
Aims Phase of testing Methods
Initial phase of
questionnaire
development
3- Expert group
Group
discussion among
design and matter
experts,
sometimes
with users
********************
********************
Questionnaire Testing Methods -Cont
![Page 27: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/27.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
27
Aims Phase of testing Methods
• To check
concepts,
definitions,
vocabulary
against
survey’s
objectives
• To discuss data
processing
requirements
Initial phase of
questionnaire
development
3- Expert group
Group
discussion among
design and matter
experts,
sometimes
with users
********************
********************
Questionnaire Testing Methods -Cont
![Page 28: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/28.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
28 Focus Specifically on Methods
• Terms and wording of questions;
• Structure of questions;
• The response alternatives;
• Order of questions;
• Navigational rules
• Instructions to interviewers .
• Confusing layout;
• Typographical errors
3- Expert
group
Group
discussion among
design and matter
experts,
sometimes
with users
********************
********************
Questionnaire Testing Methods -Cont
![Page 29: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/29.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
29
Aims Phase of testing Methods
4- In-depth or
qualitative
interviews
(interviews with
respondents)
Questionnaire Testing Methods -Cont
![Page 30: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/30.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
30
Aims Phase of testing Methods
Early stage of
development and
testing the
questionnaire
4- In-depth or
qualitative
interviews
(interviews with
respondents)
Questionnaire Testing Methods -Cont
![Page 31: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/31.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
31
Aims Phase of testing Methods
• To evaluate
respondents’
viewpoint and
understanding of
the
questionnaire
• Explorative
nature
Early stage of
development and
testing the
questionnaire
4- In-depth or
qualitative
interviews
(interviews with
respondents)
Questionnaire Testing Methods -Cont
![Page 32: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/32.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
32
Aims Phase of testing Methods
5- Cognitive
interviews :(one-
to-one
in-depth,
structured
interviews
with specially
trained
interviewers and
researchers)
Questionnaire Testing Methods -Cont
![Page 33: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/33.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
33
Aims Phase of testing Methods
Middle of the
development
process, once a
draft
questionnaire
has
been developed
5- Cognitive
interviews :(one-
to-one
in-depth,
structured
interviews
with specially
trained
interviewers and
researchers)
Questionnaire Testing Methods -Cont
![Page 34: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/34.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
34
Aims Phase of testing Methods
• To gain
qualitative
information on
how a
questionnaire is
understood and
answered
Middle of the
development
process, once a
draft
questionnaire
has
been developed
5- Cognitive
interviews :(one-
to-one
in-depth,
structured
interviews
with specially
trained
interviewers and
researchers)
Questionnaire Testing Methods -Cont
![Page 35: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/35.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
35
Aims Phase of testing Methods
6- Observational
Interviews
(observation of
respondents while
completing a
questionnaire)
Questionnaire Testing Methods -Cont
![Page 36: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/36.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
36
Aims Phase of testing Methods
Middle of
development,
when a
tested
questionnaire
exists
6- Observational
interviews (observation of
respondents while
completing a
questionnaire)
Questionnaire Testing Methods -Cont
![Page 37: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/37.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
37
Aims Phase of testing Methods
• To check self
completed
questionnaire by
observing
potential
respondents in
the lab
Middle of
development,
when a
tested
questionnaire
exists
6- Observational
Interviews
(observation of
respondents while
completing a
questionnaire)
Questionnaire Testing Methods -Cont
![Page 38: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/38.jpg)
Questionnaire Testing Methods -Cont’
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
38
Aims Phase of testing Methods
7- Behaviour
coding
(coding behaviour
and interaction of
interviewers and
respondents )
![Page 39: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/39.jpg)
Questionnaire Testing Methods -Cont’
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
39
Aims Phase of testing Methods
After a set of
pre-field
methods have
been
conducted
7- Behaviour
coding
(coding behaviour
and interaction of
interviewers and
respondents )
![Page 40: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/40.jpg)
Questionnaire Testing Methods -Cont’
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
40
Aims Phase of testing Methods
To evaluate the
question-
answering
process by
standardized
methods
and coding
scheme
After a set of pre-
field
methods have been
conducted
7- Behaviour
coding
(coding behaviour
and interaction of
interviewers and
respondents )
![Page 41: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/41.jpg)
Error Detection
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
41
Completely identify the errors
is virtually impossible;
therefore, the goal should be to
design a testing procedure
capable to catch as many
errors as possible
![Page 42: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/42.jpg)
Errors Detection
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
42
Q-by –Q Analysis of single questions .
Testing by task Assignment of specific tasks to different testers, so that each
of them focuses on given issues.
Scenario testing Hypothesizing some real situations, entering them in the
questionnaire and checking the performance and the results.
![Page 43: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/43.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
43
Questionnaire
Evaluation
Criteria
![Page 44: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/44.jpg)
Questionnaire Evaluation Criteria
The best-known and most comprehensive
criteria are those from the Scientific
Advisory Committee (SAC) of the Medical
Outcomes Trust*….. The SAC defined
eight attributes
that deserve considerations in evaluation.
December 12, 2014
44
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 45: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/45.jpg)
Questionnaire Evaluation Criteria
(SAC)
Eight Attributes
Of the good Questionnaire
?
December 12, 2014
45
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 46: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/46.jpg)
Questionnaire Evaluation Criteria
(1) Conceptual and measurement model,
(2) Validity, (……,……,……)
(3) Reliability,
(4) Responsiveness,
(5) Interpretability,
(6) Respondent and administrative burden,
(7) Alternative forms, and
(8) Cultural and language adaptations (translations).
December 12, 2014
46
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 47: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/47.jpg)
Specific Criteria
Within each attributes, specific criteria were defined :
(1) content validity,
(2) construct validity,
(3) criterion validity,
(4) internal consistency
(5) reproducibility,
(6) responsiveness,
(7) floor and ceiling effects, and
(8) interpretability. December 12, 2014
47
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 48: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/48.jpg)
?
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
48
Validity
&
Reliability
![Page 49: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/49.jpg)
Validity
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
49
Validity addresses the
amount of systematic or
"built-in" error contained
in the measure.
![Page 50: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/50.jpg)
Reliability
Reliability refers to random error in measurement.
Reliability indicates the accuracy or precision of the measuring instrument .
The pilot test seeks to answer the question,
Does the questionnaire consistently measure whatever it measures?
December 12, 2014
50
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 51: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/51.jpg)
Reliability
Several methods ……………all involve administering
the instrument to a small sample during a pilot test.
A common procedure………is the test/re-test
The instrument is given to the same group of individuals
twice (about one week apart) and the two sets of
scores are correlated, resulting in a coefficient of
stability.
A correlation above 0.7 would indicate acceptable
reliability.
Other techniques …….Cronbach's Alpha.
December 12, 2014
51
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 52: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/52.jpg)
Specific Criteria
Within each attributes, specific criteria were defined :
(1) content validity,
(2) construct validity,
(3) criterion validity,
(4) internal consistency
(5) reproducibility,
(6) responsiveness,
(7) floor and ceiling effects, and
(8) interpretability.
December 12, 2014
52
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 53: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/53.jpg)
1-Content Validity
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
53
Content validity examines
the extent to which the concepts
of interest are comprehensively
represented by the items in the
questionnaire
![Page 54: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/54.jpg)
1- Content Validity-cont;
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
54
To evaluate content validity the following
aspects should be considered :
1. Measurement aim of the questionnaire
2. Target population
3. Concepts that intended to measure.
4. Item selection and item reduction
5. Interpretability of the items.
Completing the questionnaire should not require reading skills beyond that of
a 12-year-old to avoid missing values and unreliable answers
![Page 55: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/55.jpg)
2- Internal Consistency
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
55
Internal consistency is a measure of the extent to
which items in a questionnaire (sub)scale
are correlated (homogeneous), thus measuring
the same concept.
Internal consistency is an important measurement
property for questionnaires that intend to measure
a single underlying concept (construct) by using
multiple items.
![Page 56: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/56.jpg)
3- Construct Validity
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
56
Construct Validity is used to ensure that the
measure is actually measure what it is intended to
measure (i.e. the construct), and not other variables.
Using a panel of “experts” familiar with the
construct is a way in which this type of validity can
be assessed.
The experts can examine the items and decide what
that specific item is intended to measure.
![Page 57: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/57.jpg)
4- Reproducibility
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
57
Reproducibility concerns the degree to which repeated measurements in stable
persons (test-retest) provide similar answers.
![Page 58: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/58.jpg)
5- Criterion Validity
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
58
Criterion validity refers to
the extent to which scores on
a particular instrument
relate to a gold standard
![Page 59: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/59.jpg)
6- Responsiveness
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
59
The ability of a questionnaire to
detect important changes over time,
even if these changes are small
Responsiveness is measure of
longitudinal validity.
![Page 60: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/60.jpg)
6- Responsiveness-cont,
A longitudinal study with pre- and post-
testing is required for determining
responsiveness.
It is important in this process that the
design and potential intervention being
used represent the situation in which the
questionnaire will be used in the future.
December 12, 2014
60
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
![Page 61: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/61.jpg)
7- Floor or Ceiling Effects
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
61
The number of respondents who achieved the
lowest (floor/ground/basement) or highest
(ceiling) possible score.
Calculation : The % of subjects who achieved the
maximum score (ceiling) or the minimum score (floor).
These effects are considered when 15% of
respondents reach the ceiling or floor scores.
Their effects :implications on the questionnaire’s
reproducibility and responsiveness
![Page 62: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/62.jpg)
7- Floor or Ceiling effects
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
62
The number of respondents who achieved the lowest or highest possible score.
If floor or ceiling effects are present, it is likely that extreme items are missing in the lower or upper end of the scale, indicating limited content validity. As a consequence, subjects with the lowest or highest possible score cannot be distinguished from each other, thus reliability is reduced.
Furthermore, the responsiveness is limited because changes cannot be measured in these patients.
![Page 63: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/63.jpg)
A floor effect
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
63
A Floor Effect is when most of your
subjects score near the bottom.
There is very little variance because
the floor of your test is too high.
The question is too hard for the
group you are testing.
![Page 64: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/64.jpg)
A ceiling effect
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
64
A Ceiling Effect :Most of subjects
score near the top. There is very
little variance because the ceiling of
your test is too low.
The question is too easy for the
group you are testing.
![Page 65: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/65.jpg)
8- Interpretability
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
65
Interpretability is defined as
the degree to which one can
assign qualitative meaning to
quantitative scores.
![Page 66: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/66.jpg)
Systematic Evaluation of Quest.
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
66
How do we find questionnaire
problems????????
By conducting a s tructured
expert review
( Technical Review )
Example: QAS-99
![Page 67: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/67.jpg)
Questionnaire Appraisal System
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
67
Questionnaire Appraisal System
(QAS-99)
QAS-99 is based on a system that developed for
Behavioral Surveillance Branch of the Centers for
Disease Control and Prevention for use in evaluating
questions for the Behavioral Risk Factor Surveillance
System (BRFSS).
http://www.cdc.gov/nccdphp/brfss
![Page 68: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/68.jpg)
Questionnaire Appraisal System
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
68
For structured expert reviews a set of criteria
(coding schemes ) by which each question is
to be examined are implemented.
The coding scheme needs to be filled in for
each question and consequently the draft
questionnaire needs to be completed by the
standardized coding categories .
![Page 69: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/69.jpg)
QAS-99 ( Items )
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
69
1. PROBLEMS WITH READING:.
2. PROBLEMS WITH INSTRUCTIONS
3. PROBLEMS WITH ITEM CLARITY:
4. PROBLEMS WITH ASSUMPTIONS
5. PROBLEMS WITH KNOWLEDGE/MEMORY:
6. PROBLEMS WITH SENSITIVITY/BIAS:
7. PROBLEMS WITH RESPONSE CATEGORIES
![Page 70: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/70.jpg)
1. PROBLEMS WITH READING:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
70
Determine if it is difficult for the interviewers to read the question uniformly to all
respondents.
1a – What to read: Interviewers may have difficulty determining what parts of the question are to be read.
1b – Missing information: Information the interviewer needs to administer the question is not contained in the question.
1c – How to read: Question is not fully scripted and therefore difficult to read.
![Page 71: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/71.jpg)
2. PROBLEMS WITH INSTRUCTIONS:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
71
Look for problems with any introductions,
instructions, or explanations from the
respondent’s point of view.
2a – Conflicting or inaccurate instructions,
introductions, or explanations.
2b – Complicated instructions, introductions, or
explanations
![Page 72: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/72.jpg)
3. PROBLEMS WITH ITEM CLARITY:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
72
Identify problems related to communicating the intent or meaning of the question to the
respondent 3a – Wording: The question is lengthy, awkward,
ungrammatical, or contains complicated syntax.
3b – Technical terms are undefined, unclear or complex.
3c – Vague: The question is vague because there are multiple ways in which to interpret it or to determine what is to be included and excluded.
3d – Reference periods are missing, not well specified, or are in conflict.
![Page 73: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/73.jpg)
4. PROBLEMS WITH ASSUMPTIONS:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
73
Determine there are problems with assumption if s made or the underlying logic.
4a – Inappropriate assumptions are made about the
respondent or his/her living situation.
4b – Assumes constant behaviour: The question
inappropriately assumes a constant pattern of
behaviour or experience for situations that in fact vary.
4c – Double-barrelled question that contains multiple
implicit questions.
![Page 74: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/74.jpg)
5. PROBLEMS WITH
KNOWLEDGE/MEMORY:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
74
Check whether respondents are likely to not know or have trouble remembering
information.
5a – Knowledge: The respondent is unlikely to
know the answer.
5b – An attitude that is asked about may not
exist.
5c – Recall failure.
5d – Computation or calculation problem
![Page 75: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/75.jpg)
6. PROBLEMS WITH SENSITIVITY/BIAS
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
75
: Assess questions for sensitive nature or wording, and for bias.
6a – Sensitive content: The question is on a topic
that people will generally be uncomfortable
talking about.
6b – A socially acceptable response is implied.
![Page 76: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/76.jpg)
7. PROBLEMS WITH RESPONSE
CATEGORIES:
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
76
Asses the adequacy of the range of responses to be recorded.
7a – Open-ended question that is inappropriate or difficult.
7b – Mismatch between question and answer categories.
7c – Technical terms are undefined, unclear, or complex.
7d – Vague response categories.
7e – Overlapping response categories.
7f – Missing response categories.
7g –Illogical order of response categories
![Page 77: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/77.jpg)
QAS-99 ( Reference)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
77
Question Appraisal System QAS-99 . By:
Gordon B.Willis and Judith T. Lessler . Research
Triangle Institute Suite 420 6110 Executive
Blvd. Rockville, MD 20852 August, 1999
http://www.cdc.gov/nccdphp/brfss
Biemer, P.P. and Lyberg, L.E. (2003).
Introduction to Survey Quality. Hoboken, New
Jersey: John Wiley&Sons
![Page 78: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/78.jpg)
Cited References
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
78 http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf
http://www.joe.org/joe/2007february/tt2.php
https://readability-score.com/
http://www.joe.org/joe/1990summer/tt2.php
http://www.emgo.nl/kc/preparation/research%20design/8%20Questionnaires%20selecting,%20translating%20and%20validating.htm
Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002;11:193e205.
http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf
http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf
![Page 79: How to assess questionnaires](https://reader033.vdocument.in/reader033/viewer/2022052911/559ebc141a28ab782a8b46b4/html5/thumbnails/79.jpg)
December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
79
Thank You