Basics of Survey & Scale DesignChan Kulatunga-Moruzi, PhD

Department of Family MedicineMcMaster University

Agenda

Presentation: Overview of scales and surveys

Survey and scale - similaritiesSurvey and scale – differences

Guidelines to scale and survey construction

Group work: Identify common mistakes in survey questions

Surveys & Scale: Similarities

Tools of research Usually ask a series of questions Gather data pertaining to central construct May contain sub-constructs

Survey: Construct- Perception of PAs Scale: Construct - Professional Burnout

Surveys & Scale: Similarities

Often use rating scalesLikert-type, semantic differential

Often self-administered Often based on self-report Similar issues/problems

Social desirability bias – jeopardize validity

Scale: Description

Also known as an index or inventory Responses: categorical, more likely rating scale* Combine an individual’s data to one meaningful

number (interval level)**– Eating Disorders Inventory– Quality of Life Index– Minnesota Multiphasic Personality Inventory– Suicidal Ideation Scale

Scale: Function

Used to describe a population/construct Overall scores or sub-scores used to make– Inferences– Identify, describe and compare –Make decisions (e.g. treatment)– Further research

Scale: Construction

Knowledge of constructDepression: symptoms DSM, ICD-9, differentials

Knowledge of psychometricsReliability: test-retest, internal consistency, discriminationValidity: construct, external (concurrent/predictive)Reliability sets upper limit of validity

Scale: Construction

Research to find existing measurement scale(s)

Use previously validated scale Amend previously validated scale to suit your

needs

Survey: Description

Response format: mixture preferredRating scale - Likert/semantic differentialMultiple choice-categoricalRank orderOpen-ended

Do not combine individual’s data to produce one meaningful number

Survey: Function

Often used simply to describe a population Used to inform policy /administration, Used for program evaluation

Individual questions may be used to make inferences, compare cohorts/populations

Survey: Construction

Requires some knowledge of construct May be exploratory to learn about the construct

Reliability & Validity assumed:– by securing representative sample– by asking well written questions– by using well constructed response options– by sound analyses

Survey & Scale Development

Broad general topic Narrow down focus

- Identify research question(s)- Operationalize/define concepts

Objective: What is it that you want to know? Can you state your objective clearly and succinctly? What information is necessary to meet objective? Start with the end in mind

Survey & Scale Development

Each question addresses research question Each question relevant to objectives Limited time/Survey fatigue

Anticipate results you might receive Think about how you might analyze data Will help to construct better questions Will help use best questions formats

Survey & Scale Development

Keep your respondents in mind Who will complete your survey?

representative sample Respondents able to understand the question? Respondents able to answer the question? How can you make it easy to complete? Are questions relevant to all respondents?

Question Design: “BOSS”

Be BRIEF Keep questions short and to the point Avoid long list of response alternatives to choose

from or to rank order Take time to edit

meaningvisual clutter

Be OBJECTIVE Ensure questions are neutral

Avoid leading questionsAvoid built in assumptionsAvoid loaded questions

Be cognizant of the possible impact of words chosen and question phrasing/framing

Question Design: “BOSS”

Be Simple Use simple language Avoid jargon and technical terminology Avoid double-barrel questions

Question Design: “BOSS”

Be Specific Avoid broad questions May be interpreted differently by respondents

May need to define/specify what you mean

Question Design: “BOSS”

Group Work: 4 Cases

Identify any problems you see with the item Re-write the items to address problems.

Pay attention to stem & response options.

Is there a better way to ask the question to meet the objectives of the research?

Case 1: Age

Stem provides no context for the question a. To which age category do you belong? (nominal level)b. How old are you? (interval level)

What is your date of birth?a. Easy to fill, increase response rate, personal questionb. Enable better analysis, option to group later

Case 1: Age

Problems with response options: inconsistent - words/hyphens not exhaustive – older/younger students not exclusive – 16 included in 2 options intervals not equal - 3 vs. 4 years

Case 2: Communication Skills

Language used in the questionVague, wordy, jargon/too advanced

Leading question Researcher assumptions

“metamorphosized over the duration of…”

Expects students are able to remember and accurately report back from the beginning

How might the researcher better meet his objectives?

Student rate his/her communication skills after each patient encounter through out year

SP rate students’ communication skills after each patient encounter though out year

Video tape students throughout the year, ask blinded expert to rate communication skills

Case 2: Communication Skills

Case 3: Engagement & Learning Outcomes

Vague stem Which of these activities do you engage in?What do we mean by engage in?

Dichotomous response options (yes/no)Reduce variability, reliability, validity

Scaled response (5-7 pts)increase variability, reliability, validity

Inconsistent pronouns (you/I)

Double barreled questions (class & office hrs)

Improper punctuation (?)

Case 3: Engagement & Learning Outcomes

Case 4: Diversity & Barriers to Higher Ed

Loaded question Researcher’s assumptions Leading question

Double barreled question

Response options (odd vs. even number)