developing a questionnaire for use in outcomes assessment

7/30/2019 Developing a Questionnaire for Use in Outcomes Assessment

1/22

DEVELOPING A QUESTIONNAIRE FOR

USE IN OUTCOMES ASSESSMENT

Kathleen A. Martin, D.P.E.Office of Institutional Research & Assessment

Bucknell University


2/22

POINT OF CLARIFICATIONTerms survey and questionnaire are often usedinterchangeably

Survey refers to a descriptive research

technique or methodology (also includesinterviews, and focus groups)

Questionnaire refers to a paper and pencilinstrument through which information is

obtainedFor the purposes of outcomes assessment,questionnaires are frequently developed to surveya population


3/22

QUESTIONNAIRES & ASSESSMENTUseful for surveying groups of any sizeTypically designed to determine opinions, attitudes

or present practicesLimitation: rely on self report information v.observed behavior

Challenges to validity become a concern


4/22

VALIDITY CHALLENGESValidity: the extent to which the test measures what itpurports to measure

Internal Validity: Relates to instrument (questionnaire)

quality. Are the appropriate questions asked clearly andlogically?

External validity: can the results be generalized?

Measurement error: difference between thecharacteristics of your sample and the characteristics ofthe population

Non-response bias: answers of respondents do notmirror the characteristics of those who refusedparticipation.


5/22

MAXIMIZING VALIDITYCreate a questionnaire that is clear and logicalSupport content relevance and content representativeness

Content Relevance: are the questions relevant to the

purpose of the questionnaire? Unimportant contentshould be eliminated.

Content Representativeness: Are the questions anadequate representation of the universe of possiblequestions?

Solicit the input of experts in scale development and thecontent of the questionnaire for an objective evaluation

Select a large, representative random sample from thepopulation of interest or census the finite population

Increase response rate using formal follow up procedures


6/22

INITIAL CONSIDERATIONSIs the development of a new questionnaire . . .Necessary?

Feasible?Select mode of administration

Web Based?

Paper and pencil?

Combination?


7/22

QUESTIONNAIRE PLANNING1. Determine the ObjectiveWhat do you want to ask and of whom?

Consider how the data will be analyzed. What information

is needed?2. Determine the Mode of Administration

Paper and pencil

Cost effective, but low response rate

Web / E-mail based

Cost effective, higher response rate, some suspicionof the medium

Combination

Can be administratively cumbersome


8/22

QUESTIONNAIRE PLANNINGDetermine the Sampling MethodologyTo whom will you send the questionnaire? Who cansupply the information?

Will everyone be surveyed?

ALL current students? ALL alumni?

If yes, this is a census of the finite population

The selection of a representative sample of the

population may be more effective based on time and costof administration

Decision depends on the size of the finite population


9/22

CONSTRUCTING the QUESTIONNAIREConsider what objective each question measuresThree to five questions per objective is recommended

How many questions?

Find a balance: long enough to obtain the necessaryinformation but short enough so respondents wont loseinterest.

Questionnaires that are too long wont be completed

Goal: 25-30 minutes MAX


10/22

DESIGNING THE QUESTIONNAIREQuestion Formats: Open Ended QuestionsAllow respondents to express feelings and expand onideas.

Question should be phrased to avoid a one wordresponse.

What aspects of the Sociology program bestprepared you for your current position?

Open ended questions at the beginning of thequestionnaire may be discouraging to respondentinclude one or two at the end

Analysis of responses can be challenging


11/22

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed QuestionsRanking

Forces the respondent to rank order responses based on a

value judgment.Rank the following items with regard to how you spend timeat Bucknell from 1 (spend the most time) to 5 (spend theleast time).

_____ Studying

_____ Socializing_____ Volunteering

_____ Working at a part time job

_____ Exercising


12/22

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed QuestionsChecklist

A number of possible answers are provided and the

respondent is asked to choose one or choose all that apply.How did you finance your Bucknell University education?Choose all that apply.

_____ Parental contribution

_____ Student Loan

_____ Work Study

_____ Off campus employment

_____ Other (please specify): _____________


13/22

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed QuestionsScaled items

Very commonIndicate strength of agreement or disagreement with a

statement.

Numbers are assigned to each response on the continuum

The English curriculum at Bucknell University adequately

prepared me for my current position.

1 2 3 4 5


14/22

QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed QuestionsSuggestions for writing scaled items

Use positively worded or negatively worded itemsconsistently

Use an odd number of responses along the continuum to

allow for a neutral response

Anchors may be different based on focus of the item

(Strongly Agree

Strongly Disagree; Frequently

Never;Very Good Very Poor)

Keep items short


15/22

QUESTIONNAIRE DEVELOPMENTQuestion Formats: CategoricalTypical demographic questions:

Yes/No/?, Male/Female, F/S/J/S/G

Answers fall into certain categories

respondents pick theone that applies to them

Analyzed using frequency, proportion, percentages

2 considerations:

Questions on a continuum should stay on a continuum if

they will be directly analyzedExample: Age allow respondent to write in theirage rather than check off a box for their age range

Categories must be collectively exhaustive & mutuallyexclusive (next slide)


16/22

QUESTIONNAIRE DEVELOPMENTProblematic:

Number of hours spent

studying per week:

1821

2124

24 27Categories are notmutually exclusive

Corrected:

Number of hours spent

studying per week:

18 21

22 25

26 29


17/22

IMPORTANT CONSIDERATIONSAPPEARANCE AND DESIGNHave clear directions that are age appropriate & easy tofollow

Format, size, reproduction should facilitate completionNo less than 12 pt

Times, Arial, Courier

PILOT STUDY

Give the questionnaire to a few people to try it out

Have them evaluate readability and clarity of questions

Estimate time required to complete the questionnaire

Suggestions for revisions


18/22

SENDING IT OUTINITIAL CONTACT / COVER LETTERShort & to the point

Identify person by nameConvince respondent that participation is important

Ensure confidentiality

Deadline for return of questionnaire

SASE for return (mailed questionnaires only)


19/22

Institutional approvalInstitutional Review BoardResearch involving human subjects must be approved bythe Bucknell University IRBFor more information, visit the IRB website at

http://www.departments.bucknell.edu/inst_research/IRB/irb.htm

Campus Survey CoordinationThe timing of all university-wide surveys must becoordinated through the Campus Survey CoordinationGroup.For more information, visit the CSCG website athttp://www.departments.bucknell.edu/inst_research/cscg.htm
http://www.departments.bucknell.edu/inst_research/IRB/irb.htmhttp://www.departments.bucknell.edu/inst_research/IRB/irb.htmhttp://www.departments.bucknell.edu/inst_research/IRB/irb.htmhttp://www.departments.bucknell.edu/inst_research/cscg.htmhttp://www.departments.bucknell.edu/inst_research/cscg.htmhttp://www.departments.bucknell.edu/inst_research/cscg.htmhttp://www.departments.bucknell.edu/inst_research/cscg.htmhttp://www.departments.bucknell.edu/inst_research/cscg.htmhttp://www.departments.bucknell.edu/inst_research/IRB/irb.htmhttp://www.departments.bucknell.edu/inst_research/IRB/irb.htm


20/22

FINAL THOUGHTSFOLLOW UP PROCEDURESSend to all participants to preserve confidentiality (If youhavent already returned the questionnaire..)

May include a second copy if available

Usually done in 2 week intervals

RESPONSE RATE FOR MAILED QUESTIONNAIRES

Typically low - 30% is considered good

Higher response rate with web-based questionnaires

Good follow-up increases response rate


21/22

SUMMARY OF STEPS1. Determine the Objective (consider analyses)2. Determine the Mode of Administration

3. Determine the Sampling Methodology

4. Construct the Questionnaire

5. Institutional Approval

6. Conduct the Pilot Study

7. Write the Initial Communication

8. Send the Questionnaire9. Follow up

10. Analyze the Results


22/22

FOR MORE INFORMATIONContact Kathy Martin @ 71960 [email protected]

developing a questionnaire for use in outcomes assessment

Documents