survey participation

8/14/2019 Survey Participation

1/18

Survey Participation:

A Study of Student Experiences and Response Tendencies

Allison M. Ohme

Heather elly !saacs

"ale #. Trusheim

Presented at the 45thAnnual Forum of the Association for Institutional ResearchSan Diego, CAJune 1, 25


2/18

A!stract

"ecause there are a #ariet$ of factors that can contri!ute to lo%er than e&'ected student sur#e$res'onse rates, researchers at a four($ear 'u!lic uni#ersit$ de#elo'ed a multi(method stud$ tounderstand students) e&'eriences and res'onse tendencies %ith institutionall$(related sur#e$s*

+his stud$ utilied focus grou's, tele'hone and in('erson inter#ie%s of small sam'le sies andsought to understand the num!er and t$'e of sur#e$s students recei#ed in an academic $ear, theirreasons for either res'onding or not res'onding to a sur#e$, and their suggestions to increasestudent res'onse rates* +his 'a'er discusses the !ac-ground, methodolog$, and findings of thestud$*

Page 2


3/18

!ntroduction

In fall 2., the /ffice of Institutional Research and Planning at the 0ni#ersit$ ofDela%are de#elo'ed and conducted a stud$ to understand students) e&'eriences %ithinstitutionall$(related sur#e$s and their res'onse tendencies to such sur#e$s* +he /ffice of

Institutional Research and Planning t$'icall$ utilies large sam'le sies %hen distri!uting 'a'erand on(line sur#e$s* +his stud$, ho%e#er, used three different methods of small sam'le sies togather ualitati#e information from students regarding ho% the$ e&'erience and res'ond tosur#e$s at the 0ni#ersit$ of Dela%are* +his 'a'er discusses the !ac-ground, methodolog$, andfindings of the stud$*

$ac%&round and O'(ectives

+he /ffice of Institutional Research and Planning occasionall$ administers sur#e$s tostudents to assist in the e#aluation of 'rograms and ser#ices at the 0ni#ersit$ of Dela%are 0D3*+hese sur#e$s include the Career Plans Sur#e$, Student /'inion Sur#e$, and the conomic

Im'act Stud$, among others* +he sur#e$s are t$'icall$ 'a'er uestionnaires sent through themail to a sam'le of students* /#er the 'ast se#eral $ears, the /ffice of Institutional Research hasseen declining res'onse rates to 'a'er sur#e$s at 0D* In 1, an n#ironmental AttitudeSur#e$ %as administered to students using a %e!(!ased methodolog$ to e&'lore the ne%methodolog$ and its resulting res'onse rates* Des'ite the change from 'a'er to on(lineadministration, the stud$)s ne% methodolog$ $ielded a res'onse rate !elo% 256, %hich is alo%er res'onse rate than %hat students res'ond to traditional 'a'er sur#e$s at 0D*

Research and %riting com'aring the 'a'er and on(line methodologies and theirtechniues is discussed regularl$ in the field of Institutional Research 7and%er-, Carson 8"lac-%ell, 29 Porter, 243* +hese resources gi#e researchers a #ariet$ of tools to em'lo$ asthe$ e&'eriment %ith sur#e$ administration at their institutions* Sa&, :ilmartin and "r$ant)sstud$ found lo%er res'onse rates %ith on(line sur#e$s as com'ared to 'a'er sur#e$s, ho%e#er intheir literature re#ie% the$ note that effecti#eness of !oth methods can fluctuate and elicit higheror lo%er res'onse rates de'ending on the s'ecific sur#e$ techniues utilied in the stud$ 2.3*

/ther issues in addition to a sur#e$)s methodolog$ also affect student res'onse rates*:ro#es, Cialdini and Cou'er descri!e the #aried emotional and 's$chological conce'ts thatinfluence sur#e$ 'artici'ation 123* Res'ondents) mood and 'erce'tions of the issuesaddressed %ithin the sur#e$ can ha#e a considera!le effect on res'onse rates* /ther nota!lefactors include ho% 'artici'ants #ie% the authorit$ administering the sur#e$ and %hether the$feel the$ ha#e am'le o''ortunities to share their o'inion through other means* Students %ho feelthe$ ha#e range of o''ortunities or %ho feel the !urden of these o''ortunities ma$ feel ;o#er(sur#e$ed*< +his has !een sho%n to ha#e a 'rofound effect on their res'onse rate as %ell as the#alidit$ of their res'onses :ro#es et al*, 129 Asiu, Antons 8 Flut, 1=3*

In fall 2., the /ffice of Institutional Research and Planning ac-no%ledged theseconcerns and conducted a series of focus grou's, tele'hone and in('erson inter#ie%s among asam'le of 0ni#ersit$ of Dela%are undergraduates* +he research o!>ecti#es %ere to disco#er ho%man$ sur#e$ reuests t$'icall$ im'act an undergraduate and %hat factors ma-e students li-el$ to

Page .


4/18

res'ond or not res'ond to an unsolicited sur#e$* "ecause of the issues in#ol#ed %ith studentsur#e$ 'artici'ation discussed a!o#e, %e chose focus grou's and inter#ie%s as our methods ofdata collection des'ite the amount of time and effort in#ol#ed in their coordination andadministration* ?e e&'ected that the focus grou's %ould $ield #alua!le findings !ecause oftheir recognied a!ilit$ to allo% and encourage discussion %here 'artici'ants can e&'ress and

dee'en their thoughts throughout the duration of the discussion "loor, Fran-land, Rhomas 8Ro!son, 213* Indeed, focus grou's are freuentl$ used in multi(method research and canoften 'ro#ide findings %ith more de'th and !readth than those of a t$'ical sur#e$ uestionnaire@organ, 13* Deciding to include the tele'hone inter#ie%s B and su!seuentl$ adding the in('erson inter#ie%s B %e designed this stud$ in order to cross(#alidate this data %ith the focusgrou's results* "$ allo%ing students to ma-e comments and suggestions regarding sur#e$administration and ho% to increase res'onse rates, this 'ro>ect see-s to 'ositi#el$ influencefuture sur#e$ design, administration and res'onse rates at the 0ni#ersit$*

Methodolo&y

+he initial research design for this stud$ incor'orated t%o different methodologies Bfocus grou's and tele'hone inter#ie%s B to collect information from a total of 1 students* +hefocus grou' method %as chosen for its a!ilit$ to encourage a forum for students to share theiro'inions and engage in candid discussion of the issues* ?e included the additional method oftele'hone inter#ie%s since tele'hone calls %ere used to in#ite students to 'artici'ate in the focusgrou's, and %e could therefore offer students %ho could not attend a focus grou' the o''ortunit$to still contri!ute to the stud$* In('erson inter#ie%s %ere incor'orated later in the stud$ toaccommodate challenges 'osed !$ the initial methodolog$ and to create a more ro!ust set offindings*

A random sam'le of 5 students %as dra%n re'resenting continuing students %ho %erematriculated during the 're#ious academic $ear 22(2.3 as full(time undergraduates* +hesestudents %ere contacted !$ tele'hone and as-ed a screening uestion to determine if the$ hadrecei#ed at least one unsolicited sur#e$ from the 0ni#ersit$ %ithin the 'ast academic $ear* Ifthe$ ac-no%ledged that the$ had recei#ed a sur#e$, the student %as in#ited to 'artici'ate in oneof fi#e focus grou's held during /cto!er and o#em!er 2.* ach focus grou' %ould containa ma&imum of 1 students and meet in 7ullihen 7all during lunchtime or at 5 'm %ithrefreshments 'ro#ided* +he focus grou' discussion uestions as-ed students a!out the num!erand t$'e of sur#e$s recei#ed, the im'act the sur#e$s ha#e on them, and their reasons for eitherres'onding or not res'onding to a sur#e$* +he students %ere also gi#en a chance to sharesuggestions for increasing student res'onse rates see A''endi& A for the list of the uestions3*During the tele'hone call, if the student declined or could not attend a focus grou', the$ %eregi#en the o'tion of res'onding to the same si& uestions !$ 'artici'ating in the tele'honeinter#ie%* /nce 5 tele'hone inter#ie%s too- 'lace, this 'ortion of the methodolog$ %asconsidered com'lete* Along %ith their res'onses, %e collected the follo%ing demogra'hicinformation a!out the student 'artici'ants from the Student Information S$stem gender, class,onEoff cam'us housing, and range of :PA*

In addition to the refreshments offered during each focus grou', students %ere informedthat !$ 'artici'ating the$ %ould !e entered into a dra%ing to %in a 1 gift certificate from the

Page 4


5/18

Do%nto%n e%ar- Partnershi', allo%ing them to redeem the certificate at 'artici'atingrestaurants and !usinesses on @ain Street and in the e%ar- Sho''ing Center* +he sameincenti#e %as offered to those student res'ondents 'artici'ating in the tele'hone inter#ie%s*

After se#eral da$s of tele'hone calls in#iting students to 'artici'ate in the focus grou's,

%e had 'rogressed through a large 'ro'ortion of our dra%n sam'le $et $ielded onl$ a #er$ small'artici'ant list* +he small 'ro'ortion of 'artici'ants com'ared to the sam'le sie ma$ ha#e !eendue in 'art to the difficult$ in reaching students o#er the 'hone, and !ecause %e chose not tocontact those %ho had an out of state area code listed* +o e&'and our student sam'le, %e dre%an additional sam'le of 1, students* In addition, %e decided to in#ite students to 'artici'atein the stud$ e#en if the$ ans%ered ;no< to the screening uestion as-ing if the$ had recei#ed asur#e$ %ithin the 'ast academic $ear* /'ening the stud$ to these students seemed reasona!lesince it too- great effort to reach students on the tele'hone and %e had !een e&cluding a num!erof students from 'artici'ating !ecause the$ ans%ered ;no< to the screening uestion* ?e alsoconcluded that these students, although not ha#ing recei#ed a sur#e$ in the 'ast $ear, could stillha#e had 'rior e&'erience %ith sur#e$s, or an association %ith other students recei#ing sur#e$s*

+o ensure that %e %ould still ha#e the inclusion of students in the focus grou's and tele'honeinter#ie%s %ho had recei#ed a 0ni#ersit$ sur#e$, %e dre% a third and final sam'le of students%ho had !een sent a sur#e$ from the conomic Im'act Stud$ earlier that fall semester* +heinclusion of these students amounted to a''ro&imatel$ 6 of the 5= students %ho 'artici'ated inthe focus grou's and tele'hone inter#ie%s* +hose students %ho re'orted recei#ing ero sur#e$sin the 'ast $ear accounted for a''ro&imatel$ 26 of these 'artici'ants*

Although %e initiall$ secured adeuate student 'artici'ants for each focus grou', at leastfi#e students from each grou' cancelled after recei#ing an e(mail reminder 'rior to the grou')smeeting, or failed to sho% u' for the grou' %ithout notice* +herefore the focus grou's each hadlo%er turnouts than %e e&'ected des'ite the #er!al commitments %e recei#ed from the student'artici'ants* In one focus grou' no students %ho had signed u' actuall$ attended %hile threeother focus grou's had onl$ a small num!er of student 'artici'ants* After the difficult$ %ee&'erienced in the first four focus grou's due to cancellations and no(sho%s, %e decided tocancel the final grou' meeting %hich, as of $et, had onl$ a small interest* +his resulted in ourconducting onl$ three focus grou's %ith a total of eight 'artici'ants* @an$ of the students %hodid notif$ us of their need to cancel 'artici'ation in a focus grou' ans%ered the tele'hone sur#e$and !ecame 'artici'ants in that section of the stud$ instead*

"ecause of these circumstances %e decided to add a third method of data collection in('erson inter#ie%s* ?ith onl$ eight student 'artici'ants in the focus grou' 'ortion of the 'ro>ect,%e decided to a''roach students in('erson in order to B at the #er$ least B meet our originalo!>ecti#e of 1 total research 'artici'ants* As in the tele'hone calls and inter#ie%s, %e utiliedthe hel' of our undergraduate em'lo$ees to a''roach and inter#ie% grou's of students in theFood Court of the +ra!ant 0ni#ersit$ Center* +his effort resulted in 5 student 'artici'ants%ithin this section of the stud$* ?e gathered the same demogra'hic information from thesestudents that %e had collected from the focus grou' and tele'hone inter#ie% 'artici'ants seeA''endi& " for a demogra'hic !rea-do%n of the 'artici'ants3* ?e also as-ed the 'artici'antsthe same uestions as those as-ed in the focus grou's and tele'hone inter#ie%s* As an incenti#eto tal- %ith us in 'erson, %e offered each 'artici'ant a 5* cou'on redeema!le for an$

Page 5


6/18

'urchase at the Food Courts in !oth the +ra!ant 0ni#ersit$ Center and the Per-ins StudentCenter*

Summary and )onsiderations 'y Method

+he follo%ing section of this 'a'er gi#es a detailed summar$ of findings %ithin each ofthe three research methods em'lo$ed in this stud$* Section A descri!es the focus grou' findings,and sections " and C, res'ecti#el$, descri!e the tele'hone and in('erson inter#ie% findings*A. *ocus +roups::i#en their inclusi#e nature and time allo%ance, the focus grou's $ielded themost detailed and com'rehensi#e discussions of student e&'eriences %ith sur#e$s andsuggestions for increasing res'onse rates* +he nature of the research uestions %as such thatthere %ere no o#ert disagreements or dissension %ithin the focus grou's* Students shared theirindi#idual and sometimes uniue thoughts as the$ made contri!utions %ithin their focus grou'*After a summar$ of factual information descri!ing the sur#e$s students recei#ed and com'leted,'oints e&tending to comments offered !$ students in other focus grou's are 'resented !elo%*

Also included in the summar$ are those indi#idual student comments stated %ith a greater degreeof im'ortance or significance %ithin their res'ecti#e grou'*

Students in all three focus grou's had recei#ed numerous sur#e$s in the 're#iousacademic $ear %ith students each re'orting a range !et%een three to t%el#e sur#e$s*/nl$ a single student in one focus grou' did not com'lete an$ of the sur#e$s he recei#ed,%hile all other students com'leted at least one or more of those recei#ed* Studentsagreed that those that the$ recei#ed originated from #arious de'artmental offices, along%ith academic 'rograms such as the 7onors Program and the 0ndergraduate Research'rogram*

All three focus grou's descri!ed similar reasons for com'leting sur#e$s* +hese included;hel'ing< the 0ni#ersit$ !$ 'ro#iding one)s o'inion %hen as-ed for it and see-ing toinfluence changes at 0D %here the su!>ect matter and goals of the sur#e$ a''lied to theres'ondent* In one focus grou', a student commented on feeling self(em'o%ered throughsur#e$s and 'leased that 0D is ;'utting effort into finding out %hat 'eo'le thin-*< +heother student in that focus grou' filled out sur#e$s !ecause the$ focused on matters thataffected them 'ersonall$, namel$ those from Dining Ser#ices*

In the other t%o focus grou's man$ of the students commented on the num!er of thedining sur#e$s as %ell as the #arious %a$s the$ reach students* ?hile man$ are sent #iae(mail, some students mentioned the$ en>o$ed com'leting 'a'er sur#e$s %hile in a

dining hall %ith an incenti#e cand$ !ar3 gi#en if the student returned the sur#e$immediatel$* /ne student in a focus grou' mentioned t%ice that he %as ;anno$ed< !$the num!er of dining sur#e$s he recei#ed #ia e(mail* Students in t%o of the focus grou's'ointed out that the$ don)t fill out multi'le sur#e$s from the same de'artment if the$don)t see the changes im'lemented since the 're#ious sur#e$*

In terms of the mechanics of a sur#e$, all the students in the three focus grou'scommented on sur#e$s the$ %ould not fill out !ecause the$ %ere too long and in#ol#ed*

Page G


7/18

A short sur#e$ %as defined !$ a cou'le of students as 1 to 15 uestions, and ta-ing onl$. seconds to com'lete* De'ending on the student, some 'referred multi'le choiceuestions %hile others %ould rather ha#e s'ace to %rite in res'onses and comments*

@ost of the students agreed that there %ere !oth ad#antages and disad#antages to !oth

'a'er and e(mail sur#e$s* ?hile students can %or- #er$ fast on the internet and;multitas-< %hile the$ are com'leting other com'uter %or-, some disad#antages of e(mail sur#e$s include the o#era!undance of S'am in student in!o&es, and on(line sur#e$sthat are ;dee' and com'licated< i*e*, too long or consisting of multi'le 'ages andsur#e$s that don)t let $ou -no% ho% man$ 'ages are reuired to com'lete the sur#e$3*Students agreed that the$ %ould not automaticall$ delete or !e s-e'tical of a sur#e$ sent#ia e(mail to their 0IH ID num!er address rather than their username3 !ecause'rofessors use their 0IH ID num!er %hen sending messages related to class*According to the students, the$ %ould !e discouraged from res'onding if a sur#e$ %assent as a mass email to a large list of other num!ered accounts* /ne student mentionedthat since he had 're#iousl$ recei#ed S'am to his num!ered account, he might dou!t the

source of a sur#e$ sent to that account claiming to !e from the 0D administration* /therstudents s'o-e of their 'reference of 'a'er sur#e$s, and ho% the$ %ere more li-el$ to fillout a 'a'er sur#e$ if it %as gi#en to them in 'erson %ith an immediate incenti#e* #en%hen students ha#e the intention to com'lete a 'a'er sur#e$ sent in the mail, man$ ofthem agreed that the$ find other mail or home%or- to !e their 'riorit$ once the$ get totheir dorm room or a'artment and therefore set the sur#e$ aside or thro% it a%a$*

Students had man$ e&am'les of !eing a''roached or #olunteering to com'lete an in('erson sur#e$ %hile on cam'us* +hese sur#e$s %ere accom'anied !$ an incenti#e tota-e %ith them such as the dining sur#e$ mentioned a!o#e3* Se#eral students agreedthat as long as the$ had 15 minutes of free time in the Student Center or near the !us sto',

that the$ %ould not mind filling out a sur#e$* /ne student mentioned that he %ouldcom'lete a sur#e$ as long as he %asn)t on his %a$ to class and could com'lete and returnthe sur#e$ immediatel$ as he %ould not remem!er to send it in later*

?ithin each focus grou', all of the students agreed that incenti#es ma-e it more li-el$ forthem to com'lete a sur#e$* ?hen as-ed %hat incenti#es are !est, the students mentionedmone$, and readil$ a#aila!le food instead of a cou'on to redeem later* /ne student notedthat a 'rie dra%ing is not enough incenti#e for students since the$ thin- their chancesare #er$ lo% gi#en the sie of the student !od$*

All three focus grou's s'o-e a!out the im'ortance of -no%ing ho% and %hen the sur#e$

results %ill !e used and 'ossi!l$ im'lemented* /ne student used an e&am'le of ho% herecei#ed the results of a 's$cholog$ research sur#e$ ma-ing him ;feel more in#ol#edect matter im'ortantto students andEor ;student oriented*uniors, and t%o seniors* Se#en out of the eight studentsli#ed on cam'us, and half of the students had :PAs higher than .*5* Si& students ma>ored in theCollege of Arts and Sciences, %hile each of the remaining t%o students ma>ored in the Collegeof 7uman Ser#ices, ducation and Pu!lic Polic$ and the erner College of "usiness andconomics, res'ecti#el$*

Page 1

APPE


18/18

Telephone Intervie' Sample 2n5+047

?ithin the fift$ tele'hone inter#ie% 'artici'ants, 546 %ere male %hile 4G6 %ere female*+%ent$(t%o 'ercent of the 'artici'ants %ere freshmen, 446 so'homores, 1G6 >uniors, and 1=6seniors* @ost 263 of the students li#ed on cam'us* +he tele'hone inter#ie% students had

more range in their :PAs than the focus grou' 'artici'ants* +hirt$(eight 'ercent of the'artici'ants) :PAs %ere a!o#e .*, 546 %ere !et%een 2* and .*, and =6 fell !elo% 2**earl$ half 4463 of the students ma>ored in the College of Arts and Sciences*

In&)erson Intervie' Sample 2n5+047

+%ent$(t%o 'ercent of the in('erson inter#ie% 'artici'ants %ere male %hile =6 %erefemale* +%ent$(t%o 'ercent of the students %ere freshmen, 546 so'homores, 126 >uniors, and16 seniors* /ne graduate student 'artici'ated as she %as seated in one of the grou's of students%e a''roached* 0nli-e the first t%o methods of the research, more than one(third .=63 of the'artici'ants li#ed off cam'us, %hile G26 li#ed on cam'us* +%o of the res'ondents did not

disclose their :PA9 5=6 of those %ho res'onded had :PAs a!o#e .* and 426 %ere !et%een2* and .** o students re'orted a :PA !elo% 2** Fort$ 'ercent of the students ma>ored in theCollege of Arts and Sciences*

Page 1=

survey participation

Documents