differences in prevalence rates in - · boys girls total boys girls total boys girls total higher...
TRANSCRIPT
Differences in prevalence rates in school surveys: effect of questionnaire administration or inconsistent answers?
Trapencieris, MarcisPetersons A, Snikere S, Redovica I, Trapenciere I
Institute of Philosophy and Sociology, University of Latvia
Results not always conclusive❖ Some studies find higher substance use prevalence rates in computer
administered surveys❖ «…estimates of the prevalence of male-male sex, injection drug use, and sexual contact with
intravenous drug users were higher by factors of 3 or more when audio-CASI was used…» Turner et al 1998
❖ «…web SAQ leads to higher reporting rates of commonly used substances in adolescents compared with those of paper SAQ. » Wang 2005
❖ Some – no differences or mixed results❖ «…web-based administration of health indicators yields almost the same results as paper-and-pencil
administration.» van de Looij-Jansen 2005
❖ «… while CASI did not increase reported rates of substance use over PAP, it significantly improved the speed of data processing and decreased the incidence of missing data.» Hallfors 1998
❖ Others – higher prevalence rates in traditional paper-and-pencil surveys ❖ «…for the majority of lifestyle behaviors, we found that mode of administration had no significant effect on
adolescents’ responses, whereas for several questions about feelings/affective states, more socially desirable responses were found in the paper and pencil format…» Vereecken 2000
Methods – Sampling❖ Nationally representative probability sample (Grades 8-10) across Latvia in
April-May 2013
❖ One stage stratified cluster sampling
❖ 114 strata
❖ Sample frame
❖ 2441 classes in 666 schools with 42 634 students
❖ Sampled classes
❖ 231 classes
❖ Participating schools
❖ 178 classes
Methods – Study design
❖ The computer administered version of the questionnaire was prepared by scripts available in the open-source Limesurvey package, using PHP, MySQL
❖ No question skipping rules
❖ Students in each class systematically assigned to one of the two modes: p&p or computerized (web)
Methods – Questionnaire
❖ Core questionnaire of the ESPAD* study❖ first validation study but more underway, i.e. Italy
❖ Translated in Latvian and Russian
❖ 44 questions (198 variables) about tobacco, alcohol and drug use
❖ lifetime, last year and last month use
❖ 24 indicators dichotomized
❖ 6 – tobacco, 12 – alcohol, 6 – drugs
* European School Survey Project on Alcohol and other Drugs
Methods - data analysis❖ Inconsistent answers
❖ ESPAD data cleaning scripts (Hibell et al 2012)
❖ simple and pragmatic approach
❖ Four criteria: 1) >50% missing, 2) 6+ answers (of 19) refer to unlikely drinking pattern, 3) 5+ answers (of 17) 40+ times use of cannabis or other drugs, 4) 5+ (of 10) alcohol related problems 40+ times in last year
❖ Descriptive statistics and binary logistic regressions adjusting for age and gender and cluster design
❖ Stata v12
Additional data validation scenarios explored
❖ No change – keep answers «as is» (raw data)
❖ ESPAD cleaning rules – ESPAD cleaning syntax as explained before
❖ Gatekeeper approach – take the first answer as «gold standard»
❖ Global approach – check everything with everything and disregard any inconsistent answers
❖ Within substance
❖ Within substance and across time frame
❖ Across substances
❖ Inconsistencies treatment
❖ if inconsistent -> prevalenceLTP, LMP=1 (in cases where it used to be 0) and prevalenceLTP,
LMP=. (in cases where it used to be 1)
❖ if inconsistent -> prevalenceLTP, LMP=1 (in cases where it used to be 0)
❖ Sensitivity/specificity approach - need to elaborate more on cut-off points
Results
Total 14, 15 or 16 year old After simple data cleaning scripts
Boys Girls Total Boys Girls Total Boys Girls Total
P&P 666 898 1564 617 847 1464 609 845 1454
CASI 763 752 1515 724 708 1432 683 688 1371
TOTAL 1515 1564 3079 1341 1555 2896 1292 1533 2825
Results - validityP&P WEB
Boys Girls Total Boys Girls Total
C1 5 9 14 17 22 39
C2 8 2 10 17 14 31
C3 7 3 10 16 12 28
C4 4 4 8 13 11 24
Total 13 11 24 46 29 75
% of sample* 2.1% 1.3% 1.6% 6.4% 4.1% 5.2%
C1: >50% missing
C2: 6+ answers (of 19) refer to unlikely drinking frequency
C3: 5+ answers (of 17) refer to 40+ times use of cannabis or other drugs
C4: 5+ (of 10) alcohol related problems 40+ times in last year
* ALL AGES
Number of indicators differing by each modeSmoking Alcohol Drugs
Boys Girls Total Boys Girls Total Boys Girls Total
Higher in P&P
5 3 5 6 9 7 2 2 2
Higher in CASI
1 3 1 6 3 5 4 4 4
Total 6 6 6 12 12 12 6 6 6
4.7 pp for lifetime smoking0.2 pp for daily smoking
3 pp for lifetime alcohol0.1 pp for alcohol last
month 10+
2.6 pp for lifetime Spice0.5 pp for cannabis last
month
TobaccoP&P, %
(95% CI)WEB, % (95% CI) Diff, in pp p aOR (95%
CI) p
Smoking lifetime 73.3 68.5 4.8 0.80 (0.66–0.99) 0.020
Smoking lifetime 40+ 25.7 23.4 2.3 .89 (0.73–1.09) 0.266
Smoking last 30 days 33.1 29.5 3.6 0.85 (0.71–1.02) 0.082
Smoking daily 17.9 17.7 0.2 1.00 (0.82–1.22) 0.979
Alcohol (1)P&P, %
(95% CI)WEB, % (95% CI) Diff, in pp p aOR (95%
CI) p
Alcohol lifetime 93.0 (91.4–94.2)
90.9 (89.1–92.6) 2.1 0.042 0.77
(0.60–1.00) 0.054
Alcohol lifetime 40+ 22.5 (19.8–25.5)
24.8 (22.1–27.7) -2.3 0.127 1.16
(0.98–1.38) 0.074
Alcohol last 12 months
79.0 (76.4–81.5)
77.1 (74.7–79.4) 1.9 0.209 0.91
(0.76–1.09) 0.291
Alcohol last 12 months 20+
11.8 (10.1–13.7)
12.3 (10.5–14.4) -0.5 0.629 1.07
(0.87–1.31) 0.507
Alcohol last 30 days 50.0 (47.0–52.9)
49.4 (46.2–52.6) 0.6 0.756 0.99
(0.85–1.16) 0.907
Alcohol last 30 days 10+
4.0 (3.2–5.0)
4.1 (3.2–5.1) -0.1 0.939 1.02
(0.72–1.44) 0.904
Alcohol (2)P&P, %
(95% CI)WEB, % (95% CI) Diff, in pp p aOR (95%
CI) p
Drunkenness lifetime 50.2 (46.9–53.4)
48.2 (44.9–51.5) 2.0 0.318 0.94
(0.79–1.11) 0.433
Drunkenness last 12 months
31.6 (28.9–34.4)
29.9 (27.5–33.1) 1.7 0.439 0.96
(0.81–1.13) 0.597
Drunkenness last 30 days
10.4 (8.9–12.2)
10.0 (8.5–11.8) 0.4 0.684 0.96
(0.76–1.22) 0.755
Binge drinking last 30 days
34.3 (31.7–36.9)
37.3 (34.3–40.4) -3.0 0.106 1.16
(0.98–1.36) 0.076
Tried alcohol under age 13
71.5 (69.0–73.8)
69.0 (66.3–71.6) 2.5 0.139 0.88
(0.75–1.03) 0.123
Drunk under age 13 17.1 (14.8–19.7)
17.9 (16.0–19.9) -0.8 0.639 1.05
(0.84–1.32) 0.646
Illicit drugsP&P, %
(95% CI)WEB, % (95% CI) Diff, in pp p aOR (95%
CI) p
Cannabis 22.3 20.1 2.2 .89 (0.74–1.07) 0.212
Cannabis last 12 months 14.6 13.4 1.2 .91 (0.74–
1.12) 0.394
Cannabis last 30 days 5.9 6.3 -0.4 1.1 (0.8–1.50) 0.569
Spice lifetime 11.4 14.2 -2.8 1.31 (1.03–1.66) 0.028
Impact of cleaning rulesAlcohol lifetime % CASI ncasi % P&P np&p Total ntot p
Raw data 91.2 1414 92.9 1413 92.1 2827 0.072
ESPAD cleaning 91.0 1362 92.9 1403 92.0 2765 0.042
ESPAD cleaning + Global
90.9 972 92.6 1019 91.8 1991 0.134
Inconsistency 1 91.2 1006 92.8 1043 92.0 2050 0.158
Inconsistency 2 93.5 1362 94.6 1403 94.1 2765 0.176
Sensitivity/specificity (10+ (of 29))
91.4 1373 93.3 1418 92.3 2792 0.044
Sensitivity/specificity (5+ (of 29))
92.0 1264 93.3 1323 92.7 2587 0.178
Summary and conclusions❖ Computer administered data collection about substance
use in school settings in Latvia seems feasible
❖ but might not be in other cultures or contexts, yet
❖ still, computer labs might be small to accomodate whole class simultaneously - do results get affected if class is split in several labs or conducted in different times
❖ More missing, implausible data in CASI mode
Summary and conclusions (2)❖ For most (22 of 24) substance use related indicators
differences in prevalence rates were marginal
❖ 14 indicators slightly higher in P&P mode of questionnaire administration
❖ 10 indicators slightly higher in CASI mode of questionnaire administration Various data validation procedures provide provide different results and mode differences for some dissapear
❖ Factors not accounted for?
Summary and conclusions (3)
❖ Implementation of possibilities not feasible in traditional questionnaires
❖ questionnaire design, visual clues
❖ question timings as source for data validation
❖ skipping patterns
❖ Mixed mode approaches in data collection in schools
Acknowledgments
❖ The Centre for Disease Prevention and Control for funding
❖ European School Survey Project on Alcohol and other Drugs (ESPAD) country PIs for comments on earlier phases of this work❖ especially Sabrina Molinaro, Ludwig Kraus, Bjorn Hibell,
Thoroddur Bjarnasson
Update 2015❖ Methological paper accepted for publication in Comp Hum Beh
❖ Latvia was one of the few countries conducting ESPAD 2015 usign computerized questionnaire
❖ significantly lower response rate as compared with previous ESPAD studies (~80% ->~ 40%) thus reaching low student numbers
❖ planned as fully-computerized with minimal human interaction, e.g. sending e-mails, reminders, registration, etc. -> due to very low inveolvement rate after 3 weeks decision was made towards calling schools
❖ schools with more than 1 class quite often used the same code for several classes
❖ few schools did not follow methodology and gave tokens to students for folling-in questionnaire after school hours
Update 2015 (2)❖ Challenges / plans for the future
❖ sampling rate high and number of students decreasing
❖ analysis of mood change and questions skipping/no skipping while answering the questionnaire
❖ analysis of question timings -> additional ideas for data cleaning
❖ re-visiting questionnaire and considering question skipping rules -> might have more impact on results
