assessing pa with self-report: a methodological overview
DESCRIPTION
Assessing PA with self-report: A methodological overview. Louise C. M âsse, PhD University of British Columbia. Overview of talk. Utility of self-report Type of studies PA self-report are used in Type evidence needed in these studies - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/1.jpg)
Assessing PA with self-report:A methodological overview
Louise C. Mâsse, PhDUniversity of British Columbia
![Page 2: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/2.jpg)
Overview of talk
• Utility of self-report• Type of studies PA self-report are used in• Type evidence needed in these studies• How we are currently assessing the validity of
PA self-report• Discuss potential issues to consider in the
validation process
2
![Page 3: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/3.jpg)
Utility of Self-Report?
• Advantages– Most practical & economical– Only feasible method in large scale studies– Best at measuring context & type of PA performed
• Limitations– Utility in some context debated– Validity & reliability less than objective methods[Sallis & Saelens (2000); Shephard (2003); Westerterp (2009)]
3
![Page 4: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/4.jpg)
TYPE OF STUDIES PA SELF-REPORT ARE USED IN
4
![Page 5: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/5.jpg)
Type of Studies
Epidemiologicalstudies
Surveillancestudies
Interventionstudies
5
![Page 6: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/6.jpg)
Type of Studies
•Assess relations between PA and healthEpidemiological
studies
Surveillancestudies
Interventionstudies
6
![Page 7: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/7.jpg)
Type of Studies
•Assess relations between PA and healthEpidemiological
studies•Monitor levels of PA and patterns
of change at the population levelSurveillancestudies
Interventionstudies
7
![Page 8: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/8.jpg)
Type of Studies
•Assess relations between PA and healthEpidemiological
studies•Monitor levels of PA and patterns
of change at the population levelSurveillancestudies
•Generally aim to detect small changes in PAIntervention
studies
8
![Page 9: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/9.jpg)
Methodological Consideration
• Validation sample• Study design• Validation criterion• Statistical procedures
9
![Page 10: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/10.jpg)
Methodological Consideration
• Validation sample• Study design• Validation criterion• Statistical procedures
Content validation process - completed•Qualitative interviews•Cognitive interviews•Pilot testing….
10
![Page 11: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/11.jpg)
Statistical Procedures
Sensitivity to change
Agreement
Association
Level of evidence
Criterion
11
![Page 12: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/12.jpg)
Level of evidence & interpretation
• Association– Indicate whether different people who responded
to the questions did more or less PA (Ranking of the scores on the PA self-report ).
12
![Page 13: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/13.jpg)
Level of evidence & interpretation
• Association– Indicate whether different people who responded
to the questions did more or less PA (Ranking of the scores on the PA self-report ).
• Agreement– Allow the scores on the PA self-report to be
interpreted in terms of actual amount of PA the person does. Actual min would correspond to the behavior.
13
![Page 14: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/14.jpg)
Level of evidence & interpretation
• Association• Indicate whether different people
who responded to the questions did more or less PA (Ranking of the scores on the PA self-report).
• Agreement– Allow the scores on the PA questionnaire
to be interpreted in terms of actual amount of PA the person does. Actual min would correspond to behavior
Relativeinterpretation
Absolute interpretation
14
![Page 15: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/15.jpg)
Type of studies & Level of evidence
Type of studies Level of evidence
Epidemiological Association
Surveillance
Relative change over time AssociationActual amount of PA - % meet recommendations
Agreement
Intervention Sensitivity to change
15
![Page 16: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/16.jpg)
Type of studies & Level of evidence
Type of studies Level of evidence
Epidemiological Association
Surveillance
Relative change over time AssociationActual amount of PA - % meet recommendations
Agreement
Intervention Sensitivity to change
Group level
validity
Individuallevel
validity
16
![Page 17: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/17.jpg)
WHAT ARE WE CURRENTLY DOING?
17
![Page 18: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/18.jpg)
ASSOCIATION
18
![Page 19: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/19.jpg)
Associations – Epidemiological studies
• Relationship between PA and health exist
High
Level ofPA
Moderate
Low High
Moderate
Low
CVD risk
factors
Requires valid relative ranking for PA self-report
19
![Page 20: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/20.jpg)
Associations – Epidemiological studies
PASelf-Report Criterion
Correlation, regression, intra-class correlation
• Appropriate methods to assess ranking?
• Limited in their ability to account for measurement errors in the validation process
20
![Page 21: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/21.jpg)
Associations – Epidemiological studies
• Statistical models that account for both measurement errors (Buonaccorsi, 2000) and correct for attenuations in epidemiological studies are not new but deserve more attention.
Mixed modelsFamily of statistical procedures
21
![Page 22: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/22.jpg)
Associations – Epidemiological studies
PASelf-report
CVD risk factors
Full sample N = 1000Uncorrected association = .25
22
![Page 23: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/23.jpg)
Associations – Epidemiological studies
PASelf-report
Criterion(subset of sample)
CVD risk factors
Correction forAttenuation
N=200
Full sample N = 1000Uncorrected association = .25Corrected association = .35
23
![Page 24: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/24.jpg)
AGREEMENT
24
![Page 25: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/25.jpg)
Surveillance Studies
• Relative ranking – Association– If reliability is acceptable then it can be used to
assess group level change from year to year– Assume no secular trend affecting patterns and
that the questions remain the same
Can detect group level changeCannot assess % meeting recommendation
25
![Page 26: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/26.jpg)
Surveillance Studies
• Absolute interpretation – Agreement– % meeting PA recommendation need actual min
• Issue to consider– Do we have a valid criterion?
26
![Page 27: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/27.jpg)
Possible Criterion
• Accelerometers• VO2 max• Doubly Labeled water• Pedometers• 6 min walk test• Run test• Heart rate• Body fat• FEV….
27
![Page 28: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/28.jpg)
Useful Criterion
• Accelerometers• VO2 max• Doubly Labeled water• Pedometers• 6 min walk test• Run test• Heart rate• Body fat• FEV….
28
![Page 29: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/29.jpg)
Criterion
• Accelerometry– Min above a certain count can be compared– Does not capture all activities (walking based)
29
![Page 30: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/30.jpg)
Criterion
• Doubly Labeled water – TEE = RMR + TEF + AEE (Kcal metric)– Self-report PAEE = min * MET value * BW/60– High intensity activities account for 25% of AEE
need to assess moderate and inactivity (Westerterp, 2009)
30
![Page 31: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/31.jpg)
Criterion
• Doubly Labeled water – TEE = RMR + TEF + AEE (Kcal metric)– Self-report PAEE = min * MET value * BW/60– High intensity activities account for 25% of AEE
need to assess moderate and inactivity
LowModerate
High Intensity activities
Westerterp, 200931
![Page 32: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/32.jpg)
Criterion
• Transforming the instruments on the same metric does not mean that both instruments (self-report and criterion) are capturing the same thing.
VERSUS
32
![Page 33: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/33.jpg)
Agreement – Surveillance studies
PASelf-report Criterion
Bland & Altman, group comparison (t-test – Anova –
MANOVA), kappa, Kendall tau-b
• Appropriate methods to assess agreement?
• Other methods to quantify agreement?
=
33
![Page 34: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/34.jpg)
Quantifying Agreement
• Intra-class correlation
• Receiver Operating Characteristic (ROC) – (application Trost et al, 2006)
2
2 2s
s e
ICC consistency
2
2 2 2s
s m e
ICC agreement
Self-reportCriterion
% meeting PA rec.
34
![Page 35: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/35.jpg)
Quantifying Agreement
• Intra-class correlation
• Receiver Operating Characteristic (ROC) – (application Trost et al, 2006)
2
2 2s
s e
ICC consistency
2
2 2 2s
s m e
ICC agreement
Self-reportCriterion
% meeting PA rec.
35
![Page 36: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/36.jpg)
SHOULD WE FOCUS ON QUANTIFYING BIAS (OVER REPORTING) AS A WAY OF
IMPROVING AGREEMENT?
36
![Page 37: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/37.jpg)
Correcting for bias – systematic error
• Developing prediction equations– PA min from criterion– PA categories from criterion (Meeting / not meeting rec.)
• Challenges– Modeling the type of errors we have (Plankey 1995 BMI)
– Drift phenomenon - verifying that bias is consistent over time or sample (Gorber 2010 BMI)
• Learning from the educational field• Measurement error models
37
![Page 38: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/38.jpg)
SENSITIVITY TO CHANGE(INDIVIDUAL LEVEL VALIDITY)
38
![Page 39: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/39.jpg)
Intervention studies
• Is the measure able to detect differences pre / post interventions?
• Appropriate methods to assess change?
• Other methods to quantify change?
Change in PASelf-report
ChangeCriterion
Anova, t-test, Wilcoxon test
=39
![Page 40: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/40.jpg)
Quantifying change
Pre Post Comparing Pre and Post scores
Issues with this approach
Criterion (VO2 max) p<.05
Those who changed on the criterion might not be the same who changed on the PA self-report.PA self-report
p<.05
40
![Page 41: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/41.jpg)
Quantifying change
Pre Post Comparing Pre and Post scores
Issues with this approach
Criterion (VO2 max) p<.05
Those who changed on the criterion might not be the same who changed on the PA questionnaire.PA self-report
p<.05
Pre Post Pre and Post change scores
Alternative methods
Criterion (VO2 max) Change score can be compared statistically using either “correlational methods or agreement methods.” Agreement only possible if the metrics are comparable .
PA self-report
41
![Page 42: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/42.jpg)
Quantifying change
• Validating change scores with “correlational methods” (regression methods)– Accurately rank the change with PA self-report
• Validating change scores with “agreement methods”– Change score can be meaningfully quantified (in
min for example)
42
![Page 43: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/43.jpg)
UTILITY OF ADVANCED PSYCHOMETRIC METHODS
43
![Page 44: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/44.jpg)
Item Response TheoryRasch Models
• Two key assumptions are problematic– Local independence– Essential unidimensionality
44
![Page 45: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/45.jpg)
Assumptions
PA domains • 1-dominant dimension that explains about 40% of the variance. This assumes high correlations among domains.
• Errors are not correlated after you account for dominant dimension.
Yard work
Exercise / Sports
Transportation
Family
Volunteer
Household
Employment
Miscelanneous
45
![Page 46: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/46.jpg)
Assumptions
PA domains
Yard
Exercise / Sports
Transportation
Family
Community
Household
Employment / unpaid work
Miscellaneous
Correlation likely to be moderate
Correlation likely to be low or non-existent
46
![Page 47: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/47.jpg)
Assumption test – real data
• Minority women (N=250)
• Essential unidimensionality test– Factor analysis forcing a 1-factor solution
47
![Page 48: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/48.jpg)
Assumption test – real data
• Correlations among domains– Highest correlation = 0.207 – Next highest correlation = 0.137
• 1-factor explain 16% of total variance– A 2nd factor would explain 14% of total variance– Not all items load on 1st dimension
Clear violation of the assumptions48
![Page 49: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/49.jpg)
CAN ADVANCED PSYCHOMETRIC METHODS BE USED TO VALIDATE PA
MEASURES?49
![Page 50: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/50.jpg)
Take home message
Epidemiologicalstudies
Surveillancestudies
Interventionstudies
Association
Agreement
Sensitivity to change
Validation evidence
Can we estimate the % meet who PA recommendation?50
![Page 51: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/51.jpg)
Take home
• Before you select a PA instrument for your study – make sure it has been validated for how you intend to interpret the data.
51
![Page 52: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/52.jpg)
What evidence do we have?
•Lots at this level
Associations•None
Agreement
•Some but weaker designSensitivity to change
52
![Page 53: Assessing PA with self-report: A methodological overview](https://reader035.vdocument.in/reader035/viewer/2022081514/568163ab550346895dd4bd48/html5/thumbnails/53.jpg)
WILL BETTER STATISTICAL METHODS IMPROVE OUR SELF-
REPORT INSTRUMENTS?
53