the survey method paul lambert applied social science, stirling university 5.5.04, 9-11am

Nursing, Midwifery and Allied Health Professions

Research Training Scheme Training Workshop May 2004

The Survey Method Paul Lambert

Applied Social Science, Stirling University

5.5.04, 9-11am

NMAHP Research Training: Survey Method, May 2004

Resources for this talk

• Slides – the nature of survey research – issues in doing survey research

• Reading guide

• Activities sheet – introduction to research resources– example analysis of a secondary survey dataset

The Survey Method

1. The Nature of Social SurveysDefining surveys; survey sampling; types of

surveys; history of surveys & reactions to them

2. Doing Survey ResearchResearch support sources; Constructing data (collecting data; secondary data; working with variables); Data management and analysis

1. The Nature of Social Surveys

1a) Defining surveys

1b) Survey sampling

1c) Types of surveys

1d) History of surveys in the social sciences

1a) Surveys: The systematic collection of selected information from all or part of a population

(see Marsh 1982)

Surveys are characterised by ‘variable-by-case matrix’

Cases Variables 1 1 17 1.73 A . . . .

2 1 18 1.85 B . . . .

3 2 17 1.60 C . . . .

4 2 18 1.69 A . . . .

. . . . . . . . .

• Cases can be: – Any distinctive entity– Most often, they are individuals (people)

• Variables are: – Measures of selected concepts of interest– Indicators (our ‘best guess’ at representing the

concept)– Variable design: issues in choosing and

formulating appropriate variables

Example:

Cases Variables

Person Sex Age Height Health

1. Alan 1 17 1.73 A .

2. Bill 1 18 1.85 B .

3. Cath 2 17 1.60 C .

4. Dawn 2 18 1.69 A .

. . . . . .

The survey size

• Total number of cases survey size (n)

• A census covers every case in population.

• Most surveys use samples of cases.

• Larger survey size

more reliable sample estimates.

Varieties of social surveys

• Topic: choice of variables / cases

• Scale: number of variables / cases

• Method: data collection format

• Use: type of data analysis; descriptive v’s inferential

Why study survey research?

• To undertake surveys– If answers research question; if attainable– Valuable skills

• To understand / critique other people’s survey research based reports– Crucial – survey evidence is everywhere– Don’t just ignore / dismiss survey evidence

Strengths of surveys (1)

• Can be representative / large scale Probability theories justify generalisation

from samplesSurveys can handle census or other large

sample data collections and analysis Parsimonious summary of the relation

between variables on many cases

• Extensive methods research Eg, tests of reliability and validity - Bryman

2001 pp70-74:

Stability Face validity

Internal reliability Concurrent ``

Inter-observer

consistency

Predictive ``

Construct ``

Convergent ``

• Variety of data analysis formats‘Descriptive’, ‘Inferential’, ‘multivariate’Causal analysis defensible, eg longitudinalData analysis is falsifiable Report writing skills, & careful

qualifications

• Accessibility of survey researchMost research questions benefit from survey

investigationSecondary datasets widely & freely available Small scale surveys quick to conductSurvey results often convince others

• Surveys are less biased than most other social research methods

Transparency of: sampling methods; variable construction; data analysis

Falsifiability Cynicism of receiving audiences..!

Examples: surveys in health research

• General purpose and focussed cross-sectional – Scottish Health Survey: self-reported health and

lifestyle of 4000 adults– Selected population, eg sufferers of condition X

• Longitudinal follow-up studies– Birth cohort studies: parental backgrounds and

childhood health progressions– Ageing, status and sense of control (US): 1995 sample

of 3k in 1995, recontact 1.5k in 1998

• Experimental designs – Smokers’ reactions to treatment programmes

1b) Survey sampling

Role of samplingSurveys usually select only a sample of cases - aim

to be representative of wider population

• Key idea is inference

= confidence in our ability to generalise

Sampling inference = application of statistical theories in order to estimate probabilities that a

sample result is ‘likely to have been unrepresentative’

The ‘normal’ (Gaussian) curve

Theories of sampling methods

Sampling and probability theories tell us that any particular random sample is most

likely to have the same properties as the wider population. We can then estimate the

probability that sample results of a particular nature could have arisen by

chance, rather than because they are the same as the (unknown) population result.

If the cases in sample surveys were selected at random, then can use sampling theories and

thus ‘inference’

Statistical inference

..causes confusion; one of hardest parts of survey data analysis to understand..

Phrases: ‘significance level’ ‘p-value’, ‘confidence interval’, ‘hypothesis testing’, ..

Meaning: Whether results would probably generalise to a larger population

(if sample is treated as random)See: Refs on reading list (esp Wright 2002)

‘Inferential data analysis’• Variable-by-case matrix data analysis for

generalising findings to population

• Often distinguished from ‘descriptive’ data analysis (results of sample only)

• Key: joint influence of – 1) size of sample – 2) strength of data pattern

in increasing confidence about generalisations

• Doing good inferential analysis is difficult: – Reliable sampling resources expensive– Many early critiques of survey research concerned

inappropriate inferential analysis

• Contemporary survey research tends to follow 2 alternate strategies:

Large scale, often secondary, rigorous inferential methods

or Small scale, primary, claims carefully qualified

Drawing samples (case selection)Populn. Cases Variables

1 - - - - -

2 - - - - -

3 - - - - -

4 1 1 17 1.73 A

5 2 1 18 1.85 B

6 - - - - -

7 3 2 17 1.60 C

8 4 2 18 1.69 A

N=8 n=4

Sampling methods

= Ways of selecting case from population

(i) Random

(probabilistic)

Generalisable, inferential

statistics, fewer applications

(ii) Non-random

(opportunistic; purposive)

Harder to generalise,

inference contested, more widely used

a) Simple Random Sample

• A statistical method used to choose cases randomly (eg random numbers)

Every case in population has exactly the same chance of being in sample

• Most data analysis techniques initially designed for simple random samples

b) Systematic Random Sample

• Like the Simple RS, select cases from anywhere in the whole population

• An easier selection method : choose every (n)th person for the sample

• Danger of ‘periodicity’ if original population order has any structure, bias

Problems with sample methods selecting from whole population

• The ‘random’ part means it’s always possible to get a population coverage quite different from known structures

• If total population is large or dispersed, then coverage of random parts of it is expensive and time consuming: few surveys use random sampling from whole of UK

c) Stratified random samples

• Modifies random sample to ensure intended coverage of population groups– split sampling frame by stratification factors – select random samples within each factor– final sample has fixed proportions of each – Example: select 490 M and 510 F

• Properties: proportionate sample; correct representations; but more expensive & complex; may need ‘weights’ for analysis

d) Multistage cluster samples

• i) Select clusters of population at random • ii) Sample randomly within clusters• Eg: clusters = local authorities in UK

– With qualifications, may still be treated as ‘random’ for analysis purposes

– Big reduction in costs if face-to-face contacts

Most widely favoured sampling method in large scale survey collections

Example: Multistage cluster sample

• Interest: attitudes of Scottish school pupils

• Resources: 400 interviews with pupils

Edinburgh 100

Argyll 24

Islands 20Highlands 40

Glasgow 124

Aberdeen 40

Shetlands 2

Borders 10

Perth 20

Moray 20

Glasgow 150

Edinburgh 150

Stirling 60

Moray 40

Stirling 60

30 young people at Balfron School

30 young people at Stirling High

Issues in random sampling

• Only as good as underlying sampling frame (a good one may not be available, or not be as good as we think)

• Data analysis methods need adapting for stratified / clustered designs

• Other survey factors interact with sample selection issues, eg poor interviewers may discourage certain cases from response

ii) Opportunistic sampling

• Often in social research, sample design is ‘opportunistic’ (‘purposive’)

– Random sampling is expensive – Random sampling is complex – Some purported random samples are actually

purposive anyway (understanding ‘random’)

a) Quota sampling

• Fill up quota’s of groups of interest

• Quota’s can ensure: – overall representation (cf systematic) – broad topic coverage (eg types of voter)

• Example: market researchers in street; telephone call centres vetting contacts

• Biasses: issues in how a quota ‘fills up’

b) Snowball sampling

• Also ‘focussed enumeration’• Technique for contacting cases from

populations rare / difficult to access• Ask first obtained contact for suggested

further contacts snowball gathers size… • Eg – smaller ethnic minority groups• Problem: social networks are non-random!

c) Convenience sampling

• Samples whatever cases from population were easiest to reach, eg personal contacts

• Often no other sampling strategy involved

• Biasses likely in convenience process

• Examples: …most student survey projects are ‘convenience’..!

Random v’s Opportunistic

• Random sampling difficult and expensive – mainly government funded surveys

• Much data analysis / inference assumes random sample, but not applied to

• But random sampling is not a panacea...

• And opportunistic data is often robust…

Rule: Use survey documentation to report sampling process and any errors

1b) Survey sampling

Varieties of social survey designs

• Micro-social data: 1. Census’s2. Cross-sectional surveys3. Longitudinal surveys4. Cross-nationally comparative surveys5. Experiments or ‘quasi-experiments’

• {Macro-social data} 6. Single summary statistics describing outputs

{from survey analyses}

• Census’s– General overview of whole population– ‘Disclosure risk’ issues

• Cross-sectional surveys – Very widely used format– Huge range of topic coverage– Often used to study particular or rare

subpopulations

Longitudinal datasets : studies involving time• Repeated cross-sections

– Chart changes over time, eg yearly means

• ‘Panel’ and ‘cohort’ samples – recontact an initially random sample– Learn about changers and causes of actions– Problems of attrition

• Retrospective sample– Rely on recall evidence of random selection– Problems of selective recall

Strengths: understand process and causality Problems: sampling and attrition; complexity

Cross-nationally comparative datasets

– Focussed surveys (IPUMS census’s; ISSP; World Values Survey; European Social Survey)

– Longitudinal studies (LIS; ECHP; CHER)– Many analytical attractions, but issues of

comparable analysis are complex

Experimental / quasi-experimental designs

• Experiment: researcher intervenes in the process of study (quasi-experiment: ‘observe’ intervention)

• Dream of the randomised controlled trial • Rare in sociology: cost & ethics; more common in

psychology & certain health research fields• Consequences:

– Different methods of analysis (see eg Robson 2002 c5)

– Less concern over inference / v large samples

Simple and complex survey data

• Simplest variable-by-case matrix has one sample of independent cases from 1 period

• More interesting social science data has more complex designs, eg…– Multiple records per case– Relations between cases– Experimental matching of designs

Working with complex survey data..

• Advanced tasks in data management– File matching– Variable transformations and treatments

• Advanced methods of data analysis• Complex findings not easily summarised or

communicated• …but it is simplicity of simple survey data

that many people criticise about SDA…

1b) Survey sampling

Rise of the survey method..

• 1920’s to 1960’s saw vast expansion in survey examples

Government, commercial, political, academic all conducting surveys on own topics

• Method of choice of social research

Eg ‘only empirical method’, dominates teaching of research methods

Fall of the survey method..

• Early dominance of surveys produces backlash 1960’s onwards (eg Cicourel 1964): Philosophical – research epistemology Pragmatic – many badly conducted surveys Prejudiced – fear of working with statistics leads

researchers to avoid, ignore or oppose surveys Political – white male power Populist – ‘lies, dammed lies and statistics’

..Resurgence of survey method

• Critiques provoke defence – Methodological research to avoid simplifying

mistakes, improve sampling designs, and avoid unjustified claims

• Substantial support for survey research always there, eg Government, commerce

• Funding organisations now try to redress over-reaction, encourage survey research teaching and activities (eg ESRC)

Current state of play

• Survey and other research methods are largely separate entities– Pragmatic: Involve different people and skills

– But not methodologically appealing (eg Bryman 1992 on ‘mixed methods’, extract in Seale 2004)

• Contemporary social science survey research shows strong disciplinary separatisms, eg: – UK sociology = tables & bivariate; US sociology =

regressions; Psychology = Anova; Economics = regression extensions. (Accusations of ‘methodolatry’)

Importance of studying Surveys

Whatever your views on methods, surveys are & always will be one of most important

social research tools - they should never be ignored!

The Survey Method

2. Doing Survey Research

2a) Research support

2b) Constructing data

2c) Data management and analysis

Research support with surveys

• References: Hundreds of textbooks and papers – See reading list, esp Buckingham and Saunders 2004

• Internet resources, eg ESDS, RMS – See activities sheet

– Learn from prior examples: qnre schedules, samples,

• Learning by doing– the ‘apprentice model’ of social survey research

– workshops and short courses

Data construction issues:

• 2.1a: Collecting and entering data

• 2.1b: Accessing secondary data

• 2.1c: Variable operationalisations

2b -1 Good practice in survey collection

Extensive history of methodological research into how to collect data with greater

reliability and validity

- see some eg’s on reading list (..De Vaus chpts 7-8; Gilbert chpt 6..)

Data collection issues:

• Appropriate sampling strategy (adhered to)

• Question wording

• Interviewer skills and contexts

• Minimise non-contacts / refusals

• Document data collection assiduously

• …every badly conducted survey makes it harder for surveys in the future…

Questionnaire data entry

• ‘Codebook’ : devise system to represent variables through numeric codes

Choices in variable operationalisations and categorical groupings

Small scale: tap values into computer from schedule (laborious)

Large scale: data entry software / specialist staff

2b-2 Secondary survey data

•Vast quantity of surveys conducted

An efficient step would be to analyse existing data (secondary) rather than personally collect your own (primary)

•Data archives collate survey datasets and supply them for secondary analysis

• Government funds many large surveys (also EU; LA’s; charities; commercial)• Often made available freely or at low cost An ideal research tool (see ESRC):

– Quick to access– Methodological rigour in sample &

questionnaire design, & interview collection– Falsifiable – others can access also– Generalisable & larger scale – Multivariate

Secondary analysis of surveys

• Makes particular sense when large scale datasets are desirable

• Also often applies to smaller surveys

• Involves particular issues of data analysis, management and interpretation

• …Is a highly marketable skill!

Accessing Secondary datasets

• Internet and computing developments have revolutionised delivery of data resources (see activities sheet)

• Three steps to data access:1. Find out survey details / documentation

2. Apply for access from archive or collectors

3. Obtain and analyse the data

Some drawbacks

• Distance from data collection – Harder to assess reliability / validity– Many variables already pre-coded – Can’t change / add anything in study

• Time delays between collection & results

• Data analysis / management complex

• May be bracketed with survey originators

2b-3 Variables in data analysis

• Variable operationalisation key to surveys• Choices: - in initial data collection

- in data recoding / analytic treatment• Existing comment / research on many widely used

variables (eg Burgess 1986)• Critiques of survey research most often

concentrate on variable operationalisations…

Eg: Education and occupation:

• Education– Changing ‘levels’ of education over time– Education as proxy for ability, intelligence?

• Occupation– Contested meanings of labour market status– Occupational indicators of stratification– Occupational gender segregation

Variable operationalisations

• Methods guidelines on appropriate handling

‘Harmonised concepts and questions’; textbooks; papers / debates specific issues

• Choices / approximations always used

• Research reports and methods appendices must explain and justify position taken

Data management and analysis

• 2c-1 : Manipulating variables and cases

• 2c-2 : Mainstream techniques of data analysis

• 2c-3 : ‘Robust’ data analysis

2c-1 Data management

• ..is core skill in using primary and secondary surveys…– Occupies more time than analysis in most cases

Main techniques cover:

• Matching data files

• Coding / transforming variables

• Dealing with ‘missing’ data

Data management• Software packages – SPSS, SAS, STATA, .. –

with wide (& revolutionary) capabilities • Good and bad practices

– Keep logs and records of transformations– Follow previous literature for best variable treatments

Secondary dataset management tends to be:• More complex • More error prone • Subject to external scrutiny

2c-2 Techniques of data analysis

• Good practice– Reflects properties of variables– Describes output in appropriate context

• Bad practice – Forcing data into style of analysis – Attributing false properties to data– Over zealous conclusions

= Ways of summarising the relationship between variables

Three definitions locate nearly all SocSci techniques of QnDA:

i Level of measurement of variables

ii Number of variables being analysed

iii Aims of analysis (‘descriptive’ cf

‘inferential’)

i) Key organising principle: Level of measurement of a variable

• 4 levels classically (eg Blaikie 2003:22-7)

– Nominal (gender; ethnicity)

– Ordinal (exam grades; attitudes)

– Interval (age; height)

– Ratio (some monetary measures)

Levels of Measurement

• Really only 1 distinction is most important to analysis opportunities:

Categorical = nominal and ordinalMetric = interval and ratio

• There is also some flexibility in assigning levels (eg Blaikie 2003: can usefully treat many vars in different forms)

ii) Number of variables:

• Univariate: Patterns in the distribution of cases within one variable (summary stats)

• Bivariate: Relations between patterns in the distribution of cases over two variables (crosstabs and association measures)

• Multivariate: Relations between patterns in the distribution of cases over three of more variables (complex crosstabs & statistical models)

Statistics = numerical summary indicators of the properties of

variables

• Univariate Statistics: describe the distribution of one variable

• Bivariate / Multivariate Statistics: describe how the distribution of two / more variables are related

iii) Aims of analysis

• Descriptive and explanatory– Summarise patterns in sample / data

– Use statistical procedures to highlight ‘real’ patterns

– Examine how some variable patterns ‘effect’ others

– End in itself for some data (eg census)

• Inferential– Assess the possibility of generalising our findings

– Prominent in sample survey analysis, but needs qualifications (eg a ‘random’ sample)

Inference and statistics

• Key idea of inference is to make a qualified estimate of degree of confidence in our ability to generalise

Sampling inference = application of statistical theories in order to estimate

probabilities that a sample result is ‘likely to have been unrepresentative’

Common QnDA techniques by typology (c/m: categorical/metric; descriptive / inferential)

Frequency table : c Mean, standard deviation : m

Cross-tabulation : cc Confidence interval : m

Chi-square test : cc Scatterplot : mm

Multiple crosstab: 3+c Table of means : mc

Boxplots: mc Anova : mc

Logistic regression: 2+ c,m Multiple regression: 2+ m,c

Example: Ginn 2001, p293 in Gilbert= multivariate, categorical, & descriptive

= multiple cross-tabulation

Men Women

1987 1993 1987 1993

column percents, adults 20-59

Occup pension 46 40 22 25

APP private pension n/a 13 n/a 9

Work but no pension 25 12 38 27

Self-emp 14 15 5 5

Not employed 15 20 35 34

2c-3 Robust Data Analysis

Incorporates: Use appropriate data analysis for resourceAdvanced data analysis techniques

specifically sensitive to survey dataReport analysis results appropriately

Appropriate data analysis

• Reflect appropriate levels of measurement

• Keep categories in defensible forms

• Think about social theories behind analysis

• Analysis ought to be multivariate

• Remember missing data issues

Appropriate data analysis

• The biggest controversy: don’t misuse inferential statistics

Strictly: should only be for random…

But: can be defended for non-random, with appropriate language / qualifications

Advanced data analysis..

• Statistical research to improve accuracy of social survey analysis, proposes eg: – Weighting (more complex than appears)– Missing data models– Multilevel models– Latent variable analysis– Selection models– Other difficult models…

Report results appropriately

• Avoid over-zealous claims

• Care about term ‘causality’

• Use simpler language, tables and graphs

• Give methodological details, eg Appendix

• Many report writers have become very good at this – hard to accuse of bad practice

..the End

the survey method paul lambert applied social science, stirling university 5.5.04, 9-11am

Documents

cross-national data in dames and ge*de paul lambert,...

1 what is… the analysis of longitudinal survey data paul...

phone contact office hours educational learning platform...

stirling alliance city... · stirling alliance stirling...

paul lambert, university of stirling

sep 2005:lda - ons1 event history data structures and data...

paul lambert (univ. stirling) dave griffiths (univ....

applications of social network analysis on historical data...

strong and weak ties as predictors of occupational position...

in the chancery court of shelby county, tennessee...

dealing with variables: resources and topics in enhancing...

dr paul lambert and professor vernon gayle university of...

the stirling engine as an alternative for energy production...

geode - nesc workshop, oct 2006 geode: grid enabled...

sep 2006: lda1 data sources and data structure: panel data...

‘escape from poverty’ and occupations...paul lambert,...

‘escape from poverty’ and occupations paul lambert,...

‘interpreting coefficients from longitudinal models’...

log-concavity of stirling numbers and unimodality of...

occupational marriage networks in the usa, 1970-2010 dave...