Research MethodsAn Overview of Survey

Techniques, Sampling Strategies, Modes of Observation,

Interviewing , Focus Groups

1

Lecture outline: Overview of material covered in chapters 6, 8-12 in text

Modules created for each chapter in text

Ch6: Measuring Crime

Ch. 8:Sampling

Ch. 9: Survey Research

Ch. 10: Qualitative Interviewing

Ch. 11 Field Observation

Ch 12: Agency Records, Content Analysis, and secondary Data

2

Module 1: Surveys( see Chapter6, 9 in text)1. Survey data collection methods:A look at

UCR, Self Report , and Victimization Surveys

2. The Basics of Survey Design

3. Types of Surveys

3

Topics appropriate to survey researchCounting crime – asking people about

victimization counters problems of data collected by police

Self-reports – dominant method for studying the etiology of crimeFrequency/type of crimes committed

Prevalence (how many people commit crimes) committed by a broader population

4

Topics appropriate to survey researchPerceptions and attitudes – to learn how people

feel about crime and CJ policy( e.g. Death Penalty)

Policy proposals – search for ways to respond to crime that are supported by the general public( e.g Downsizing Prisons)

Targeted victim surveys – used to evaluate policy innovations & program success( e.g. domestic violence)

Other evaluation uses – e.g., measuring community attitudes, citizen responses, etc.( e.g. criminal thinking scales)

5

The Basic Structure of a Survey InstrumentIntroduction: state the general purpose of survey and

emphasize voluntary nature of survey,confidentiality and protection of human subjects.

Section1: Factual non-threatening questions( demographics here)

Section 2: Core questions( ordering questions; contingency questions)

Section 3: Scales( Likert, Gutmann)Example of Matrix questions--Likert Scale: TCU

Criminal Thinking Scale used by Taxman, et. al. in recent study of criminal thinking( class handout, Nov. 26th)

6

Types of questions: Open- and close-ended questionsOpen-ended – respondent is asked to provide his

or her own answerWhat were your initial expectations of the program?

________________________________________________

Closed-ended – respondent selects an answer from a list or scaleOn a scale of 1 to 5, where 1 is low and 5 is high, rate

your initial expectations of the program.

7

1 2 3 4 5

Low High

Open versus closed?Choice between these question types will

depend on the aim of your research, the study design, the population being study, etc.

Advantages and disadvantages to both.

Possible gains in reliability (closed-ended) balanced with possible losses in breadth of information and validity.

One strategy is to combine open- and close-ended questions.

8

Designing questions… Make items clear – avoid ambiguous questions;

do not ask “double-barreled” questions

Short items are best – respondents like to read and answer a question quickly

Avoid negative items – leads to misinterpretation

Avoid biased items and terms – do not ask questions that encourage a certain answer

Closed question response categories must be exhaustive and mutually exclusive.

9

Survey data collection methods1. Self-administered mail surveys,

computer based surveys, and response rates

2. In-person interview: best for complex questions ( role of interviewer is key)

3. Telephone interviews; fast, low cost; CATI- computer assisted telephone interviewing

4.A comparison of three methodsFocus: Racial Profiling Research( see text

pp 256, 257)

10

Focus: Australia’s National Crime Victimization Survey Methodology

The MPHS was conducted as a supplement to the monthly LFS. Each month one eighth of the dwellings in the LFS sample were rotated out of the

survey. In 2012-13, all of these dwellings were selected to respond to the MPHS each month . In these dwellings, after the LFS had been fully completed for each person in scope

and coverage, a person aged 15 years and over was selected at random (based on a computer algorithm) and asked the various MPHS topic questions in a personal interview.

If the randomly selected person was aged 15–17 years, permission was sought from a parent or guardian before conducting the interview. If permission was not given, the parent or guardian was asked the crime questions on behalf of the 15–17 year old.

Questions relating to sexual assault, alcohol or substances contributing to the most recent physical or face-to-face threatened assault were not asked of proxy respondents. Only those persons aged 18 years and over were asked questions on sexual assault.

Data was collected using Computer Assisted Interviewing, whereby responses were recorded directly onto an electronic questionnaire in a notebook computer, usually during a telephone interview.

11

A Link to the USA National Crime Victimization Survey

http://www.bjs.gov/index.cfm?ty=dcdetail&iid=245

12

http://www.ojp.usdoj.gov/bjs/glance/tables/4meastab.htm

In Review: Comparing UCR and NCVS—See Text Chapter UCR data are based on reported criminal acts

(offender characteristics)NCVS data based on individuals actually

victimized (characteristics of victims)Whenever possible, review both data sources

and compare findings

Assessment of NCVSDocument a massive amount of crime that goes

unreportedUnderestimate crime rateInsignificant crimes tend to be forgottenVictims of several crimes may also forget about

all the crimesFemales do not report victimization if her

abuser live in the same household Whites and college graduates are more likely to

report being victimized

Assessment of NCVSNCVS respondents are interviewed every six

months (7 interviews)Reported victimization rates usually decease

with each interview (awareness of victimization)

Overestimation of some crimesRespondents might mistakenly interpret some

noncriminal events as crimes“Telescoping “ effect

Self-reports data Created to complement UCR and NCVSInterviews or questionnairesDemonstrate the prevalence of offending (the

proportion of respondents who have committed a particular offence)

Incidence of offending (the average number of offences per person in the study)

Samples for self-reportsTarget: Adult inmates of jails and

prisons: PREA findings on sexual assaultTarget: Adolescents, usually high

school studentsThe most important finding: delinquency

is very common Middle-class youth commit as much crime

as working-class youthTesting criminological theories: Travis Hirschi’s

Research Study testing Control Theory( compared to subculural & Strain)

Assessment of self-report studiesFocus on minor and trivial offenses

(truancy, running away from home, minor drug and alcohol use)

Although recent studies (NYS) asked subjects about rape and robbery

Respondents might not to tell the truth (reliability issues)

If respondents lie…. Self-report data can be checked against police

records, school records, interviews with teachers and parents

The use of, or threat of , polygraph validation (20% change their initial responses when threatened with a “lie detector”)

Subsequent interviewing of subjects permits probing regarding the details and context of acts

Use of “lie scales”

Focus: Self Reported Substance Use Among Detainees in Australiahttp://www.aic.gov.au/media_library/conferen

ces/evaluation/mcgregor.pdf

http://192.190.66.70/documents/7/E/8/%7B7E8D4A8E-A5AF-4D3B-8821-ED8A1BA489B6%7Drpp93.pdf

21

Focus: Research on Extent of Substance Use by High school students in USAQuestion: Can we collect reliable estimates

from survey questionnaires distributed to high school students across the country?

The Survey: Lets take a close look:http://

www.casacolumbia.org/addiction-research/reports/national-survey-american-attitudes-substance-abuse-teens-2012

http://www.drugabuse.gov/publications/drugfacts/high-school-youth-trends

22

UCR, NCVS, and self-reportsNone of the three is perfectFor the best estimates of the actual number of

crimes, NCVS data are preferable For the best estimates of offender

characteristics, self-reports and NCVS are preferable

UCR are superior for understanding the geographical distribution of crime

Self-administered questionnairesCan be home-delivered

Researcher delivers questionnaire to home of sample respondent, explains the study, and then comes back later

Mailed (sent and returned) survey is most commonResearchers must reduce the trouble it takes

to return a questionnaire

Goal is High Response Rate

24

Computer-based self-administration

Via email, website Issues:» representativeness»mixed in with, or mistaken for,

spam» requires access to Web»sampling frame?

25

Module 2:More Survey Research What we will Cover:

1.Interviewing techniques and Focus Groups2. A Comparison of Survey Methods: phone/ computer( survey monkey),

mail, and in-person3. Field Research: The good, the bad, and the ugly

4. Agency Records, Content Analysis and Secondary Data

26

In-person interview survey: See text, ch 9

Typically achieve higher response rates than mail surveys (80-85% is considered good)

Demeanor and appearance of interviewer should be appropriate; interviewer should be familiar with questionnaire and ask questions precisely

When more than one interviewer administers, efforts must be coordinated and controlled

Practice interviewing

27

Specialized interviewing• Two variations:»General interview guide – less

structured, lists issues, topics, questions you wish to cover; no standardized order

» Standardized open-ended interview – more structured, specific questions in specific order; useful in case studies, retrieves rich detail in responses

28

Telephone surveys 94% of all households now have

telephones Random-Digit Dialing» Obviates unlisted number problem» Often results in business, pay phones, fax

lines Saves money and time, provides safety

to interviewers, more convenient may be interpreted as bogus sales calls;

ease of hang-up

29

Comparison of the three methods Self-administered questionnaires are

generally cheaper, better for sensitive issues than interview surveys

Using mail – local and national surveys are same cost

In person or phone Interviews – more appropriate when respondent literacy may be a problem, produce fewer incompletes, achieve higher completion rates

Validity low in survey research; reliability high Surveys are also inflexible, superficial in

coverage

30

Tips on self-report items Convince subjects you will

guarantee confidentiality and anonymity

Minimize possible social undesirability you are asking respondents to admit

Phrase questions in non-judgmental manner

Bear in mind “fading memory” when setting time frame

31

Focus groups: See Putting it all together, pp256,257 in text

12-15 people brought together to engage in guided group discussion of some topic( e.g. addicts& recovery)

Members are selected to represent a target population, but cannot make statistical estimates about population

Most useful when precise generalization to larger group is not necessary

May be used to guide interpretation of questionnaires following survey administration

Examples: Drug addicts and reentry, homeless sex offenders, teenagers and sexting

32

Getting OutModule 3: Field

Research

33

Field Research: See Text Chapter 11• Field research encompasses two different

methods of obtaining data:

• Direct observation( example-bar room behavior; seat belt use, community disorder indices)

• Asking questions; modes of recording observations

• May yield qualitative and quantitative data

• Often no precisely defined hypotheses to be tested

• Can be used to make sense out of an ongoing process

Topics Appropriate to Field Research• Gives comprehensive perspective – enhances validity

• Go directly to phenomenon, observe it as completely as possible

• Especially appropriate for topics best understood in their natural setting• How street-level drug dealers distinguish customers

• Studies of ‘vice’, e.g., prostitution and drug-use.

• Aspects of physical settings, Disney World, social control

Various Roles of the Observer (Gold, 1969)• Complete participant – participates fully; true

identity and purpose are not known to subjects• E.g., posing as a bar patron; becoming a police officer; or

corrections worker

• Participant-as-observer – make known your position as researcher and participate with the group• E.g., study of active drug users; Julie Mueler and the Guardian

Angels

• Observer-as-participant – make known your position as a researcher; do not actually participate • E.g., Observational study of police patrol—”ride-along” ; research

on gangs

• Complete observer – observes without becoming a participant • E.g., court observation, Chicago neighbourhood

study( windshield studies)

Observer statusBe aware of, and document role of

researcher (extent of participation)

Be aware that all observation is subjective.

Be aware of the possible “effect of participation”.

Be aware of often competing ethical and ‘scientific’ values related to all observational studies.37

Asking Questions• Field research is often a matter of going where the action is and simply watching and listening

• Also a matter of asking questions & recording answers

• Field research interviews are much less structured than survey interviews

• Ideally set up and conducted just like a normal, casual conversation

Preparing for the Field• Access to formal organizations

• Find a sponsor, write a letter to executive director, arrange a phone call, arrange a meeting

• Access to subcultures

• Find an informant (e.g., person who works with offenders), use that person as your “in”

• Snowball sampling is useful as informant identifies others, who identify others, etc.

Module 4:Sampling

An Overview of Sampling Techniques: See Ch.8 in textSampling DistributionsThe Normal CurveSampling ErrorProbability SamplingNon-Probability SamplingIllustrations: NCVS, British Crime Survey,

Australian Crime Survey

41

Key Terms to Remember: SamplingSimple random samplingTable of random numbersConfidence intervalsStratified samplingPopulationProbability vs. non-probability samplingEqual probability of selection

method( EPSEM)Focus: random sampling from probation case

files42

Module 6: Recording Observations in the Field

• Note taking, tape recording when interviewing and when making observations

• Videotaping or photographs can make records of “before” and “after” some physical design change

• Field notes – observations are recorded as written notes, often in a field journal; first take sketchy notes and then rewrite your notes in detail

• Structured observations – observers mark closed-ended forms, which produce numeric measures

Linking Field Observations and Other Data

• Useful to combine field research with surveys or data from official records• Baltimore study of the effects of

neighborhood physical characteristics on residents’ perceptions of crime problems (Taylor, Shumaker, & Gottfredson, 1985)

• Perceptions – surveys;

• Physical problems – (1) observations, (2) actual population and crime information - census data & crime reports from police records

Strengths and Weaknesses of Field Research

• Provides great depth of understanding

• Flexibility (no need to prepare much in advance)

• More appropriate to measure behavior than surveys

• High validity; quant. measures – incomplete picture

• Low reliability – often very personal

• Generalizability – personal nature may produce findings that may not be replicated by another

• Precise probability samples can’t normally be drawn

Other Sources to ConsiderModule 7: Secondary Data, Content Analysis

46

Secondary Data: See text, ch. 12• Data from agency records – agencies collect

a vast amount of crime and CJ data

• Secondary analysis – analyzing data previously collected

• Content analysis – researchers examine a class of social artifacts (typically written documents) See Text example; p 329 Terrorist Recruitment

Topics Appropriate for Agency Records

• Most commonly used in descriptive or exploratory studies

• Content analysis often center on links between communication, perceptions of crime problems, individual behavior, CJ policy

Types of Agency Records

• Published Statistics – gov’t organizations routinely collect and publish compilations of data (e.g., ABS, BOCSAR, AIC) often available in libraries and online

• Nonpublic Agency Records – agencies produce data not routinely released (e.g., police departments, courthouses, correctional facilities)

• New Data Collected by Agency Staff – collected for specific research purposes; less costly + more control

Units of Analysis in Criminal Justice Data

Criminal Activity• Incidents• Crimes violated• Victims• Offenders

Court Activity• Defendants• Filings• Charges and Counts• Cases• Appearances• Dispositions• Sentences

Apprehension• Arrests• Offenders• Charges• Counts

Corrections• Offenders• Admissions• Returns• Discharges

Sources of Reliability and Validity Problems

• Virtually all CJ record keeping is a social process – “social production of data”

• Records reflect decisions made by CJ personnel as well as actual behavior by juveniles and adults

• Discretion factors in to recordkeeping

• CJ organizations are more interested in keeping track of individual cases than in examining patterns

• Potential for clerical errors due to volume of data

Content Analysis• Systematic study of messages – can be

applied to virtually any form of communication

• Decide on operational definitions of key variables

• Decide what to watch, read, listen to & time frame

• Analyze collected data

• Well suited to answer “who says what, to whom, why, how, and with what effect?”

Aspects of Sampling and Coding in Content Analysis 2

• Reminders:

• Remember operational definition of variables, and their mutually exclusive & exhaustive attributes

• Pretest coding scheme

• Assess coding reliability via intercoder reliability method and test-retest method

Secondary Analysis• Sources – websites, libraries

• AIC, http://www.aic.gov.au/

• BOCSAR, http://www.bocsar.nsw.gov.au/

• BJS, NCVS, ICPSR, NACJD

• Advantages – cheaper, faster, benefit from work of skilled researchers

• Disadvantages – data may not be appropriate to your research question; least useful for evaluation studies (which are designed to answer specific questions about specific programs)

Review: Key Terms in Chapters 6-12 in text