Using Qualitative Data Analysis Software

Michelle C. Bligh, Ph.D.

Claremont Graduate University

March 18, 2005

Why Qualitative Assessment?

“Study the box.”

What is Qualitative Research?

"Qualitative inquiry is an umbrella term for various philosophical orientations to interpretive research.” - Glesne and Peshkin (1992)

"Qualitative research is a loosely defined category of research designs or models, all of which elicit verbal, visual, tactile, olfactory, and gustatory data in the form of descriptive narratives like field notes, recordings, or other transcriptions from audio- and videotapes and other written records and pictures or films.” -Preissle

Advantages of Qualitative Research

Greater depth and detailRichness and holismFlexibility/lack of constraintsFocus on naturally occurring, ordinary events in

their natural settingsData are collected in close proximity to the situation Influences of context are not stripped awayAllow emphasis on processes, of how and why

rather than just what

Advantages of Qualitative Research (continued)

UndeniabilityLead to new integrations/interpretationsCan avoid pre-judgments/halo effectsConsistencySupplement, validate, explain, illuminate,

or reinterpret quantitative data

Disadvantages of Qualitative Research

Extremely time-consuming/labor intensiveData overloadSubjectivity/researcher biasReactivityDependent on researcher’s attributes/skillsPsychologically draining

Sources of Data

Open-ended questionsLogs, journals, or diariesObservationsStoriesCase studies Individual ‘interviews’/Oral examsDiscussion groups/Focus groupsEtc.

Your Approach Depends On…

1. The focus of your study and the themes you want to address

2. The needs of those who will use the information

3. Your resources (time, energy, money, software available)

Qualitative Analysis (Miles & Huberman)

Data reduction– Selecting, focusing, simplifying

Data display– Creating organized, compressed

representations of informationConclusion Drawing and Verification

– Deciding what things mean and testing them for plausibility/validity

Coding

Coding is analysis Codes are tags or labels for assigning units of

meaning to the descriptive or inferential information compiled

It is the meaning that matters Codes are used to retrieve and organize the

chunks of information, so you can quickly find, pull out, and cluster the segments relating to a particular topic

Types of Codes

Descriptive: attributing a class of phenomena to a segment of text (e.g., spelling)

Interpretive: include a more complex, underlying meaning (e.g., unsupported argument)

Pattern: inferential and explanatory; group codes into a smaller number of themes or constructs; analogous to cluster and factor analysis in statistics (e.g., thoroughness)

The process of coding

Create a provisional “start list”– Usually anywhere from 12 – 60 – Get them on a single page for reference– Make sure they are organized/structured

Create code definitions Revise coding scheme

– Filling in: adding, reconstructing preexisting codes– Extension: recoding with a new theme or insight– Bridging: seeing new relationships– Surfacing: identifying new categories

The process of coding (cont.)

Structure is key: codes should relate to one another, they should be part of a governing structure

Structure includes larger, more conceptually inclusive codes, and smaller, more differentiated codes

Pattern codes should represent a web of meaning that is grounded in the data

Uses of Qualitative Software

Data reduction– Retrieving text that has pre-determined

significance

Text exploration– Helping researcher recognize underlying

themes of the text

Advantages of CAQDAS

Makes the sheer volume of data more manageable

Helps to selectively retrieve information– Can summarize results in structured lists and tables

Helps to evaluate the weight of supporting vs. non-supporting data– Can report results in comparative ways

Helps to provide linkages to other types of data and perspectives– Can integrate qualitative and quantitative data

Types of CAQDAS

Text retrieval– Examples: the General Inquirer, CATA,

TEXTPACK, WordStat, Diction, ZyINDEX, The Text Collector

Text analysis– Examples:

• Atlas/TI,

• ETHNOGRAPH,

• NUDIST

How to Choose

What kind of computer user am I?Am I choosing for one project or for many?What kind of projects and databases will I be

working on?What kinds of analyses am I planning to do?How important is it to maintain close

proximity to the data?What are your financial constraints/access to

programs?

Text Retrieval Programs

Designed to search for, retrieve, and/or count words and phrases

Search programs– Used in preliminary data analysis to determine

whether and where pre-specified words and phrases appear and in what context

Content Analysis programs– Take inventories (make frequency distributions) of all,

or pre-specified, words contained in text

Text Retrieval: Primary Questions

What words are addressed in a text?Where are particular words used in a text?How do documents differ in terms of

vocabulary usage?What concepts are addressed in a text?To what extent are concepts of interest

addressed in a text?

Typical Features of Text Retrieval Programs

Generate text frequency distributionsGenerate vocabulary comparisons among different

textsWork with key-word listsGenerate key-word in context lists (KWIC)Search for root words (innovat*)Generate words category counts and statisticsConduct proximity searches (w/i 5 words)Conduct Boolean operator searches (innovation if

creativity not w/i 5 words)

Text Analysis Programs

Developed explicitly for the purposes of description, interpretation, and theory building

Facilitate identifying and coding elements of theoretical interest, establishing relationships and building connections

A.k.a. Code-and-Retrieve Programs (HyperQual2, Kwalitan, the Data Collector)/Code-Based Theory Builders (ATLAS/ti/NUDIST, Code-a-Text)

Primary Questions

How often do specific codes occur?How often do specific code sequences

occur?Are code sequences indicative of themes?Are code linkages indicative of conceptual

relationships?

Primary Functions of Text Analysis Programs

Attaching codes to segments of textSearching for and assembling coded

segments of textSearching for code sequences (look for

closely related or overlapping codes to identify patterns and relationships)

Counting frequencies of codes, code sequences, or counter-evidence

Practical Issues

Different types of programs can be used in concert or sequentially

Text must be computer readable: transcription, scanning, or importing

Special attention must be paid to formatting issues

All CQDA programs still require interpretation on the part of the researcher

Practical Issues (continued)

Reliability problems usually due to the ambiguity of word meanings, category definitions, or coding rules

Construct validity: constructs should be correlated with other measures of the same construct

Hypothesis validity: constructs should relate in theoretical ways to other constructs

Face validity: constructs should appear to measure what they do

Semantic validity: persons familiar with the language of the texts should agree that the list of words in a category have similar meanings

Advantages of CAQDAS

Stability of the coding scheme leads to increased consistency

Explicit coding rules yielding comparable results across multiple graders and over time

Saves time, freeing instructor to focus on interpretation and explanation

Easy manipulation of text to create different types of output and emphases

Ability to process large amounts of data in less time and saves paper

Limitations of Text Retrieval Programs

Lack of natural language processing capabilities (ambiguous concepts, broader context is lost)

Insensitivity to negation, irony, tone Inability of researcher to provide a completely

exhaustive listing of key words Inability of software to resolve references back

and forth to words elsewhere in the textCan result in “word crunching”: transforming

rich meanings into meaningless numbers

Limitations of Text Analysis Programs

Initial time investment Initial monetary investmentOutput can be tricky for studentsCan lead to a tendency to focus on details

rather than the big pictureThey don’t do the analysis for you!