senior scientist, sintef ikt adjunct associate professor ... · •chapter 11 – analysing...
TRANSCRIPT
Amela Karahasanović
Senior Scientist, SINTEF IKT
Adjunct Associate Professor, DESIGN
www.sintef.no
Sintef IKT, Forskningsveien 1
Plan for today
• Chapter 11 – Analysing Qualitative data
• Book chapter + some examples – Stages of analysis
– Grounded theory
– Content analysis
– Quiz before the break
– Text analysis
– NVivo example
– Multimedia analysis
2
• Do you like my app?
3
• What they are doing when they are using your app?
• What kind of problems they might have?
• What they do in their spare time?
4
• Quantitative data: time, error rate, rankings
5
0
5
10
15
20
25
30
35
40
45
1 3 5 6 7 10 11 12 13 17 18 20 21 23 24 25 26
Unified Library Application - Quit
Unified Library Application
Title Information
ReturnItem
Reserve Title
Lend Item
Insert Title Window
Find Title
Compilation
Subject_id 15 Task 2
Sum of Seconds
Minutes
Visited
• Qualitative data: interview notes, survey responses, video and audio material
• Quantitative versus qualitative data analysis
– Involves human coding (biased, subjectivity)
– Learning by doing and reading the literature/theories
6
14:04 What prohibited you in making the optimal decisions? Definitely, both radar screen was very important, by that I could decide if an airplane was ready for push back. The human/machine interface is important. This update took too long time, maybe 2-3 sec. It should be 1 sec maximum. The interfaces are important, overlapping aircraft symbols, is confusing, the clarification is not clear. My mental work flow will slow down. Another point is that the preview of what is coming next has to be well defined.
• Four controllers – 30 minutes each; question about
decisions, the process in control tower, the tools that were used, the experiment
7
Stages of qualitative analysis
• (1) Information about substance (group of users, their behaviour); identify its major components – Substance: decision making processes of ATM
controllers; components: communication, interface, procedures, airport characteristics
• (2) Study properties of each component and their relations – Characteristics of communication; how it is related
with other components
• (3) Using knowledge on each component to understand the substance
8
Online behaviour of Internet users
• (1) Behaviour influenced by personality education and ICT experience
• (2) Study these factors and relations; theories behind
• (3) Examine how each component influence user behaviour
• Experience and knowledge of the researcher critical for the interpretation
9
Grounded theory
• Not a theory
• Qualitative research method
• Goal – develop theory grounded in a systematically collected and analysed data
• Theory -> Hypothesis -> Study -> Data-> Y/N
• Study ->Data -> Theory
– Several rounds; reverse engineering
10
Remember!
• No pre-formed hypothesis!
• No favorite solutions!
• Creativity and open mind!
• Let the data to lead you!
11
Procedures for grounded theory
• Open coding – Identify phenomena
• Development of the concepts – Group phenomena into concept
• Grouping concepts into categories – Grouping and interpretation
• Formation of a theory – Crate inferential and predictive statements on
phenomena
12
Advantages
• Systematic approach to analysis of qualitative data
• Allows generating theory grounded in data and coding
• One can study data early on
13
Disadvantages
• One can be overwhelmed by details
• Theories might be difficult to evaluate
– Textual data, less strict measures, coding
• Might be biased
• Keep in mind
– Be open-minded and creative
– Listen to data
14
Content analysis
• Systematic and replicable technique for compressing text into categories by coding
• Any technique for making inferences by objectively and systematically identifying specified characteristics of messages
• Semi-quantitative method
15
Content
• Media content
– Books, journals, TV programs, websites, blogs, film, music
• Audience content
– Surveys, questionnaires, interviews, focus groups, diaries, observations
– Text, video, audio
16
Why do we need content?
• Interviews, open-ended questions…
• Evaluating a new version of Word
– Number of errors
– Solution time
– Satisfaction
– > Explanations
– > Suggestions for improvement
– > Group interaction (video)
17
Quiz time
• What are the disadvantages of qualitative analysis?
• What is grounded theory?
• What are the advantages of grounded theory?
• What is content analysis?
• What types of content have you collected in your study?
• How are you going to analyse your content?
18
Before you start content analysis
• Clear definition of the data set – Interviews of 7 Telenor costumers
• Online community – Public, private messages
– Period of time
– Population (how often they visit, do they post any messages)
• Context – Security (government versus game industry)
19
Analysis - coding
• More than word counting
• A priory coding
– Codes from the literature
– Analysis
– Several coders
– Reliability check if coding is consistent
– Work fine for known domains
20
Emergent coding
• Appropriate for new topics
• Several researchers examine the data and develop key coding categories
• Comparison, discussion, common list
• Multiple coders do the coding
• Reliability measures calculated; if ok proceed with the coding; if not go back
21
Identifying coding categories
• Very important as they lead the analysis
• Demanding
• Codes are coming from
– Theoretical framework
– Researchers interpretation (research denoted concepts)
– Participants (in-vivo codes)
22
Theoretical framework
• We start research by literature review and identifying theoretical framework related to our research topic
• Difficulties experienced by senior citizens when using computers – Human capabilities : cognitive, physical,
perceptual
• Taxonomies – Categories of users, tasks, errors
23
Researcher denoted concepts
• Identify patterns, opinions, behaviour in your data -> codes
• Open coding
• "I was looking for 'find' …and it was not there…It was so irritating" -> find, frustration
24
• In vivo-codes – Participants have a good descriptions
– Use it in your coding
– "Curriculum integration" from the one parent's response, name of a TV show in the analysis of QoE
• Building a code structure – Participants express same ideas id different ways
– Code list – nomenclature
– Several levels (different levels of details)
25
Coding the text
• Read the text (watch the video) before start the coding
• Difficult to find anything interesting – too many interesting things
• Procedure:
– Look for specific items
– Ask questions constantly about the data
– Making comparisons constantly at various levels
26
Look for key items
• Some statements have more valuable information – Objectives: computers for education
– Actions: click on
– Outcomes: error message appeared
– Consequences: I stopped using it
– Causes: my old laptop
– Context: I was in bus
– Strategies: I first browse
27
Ask questions about data
• The art of asking questions in a larger context
– Sensitizing questions
• What is happening here? What did the user click? How did she reach www.ifi.uio.no?
– Theoretical questions
• What is relationship between two factors? How does interaction change over time?
28
Making comparisons of data
• Compare instances under different categories – Frequency for different capabilities (physical,
cognitive, perceptual) for elderly
• Compare the results between different groups – Age, background, family support
• Compare to the previous published results – Same/contradictory, related studies
• Computer software – NVivo, Concordance, SPSS TextSmart
29
Ensuring high-quality analysis
• Subjective analysis – Which category? Are they in the same group? Is 'good'
and 'ok' the same?
• Validity – Use of well-established and well-documented
procedures to increase the accuracy of findings; Did we get it right?
• Reliability – Consistency of results; Would other researchers
make the same conclusions based on the same data set?
30
Validity
• Construction of a database with the collected data material: raw data (notes, documents, photos…) results f the analysis
• Also increase reliability
• Data source triangulation
– Interviews, observations, diaries
• Avoid having pet theories
• Consider alternative theories
31
Reliability
• Same word might have different meanings
• Body language, face expressions, drawings might have different meanings
• Large studies -> different coders analyse different data subsets
• Different people should code in the same way
32
• Intra-coder reliability (stability)
– Whether the same coder do the same throughout the whole process; Would he do the same next time? (50% A, 30% B, 20% C)
• Inter-coder reliability (reproducibility; investigator triangulation)
– Whether different coders would do the same? Multiple coders with different backgrounds
33
• To achieve good reliability
– Good coding instructions
– Training
– Test on the limited amount of data
• Reliability measure
% agreement = the number of cases coded the same way by multiple coders/ the total number of cases
34
• Coders can do the same by chance
• Cohen's Kappa (0-1; 0 – coded the same by chance; 1 – perfect reliability
• K=(Pa – Pc)/(1-Pc)
• Pa – percentage of cases on which the coders agree
• Pc – percentage of agreed cases by chance
• More that 60% is satisfactory
35
36
Coded by both coders
Coded by
chance
Expected agreement when the data is coded by chance 0.37*0.39=0.14 K=(Pa – Pc)/(1-Pc) Pa = 0.26 + 0.12 + 0.35 = 0.73 Pc = 0.14 + 0.04 + 0.18 = 0.36 K= 0.58
Subjective versus objective coders
• Subjective/inside coders
– Designed the study, developed the coding scheme, collected the data
– (+) know the literature; know the topic; easier to interpret the data; minimal training
– (-)might be biased and unable to see new patterns, new behaviour
• Objective/outside coders
37
Analysing multimedia content
• Low cost, reach data source
• Not easy to analyse
• Coding is needed – time-consuming, tedious, impractical
• Same principles as for text
• How to make it easier – Select a subset of a data set
– Partially automated approach • Human coders annotate the data; the detector used to
automatically annotate the rest
38