agenda 2 main qa approaches coding exercise 1 coding exercise 2 slides on qualitative analysis ...

48

Upload: ada-owens

Post on 19-Dec-2015

221 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming
Page 2: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Agenda

2 Main QA approaches Coding exercise 1 Coding exercise 2 Slides on Qualitative Analysis Brainstorming Exercise (if time) Affinity Diagramming Exercise (if

time)

2

Page 3: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994)1 Affixing codes to a set of field notes

drawn from data collection2 Noting reflections or other remarks

in margin3 Sorting or shifting through the

materials to identify similar phrases, relationships between themes, distinct differences between subgroups and common sequences

Page 4: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994)4 Isolating patterns and processes,

commonalties and differences, and taking them out to the field in the next wave of data collection

5 Gradually elaborating a small set of generalisations that cover the consistencies discerned in the data base

6 Confronting those generalisations with a formalised body of knowledge in the form of constructs or theories

Page 5: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

2 general research approaches

5

deductive approach vs inductive approach

Page 6: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

deductive research approach

6

THEORY

HYPOTHESIS

OBSERVATION

CONFIRMATION

Top-down approach

Theory testing

A priori codes

Page 7: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

inductive research approach

7

THEORY

TENTATIVE HYPOTHESIS

PATTERN

OBSERVATION

bottom-up approach

Theory building

Emergent codes

Page 8: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

8

deductive or inductive

Page 9: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Often use a hybrid approach A set of a priori codes reflecting your

understanding of the topic and your research questions

Emergent codes added as you code the data and find other factors/topics/codes that you had not considered

9

Page 10: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Exercise 1

Open coding Inductive analysis Exploratory research Theory building research

http://b.socrative.com/login/student/ Room: 7f156b7b

10

Page 11: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Exercise 2

Coding with pre-defined categories Deductive analysis Theory Testing

http://b.socrative.com/login/student/ Room: 7f156b7b

11

Page 12: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming
Page 13: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Qualitative Inquiry - Purpose

The purpose of qualitative inquiry is to produce findings. The Data Collection process is not an end in itself. The culminating activities of qualitative inquiry are analysis, interpretation, and presentation of findings.

Page 14: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Qualitative Inquiry - Challenge

To make sense of massive amounts of data, reduce the volume of information, identify significant patterns and construct a framework for communicating the essence of what the data reveal

Page 15: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Qualitative Inquiry - Problem

‘…have few agreed-on canons for qualitative data analysis, in the sense of shared ground rules for drawing conclusions and verifying sturdiness’ (Miles and Huberman, 1984)

Page 16: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

The Creativity of Qualitative Inquiry

‘..the human element of qualitative inquiry is both its strength and weakness - its strength is fully using human insight and experience, its weakness is being so heavily dependent on the researcher’s skill, training, intellect, discipline, and

creativity. The researcher is the instrument of qualitative inquiry, so the quality of the research depends heavily on the qualities of that human being’

(Patton, 1988)

Page 17: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

The Science and Art of Qualitative Inquiry (Patton, 1988) The Science

The scientific part is systematic, analytical, rigorous, disciplined, and

critical in perspective The Art

The artistic part is exploring, playful, metaphorical, insightful, and creative

Page 18: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

1. Analysis Considerations

1 Words2 Context (tone and inflection)3 Internal consistency (opinion shifts during

groups)4 Frequency and intensity of comments

(counting, content analysis)5 Specificity6 Trends/themes7 Iteration (data collection and analysis is an

iterative process moving back and forth)

Page 19: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

2. The Procedures

1 Coding/indexing2 Categorisation3 Abstraction4 Comparison5 Dimensionalisation (relationships)6 Integration7 Iteration8 Refutation (subjecting inferences to scrutiny)9 Interpretation (grasp of meaning - difficult to

describe procedurally)

Page 20: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Critical Thinking ‘Critical Thinking calls for a persistent effort to

examine any belief or supposed form of knowledge in the light of the evidence that supports it and the further conclusions to

which it tends’ (Glaser, 1941) or more simply!

Critical Thinking means weighting up the arguments and evidence for and against.

Page 21: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Critical Thinking• Key points (Glaser, 1941):

– Persistence: Considering an issue carefully and more than once

– Evidence: Evaluating the evidence put forward in support of the belief or viewpoint

– Implications: Considering where the belief or viewpoint leads; what conclusions would follow; are these suitable and rational; and if not, should the belief or viewpoint be reconsidered

Page 22: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Guidance for Creative Thinking

1 Be open2 Generate options3 Divergence before convergence4 Use multiple stimuli - triangulate5 Side track, zig-zag, and circumnavigate6 Change patterns of thinking7 Make linkages8 Trust yourself9 Work and play at it

Page 23: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

The Credibility of Qualitative Analysis

1 Rigorous techniques and methods for gathering high-quality data that is carefully analysed, with attention to issues of validity, reliability, and triangulation

2 The credibility of the researcher, which is dependent on training, experience, track record, status, and presentation of self

3 Philosophical belief in the phenomenological paradigm, that is, a fundamental appreciation of naturalistic inquiry, qualitative methods, inductive analysis and holistic thinking

Page 24: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

A Credible Qualitative StudyA credible qualitative study needs to address the following issues:

1 What techniques and methods were used to ensure the integrity, validity, and accuracy of the findings

2 What does the researcher bring to study in terms of qualifications, experience, and perspective

3 What paradigm orientation and assumptions ground the study

Page 25: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Principles of Analysing Qualitative Data1 Proceed systematically and rigorously

(minimise human error)2 Record process, memos, journals, etc.3 Focus on responding to research questions4 Appropriate level of interpretation appropriate

for situation5 Time (process of inquiry and analysis are

often simultaneous)6 Seek to explain or enlighten7 Evolutionary/emerging

Page 26: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Inter-rater reliability

What if you have more than one person coding? How much agreement do they have? At what point should you test their

agreement? Other than comparing counts, how can

you validate the coding/analysis? https://www.academia.edu/458025/The_place_of_i

nter-rater_reliability_in_qualitative_research_an_empirical_study

26

Page 27: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

27

Articulate:•who users are•their key tasks

User and task descriptions

Goals:

Methods:

Products:

Brainstorm designs

Task centered system design

Participatory design

User-centered design

Evaluate

Psychology of everyday things

User involvement

Representation & metaphors

low fidelity prototyping methods

Throw-away paper prototypes

Participatory interaction

Task / Cognitive scenario walk-through

Refined designs

Graphical screen design

Interface guidelines

Style guides

high fidelity prototyping methods

Testable prototypes

Usability testing

Heuristic evaluation

Completed designs

Alpha/beta systems or complete specification

Field testing

Interface Design and Usability Engineering

Page 28: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

brainstorming

the point is: to generate MANY, WIDE-RANGING ideasnutty and absurd are GOOD. go for the

extremes (to get out of the rut)

riff off other’s ideas.

the point is NOT: to generate excellent, complete, feasible

ideas … pressure stifles

to develop or critique ideas … go wide. deep is for later.

Page 29: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

process

1. prepare a list of topics / questionsahead of time; or in a preliminary brainstorm

2. facilitator takes team through list of topics switch topic when energy ramps down

3. Note taker takes notes (very important)

4. switch roles so everyone can play

5. ground rules

6. Follow up

Page 30: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

brainstorming is like popcorn

Page 31: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

ground rules Postpone and withhold your judgment of

ideas: never criticize

Encourage wild and exaggerated ideas

Quantity counts at this stage, not quality

Switch topics when the popcorn slows down

Build on the ideas put forward by others

Every person and every idea has equal worth

Elect a facilitator (calls switches) and a note-taker (one thought per post it!)

Page 32: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Post brain-storm collect the notes

go through carefully, with judgment turned on

look for interesting, surprising ideas that might work ideas that will combine well promising directions on which you should

brainstorm more

keep your notes. at a later design stage, come back to them and see if anything else has become useful in the meantime.

Page 33: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Sometimes you have a lot of ideas to make sense of!

33

Page 34: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

work consolidation:abstracting specific insights

one tool: the affinity diagram can use to “consolidate” insights from collected

or generated data. for example: brainstorming about design problems

categories of problems

brainstorming about design ideas categories of ideas

comments from users categories of desirable / successful features

Page 35: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

how do you make an affinity diagram?

1. team writes down all data & insights on post-it notes; be sure you can link the post-it detail back to its source!

2. stick one post-it on the wall a whiteboard or big sheet of butcher paper is best

3. arrange the other post-its around it, grouping by affinity to each other. iteration will be required.

4. look at each group and see what it has in common; name and describe each group.

5. “snapshot” the result for documentation1. digital photo your design website or notebook2. transfer post-its onto paper, 1 sheet / notes-cluster

scan website

Page 36: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

why does an affinity diagram work?

• use physical arrangement/proximity to understand connections

• openness to serendipity

• low cost to rearrange ideas

• many variants:

arrange along axes rather than by affinity

tie causes to effects

group evidence under assertions

Page 37: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming
Page 38: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and Konstantin Beznosov. 2008. Guidelines for designing IT security management tools. In Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology (CHiMiT '08). ACM, New York, NY, USA, , Article 7 , 10 pages. DOI=10.1145/1477973.1477983 http://doi.acm.org/10.1145/1477973.1477983

38

Page 39: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Methodology (Phase I)

• Field studies• Interviews• Questionnaires• Prototyping• Cognitive walkthroughs• Surveying other literature

• Field studies• Interviews• Questionnaires• Prototyping• Cognitive walkthroughs• Surveying other literature

• Field study:• Interviews • Participatory observation

• Field study:• Interviews • Participatory observation

39

Page 40: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Methodology (Phase I)

40

Categorized List of

Guidelines

Categorized List of

Guidelines

Page 41: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

41

High level Category

Low level Category

Guideline

Guideline ID number

Page 42: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Methodology (Phase I)

42

Guidelines Framework

Page 43: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Methodology (Phase II)

43

Page 44: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Framework for classification of guidelines

44

Task SpecificTask Specific

Organizational ComplexityOrganizational Complexity

Technological ComplexityTechnological Complexity

Configuration and Deployment

Configuration and Deployment

Diverse Stakeholders

Diverse Stakeholders

General Usability GuidelinesGeneral Usability Guidelines

Specificity

Intensive AnalysisIntensive Analysis

Distributed ITSMDistributed ITSM

CommunicationCommunication

Page 45: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

45

Framework for classification of guidelinesTask Specific GuidelinesTask Specific Guidelines

General Usability GuidelinesGeneral Usability Guidelines

Technological Complexity GuidelinesMake tools combinable [8,9,20,26] Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45]

Help task prioritization [15,44] Use different presentation / interaction methods [1,4,5,29,41,48,49]

Provide customizability [9,33] Support knowledge sharing [9,12,14,27,32,37,47]

Technological Complexity GuidelinesMake tools combinable [8,9,20,26] Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45]

Help task prioritization [15,44] Use different presentation / interaction methods [1,4,5,29,41,48,49]

Provide customizability [9,33] Support knowledge sharing [9,12,14,27,32,37,47]

Organizational Complexity GuidelinesOrganizational Complexity Guidelines

Diverse Stakeholders Guidelines

Provide flexible reporting [9,18,33,35]Provide an appropriate UI for stakeholders [9,35]

Diverse Stakeholders Guidelines

Provide flexible reporting [9,18,33,35]Provide an appropriate UI for stakeholders [9,35]

Distributed ITSM Guidelines

Support collaboration [6,7,20]Work in a large workflow [8,9,20]

Distributed ITSM Guidelines

Support collaboration [6,7,20]Work in a large workflow [8,9,20]

Communication Guidelines

Provide communication integration [6,7,28,45]Facilitate archiving [17,21]

Communication Guidelines

Provide communication integration [6,7,28,45]Facilitate archiving [17,21]

Intensive Analysis GuidelinesProvide customizable alerting [20]Provide automatic detection [26,41]Provide data correlation and filtering [1,26]

Intensive Analysis GuidelinesProvide customizable alerting [20]Provide automatic detection [26,41]Provide data correlation and filtering [1,26]

Configuration and Deployment GuidelinesMake configuration manageable [3,20]Support rehearsal and planning [3,6,7,20,44]Make configuration easy to change [20,46]Provide meaningful errors [20, 34,46]

Configuration and Deployment GuidelinesMake configuration manageable [3,20]Support rehearsal and planning [3,6,7,20,44]Make configuration easy to change [20,46]Provide meaningful errors [20, 34,46]

Mor

e Sp

ecifi

cM

ore

Spec

ific

Page 46: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

Class will be 1 big group3 volunteer note takers

Problem: How to design the user interface for

a car proximity detection system

Brainstorm 3 aspects of the problem: (e.g., physical form factor, safety

issues, input techniques, etc.) go: 5 minutes

Page 47: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

affinity diagram exercise

Now take your notes from the earlier brainstorming and create an affinity

diagram

go: 8 minutes

Page 48: Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming

debrief