Considering Qualitative Evaluation

University of Calgary:

University of St. Andrews:Cork Institute of Technology:

Tableau:

Sheelagh Carpendale, Alice Thudt, Jo Vermeulen, Jagoda WalnyUta HinrichsTrevor HoganMelanie Tory

Purpose of today’s tutorial

• evaluation is essential • evaluation is a changing research area • new methods continue to emerge • will focus on methods for collecting

qualitative data– observation– interviewing

From today’s tutorial you will

• gain a richer understanding of the benefits and challenges of qualitative empirical research

• experience observational exercises• experience variation in interviewing • gain practical ideas to apply to your own

gathering of qualitative data

Albert Einstein

‘Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted’

Overview Talk 1: Setting the stage: qualitativeTalk 2: Observation as a skillActivity 1: Practicing observationActivity 2: Interviewing 1Activity 3: Interviewing 2Talk 3: Interviewing as a skillActivity 4: Interviewing 3 Panel: open discussionTalk 4: In summary – next step analysis

http://bit.ly/qualeval@QualEval

use hashtag #QualEval

introducing ourselves

Sheelagh Carpendale• Professor, University of Calgary• Passionate about qualitative research• Uncountable qualitative studies in the last 20 years

Some papers on empirical research • S. Carpendale. (2008). Evaluating Information Visualizations, in A.Kerren, J.T. Stasko, JD. Fekete, and C. North (Eds.). Information

Visualization – Human-Centered Issues and Perspectives, Volume 4950 of LNCS State-of-the-Art Survey, Springer, pp. 19-45, 2008.• H. Lam, E. Bertini, P. Isenberg, C. Plaisant, S. Carpendale. (2012) Empirical Studies in Information Visualization: Seven Scenarios. IEEE

Transactions on Visualization and Computer Graphics, 18(9): 1520-1536, 2012. Appeared online: 30 Nov. 2011.• M. Brehmer, S. Carpendale, B. Lee, M. Tory (2014). Pre-design empiricism for information visualization: Scenarios, methods, and

challenges. Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, (BELIV’14), pages 147-515. ACM Press

• P. Isenberg, T. Zuk, C. Collins, S. Carpendale. (2008). Grounded Evaluation of Information Visualizations. Proc. Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV’08) (held at ACM CHI’08), pp. 56-63, ACM DL.

• T. Zuk, L. Schlesier, P. Neumann, M.S. Hancock, S. Carpendale. (2006). Heuristics for Information Visualization Evaluation. Proc. Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV’06), (held at Advanced Visual Interfaces, AVI 2006), pp. 55–60, ACM Press.

Uta HinrichsAssistant Professor, University of St Andrews

Studying Experiences with Visualizations In-Situ

Observation • Interviewing • Focus Groups Uta Hinrichs, Simon Butscher, Jens Müller and Harald Reiterer. Diving in at the Deep End: The Value of Alternative In-Situ Approaches for Systematic Library Search. CHI 2016.

Trevor Hogan, Uta Hinrichs, Eva Hornecker. The Elicitation Interview Technique: Capturing People’s Experiences of Data Representations. IEEE TVCG. In print.

Uta Hinrichs, Stefania Forlini and Bridget Moynihan. Speculative Practices: Utilizing InfoVis to Explore Untapped Literary Collections. IEEE TVCG, 2016.

Uta Hinrichs and Sheelagh Carpendale. Gestures in the Wild: Studying Multi-Touch Gesture Sequences on Interactive Tabletop Exhibits. CHI‘ 2011.

@uta_ente

Trevor HoganLecturer, Crawford College of Art and Design, CIT, IrelandDescribing People’s Experiences with Data

Trevor Hogan, Uta Hinrichs, and Eva Hornecker. The Elicitation Interview Technique: Capturing People’s Experiences of Data Representations. IEEE TVCG (Accepted Dec 2015, in press)

Trevor Hogan, Eva Hornecker. Blending the repertory grid technique with focus groups to reveal rich design relevant insight. Proc. of DPPI’13, ACM 2013, 116-125

Trevor Hogan, Eva Hornecker. The Personal Construction of Representation Modality. In Proceedings of the 6th Irish Human-Computer Interaction conference (iHCI’13), ACM, 107-111

Eliciting subjective accounts of experience

tactiledata

@tactiledata

Alice ThudtPhD Candidate, InnoVis Group, University of Calgary

Personal Visualization for Self-Reflection and Personal Storytelling

Understanding D IVERSITY of individual approaches to InfoVis:

• semi-structured interviewsThudt, A., Hinrichs, U., Carpendale, S. The Bohemian Bookshelf: Supporting Serendipitous Book Discoveries through Information Visualization. In Proceedings of ACM CHI 2012.Thudt, A., Hinrichs, U. and Carpendale, S. A Modular Approach to Promote Creativity and Inspiration in Search. In Proceedings of ACM Creativity & Cognition 2015.

• technology probesThudt, A., Baur, D., Huron, S., Carpendale, S. Visual Mementos: Reflecting Memories with Personal Data. In IEEE TVCG 2015.

@al_ice_t

Melanie TorySenior Research Scientist, Tableau Interested in understanding interactive visual analysis• Physical and digital artifact-mediated coordination in building design M Tory, S Staub-French, BA Po, F Wu,

Computer Supported Cooperative Work (CSCW) 17 (4), 311-351.• Note-taking in co-located collaborative visual analytics: Analysis of an observational study, N Mahyar, A Sarvghad,

M Tory, Information Visualization, 2012.• Supporting communication and coordination in collaborative sensemaking, N Mahyar, M Tory• IEEE transactions on visualization and computer graphics 20 (12), 1633-1642.• User Studies in Visualization: A Reflection on Methods, Melanie Tory, Chapter in Handbook of Human Centric

Visualization, 2014.

@vizstudylady

Jo VermeulenPostdoctoral Fellow, InnoVis Group, University of Calgary

Interested in addressing interaction challenges with non-traditional interactive technologies (e.g., context-aware systems)

Observing problems people experience when interacting, including issues with:

• discoverability of action possibilities• feedback and feedforward• understanding what technology is doing

@jozilla

Jagoda WalnyPhD Candidate, InnoVis Group, University of Calgary

Interested in observing everyday thinking practices(particularly sketching) to inform the design of information visualization interfaces. Variety of observation data:

Semi-structured interviews Sketched artifacts Interactions

@jawalnut