THE 14TH INTERNATIONAL CONFERENCE ON NATURALISTIC DECISION MAKING

JUNE 17 - 21, 2019SAN FRANCISCO, CA, USA

to all of our generous sponsors!THANK YOU

TABLE OF CONTENTSAGENDA ..................................................................... 04

ORGANIZING COMMITTEE ........................................... 13

KEYNOTE SPEAKERS ................................................... 14

INVITED SPEAKERS ..................................................... 15

NOTES ........................................................................ 17

CONFERENCE ACCESS ................................................ 21

INFORMATION ............................................................ 22

HOTEL MAP ................................................................ 233

MONDAY, JUNE 17, 2019DOCTORAL CONSORTIUM8:00 REGISTRATION & LIGHT BREAKFAST9:15 OPENING ADDRESS9:30 PAPER SESSION 1

Investigating the Key Cognitive Demands and Skills for Effective Team Training of Emergency Response Teams Managing Multi-Agency Critical Incident Events Ashley Ferkol

Critical Decision-Making and the Influence of Expectations of Post-Event Scrutiny

Brad Wright

10:30 COFFEE BREAK11:00 PAPER SESSION 2

Mitigating Respiratory Compromise Via Predictive Analytics

Christopher Hansen

An Examination of British Firearms Officers’ Performance Within a Realistic, Immersive High-Risk Scenario

Jolyon Faro

Heroes and Sidekicks: A Mixed Methods Approach to Defeating the Field’s Most Difficult Problems

Kaitlyn Roose 12:30 LUNCH1:30 PAPER SESSION 3

The Future Face of Automation: Designing Interfaces to Foster Trust Generation Within Human–Machine Teams

Faye McCabe

Team Composition, Goal Priorities and Performance: An Experimental Study of Multi-Team Systems

Olivia Brown

4

2:30 COFFEE BREAK3:00 PAPER SESSION 4

A Cognitive Engineering Approach to Improve the Design of Paramedic Protocols and Clinical Decision Support Paul Misasi

Using Video-Occlusion to Test and Train Law Enforcement Officers’ Perceptual–Cognitive Skills

Mohammadreza Jalaeian

4:00 SMALL GROUP MENTORING SESSION5:00 CLOSING REMARKS7:00 DINNER

5

TUESDAY, JUNE 18, 2019PRECONFERENCE WORKSHOPS8:00 REGISTRATION8:30 WORKSHOPS A & B

Workshop A: Designing Complex Scenarios for Evaluation of the Effectiveness of Technology for Supporting Distributed Cognitive WorkEmilie Roth, Laura Militello, & Emily Patterson

Workshop B: Assessing Mental Models Using Sero! Brian Moon

12:00 LUNCH1:30 WORKSHOPS C & D Workshop C: Cognitive Task Analysis (CTA) Gary Klein & Laura Militello

Workshop D: Applying Human-Machine Teaming Methods (HMTM) to Operational Systems

Patty McDermott & Cindy Dominguez

5:30 ADJOURN

6

WEDNESDAY, JUNE 19, 2019FULL CONFERENCE8:00 REGISTRATION8:30 WELCOME9:00 OPENING KEYNOTE

Frontier Thinking: Designing Next Generation User Experiences

Ben Shneiderman10:00 BREAK

SESSION 110:20 Cognitive Support Chair: Christen Sushereba

Analyzing ‘Home Grown’ Artifacts to Uncover Cognitive Support Requirements: The Case of Puck Boards for Scheduling Air Crews Emilie Roth, Beth DePass, Jonathan Harter, Ronald Scott, & Jeff Wampler

Toward Optimal Crewing of Future Vertical Lift: Determining the Cognitive Requirements for Rotorcraft Pilots on an Attack Mission Katie Ernst, Laura Militello, Scott Wonderly, & Emilie Roth

Increasing the Effectiveness of Collection Operators through Decision Support: A Pilot Study and the First Iteration of the Flatline System James Salley, Kathryn Kopren, William Elm, & Daniel Zelik

Using Decision Ladders Prospectively in the Aviation Design Lifecycle: Lessons Learned Victoria Banks, Katie Parnell, Katherine Plant, & Neville Stanton

11:20 BREAK11:30 Practitioner Panel: Perspectives from the Front Lines Chair: Gary Klein Panelists: Phil Jose, Ron Besuijen, Eddie Freyer, & Marcus Mainz12:30 LUNCH Invited Speaker: Judith Orasanu

7

SESSION 21:30 Panel: Current Issues and Challenges in Cyberdefense Chair: Robert Hoffman Panelists: Debi Ashenden, Vincent Mancuso, & David Schuster2:30 Team Chair: Cindy Dominguez

Monitoring Changes in Cohesion Over Time in Expedition Teams: The Role of Daily Events and Team Composition Olivia Brown, Emma Barrett, & Nicola Power

Team Communication Patterns in Critical Situations Floris van den Oever & Jan Maarten Schraagen

3:00 BREAK

SESSION 33:30 Poster Session Lightning Round4:00 Poster Session

A Framework to Improve Evaluations of Novel Decision Support Tools Jesslyn Alekseyev

8

Using the Imitation Game to Explore Orientation and Mobility Specialists’ Interactional Expertise with the Blind or Visually ImpairedGuler Arsal, Joel Suss, & Paul Ward

Team Composition, Goal Priorities and Performance: An Experimental Study of Multi-Team SystemsOlivia Brown

Cognitive Challenges in Congested Terminal Airspace - Implications for UAS IntegrationJulie Diiulio, Laura Militello, & Devorah Klein

Prestissimo Con Brio: How Conductors Make DecisionsStuart Donn

An Examination of British Firearms Officers’ Performance Within a Realistic, Immersive High-Risk ScenarioJolyon Faro

iPass: A Case Study of the Effectiveness of Automated Planning for Decision SupportSachin Grover, Sailik Sengupta, TathagataChakraborti, Aditya Prasad Mishra, & Subbarao Kambhampati

The Theoretical Foundation of MacrocognitionRobert Hoffman, Gary Klein, & David Woods

Multi-Agency Emergency Teams: Investigating Cognitive Demands, Skills, and Teamwork Competencies for Effective TrainingAshley Ferkol, Amanda Martindale, & Hugh Richards

Beyond Tilt-a-Ball : Integrating Naturalistic Decision Making and Modern Game Theory to Understand Contemporary Work LearningGarrett Howardson

Using Video-Occlusion to Test and Train Law Enforcement Officers’ Perceptual–Cognitive SkillsMohammadreza Jalaeian

Re-Writing Tomorrow: Insights in Trading SurveillanceWendy Jephson, Anna Leslie, Nick Wise, & Julie Gore

Heroes and Sidekicks: A Mixed Methods Approach to Defeating the Field’s Most Difficult ProblemsKaitlyn Roose

The Future Face of Automation: Designing Interfaces to Foster Trust Generation Within Human–Machine TeamsFaye McCabe

9

5:30 DAY 1 ADJOURN6:30 CONFERENCE BANQUET AQUARIUM OF THE BAY Please see page 19 for directions.

Avoiding “Dis-arrow-entation” in Paramedic Protocol Design: Identifying the Cognitive Requirements for Effective Clinical Decision Support in EMS Support

Paul Misasi & Joel Suss

CAP: A Decision Support System for Crew Scheduling using Automated Planning

Aditya Mishra, Sailik Sengupta, Sarath Sreedharan, Tathagata Chakraborti, & Subbarao Kambhampati

A Decision-Making Framework for Agents in Complex Environments

Eugene Santos, Hien Nguyen, Clement Nyanhongo, Gregory Hyde, Keumjoo Kim, & Jacob Russel

Eliciting Hospital Bed Capacity Management Decision Processes

Matthew Woodward, Julie Gore, & Christos Vasilakis

The Influence of “Decision Resilience” on the Development of Mental Models

Brad Wright

THURSDAY, JUNE 20, 2019FULL CONFERENCE8:00 REGISTRATION9:00 OPENING KEYNOTE:

Situation Awareness and Decision Making in the Real World: Information Attacks, Social Media, and Cognitive Resistance

Mica Endsley10:00 BREAK

SESSION 410:20 Management Chair: Neville Stanton

Accelerating Expertise in Agile Decision Making for Business with Iterative Failure in a Dynamic Naturalistic Virtual Business Lia DiBello

How Managers Frame Decision Paradoxes: New Vistas for NDM Research Adi Adar-Pras & Raanan Lipshitz

Strategic Management System Engagement and Strategic Cognition Rachael Lewis, Nicole Sutton, & David Brown

11:20 BREAK11:30 Meet the Editors: Inside the Review Process of JCDEM Chair: Jan Maarten Schraagen

Panelists: Jan Maarten Schraagen, Laura Militello, & Emilie Roth

12:30 LUNCH Invited Speaker: Daniel Serfaty

SESSION 51:30 Invited Speakers Chair: Julie Gore Lois James (1:30), Rhona Flin (2:00), & Paul Falzer (2:30)

3:00 BREAK

SESSION 63:30 AI Symposium: Current Issues and Challenges in Artificial Intelligence Chair: Robert Hoffman

Panelists: Shane Mueller, Gary Klein, Robert Hoffman, Jeffrey Morrison, James Tittle, & Peter Pirolli

5:00 DAY 2 ADJOURN

10

FRIDAY, JUNE 21, 2019FULL CONFERENCE8:00 REGISTRATION9:00 OPENING KEYNOTE

Re-Writing Product Design using Behavioral Science: The Journey to Finding and Leveraging the Value of NDM, Its Place and Impacts in How We Can Design Technology for Financial Services, and Opportunities for the Future

Wendy Jephson10:00 BREAK

SESSION 710:20 NDM Models Chair: Joel Suss

A Draft Model to Bridge a Glaring NDM Gap: Motivation Brian Moon & Shannon Bildstein

Mindless Self-Explaining vs Meaningful Self- Explaining Gary Klein, Robert Hoffman, & Shane Mueller

Do You Remember? The Role of Self-Defining Memories and Sensemaking in Generating Decisions Kelsey Merlo, Alisha Ness, & Gregory Ruark

Naturalistic Distributed Decision Making in Sociotechnical Systems: The Case of the Uber Collision with a Pedestrian Neville Stanton & Paul Salmon

11:20 BREAK11:30 How Surgeons “See” Panel Chair: Lygia Stewart Panelists: Lygia Stewart, Carlos Corvera, Edward Kim, & Insoo Suh12:30 LUNCH

SESSION 81:30 Training Macrocognitive Skills Chair: Katie Ernst

Beyond “Prove It” to “Improve It”: Video- Occlusion to Train Baseball Pitch Recognition Peter Fadde

11

The Potential of Augmented Reality for Training Macrocognitive Skills in Combat Medics Christen Sushereba, Laura Militello, & Emily Patterson

An Operational Account of Mental Models: A Pilot Study Joseph Borders, Gary Klein, & Ron Besuijen

Accelerating Team Decision-Making: Integrating a Slow, Deliberate, Off-Field Learning Environment with Rapid On-Field, In-Action Decision-Making, in Elite Netball Players Pam Richards, Dave Collins, & Mandy Robbins

Seeing Like an Expert: Tuning Recognition- Primed Heuristics John Flach, Markus Feufel, & Fred Voorhorst

A Complex Cognitive Skills Acquisition Framework

Kelly Neville, Dar-Wei Chen, Sean Normand, Lauren Massey, Tyree Cowell, Gregory Burbelo, Paul Blankenbeckler, & Elizabeth Uhl

3:00 BREAK

SESSION 93:30 Black Boxes & Bias Chair: Emily Newsome

A Naturalistic Study of How Analysts Make Sense of Black Box Algorithms in Analytic- Supported Information Analysis Alexander Morison, Karl Hendrickson, Stacie Hoelscher, & Daniel Zelik

Algorithmic Transparency for Predicting In-Hospital Mortality: A Targeted Literature Review CJ Hansen, Theodore Allen, Yuval Bitan, & Emily Patterson

Combating Bias with Recognition Primed Decision Making in Game Design Joel Kurucar, Robert Seater, Richard DeLaura, & Sean Winkler

4:15 CLOSING REMARKS4:30 CONFERENCE ADJOURN

12

ORGANIZING COMMITTEECO-CHAIRSGary KleinMacroCognition, LLC & ShadowBox, LLC

Laura MilitelloApplied Decision Science, LLC; ShadowBox, LLC; & Unveil, LLC

FULL COMMITTEESteve FioreUniversity of Central Florida

Julie GoreUniversity of Bath, UK

Gary L. KleinMITRE

Kathy MosierTeamScape LLC

Shane MuellerMichigan Technological University

Emily PattersonOhio State University

Hayley ReynoldsMIT Lincoln Laboratory

Emilie RothRoth Cognitive Engineering

Jan Maarten SchraagenTNO/University of Twente

Neville StantonUniversity of Southampton

Lygia StewartUniversity of California San Francisco

Joel SussWichita State University

13

KEYNOTE SPEAKERSBen ShneidermanDistinguished Professor in the Department of Computer Science and Founding Director of the Human-Computer Interaction Lab at the University of MarylandBen is a Distinguished University Professor in the Department of Computer Science, Founding Director of the Human-Computer Interaction Laboratory, and a Member of the UM Institute for Advanced Computer Studies (UMIACS) at the University of Maryland. He is a Fellow of the AAAS, ACM, IEEE, and NAI, and a Member of the National Academy of Engineering, in recognition of his pioneering contributions to human-computer interaction and information visualization. His widely-used contributions include the clickable highlighted web-links, high-precision touchscreen keyboards for mobile devices, and tagging for photos. Shneiderman’s information visualization innovations include dynamic query sliders for Spotfire, development of treemaps for viewing hierarchical data, novel network visualizations for NodeXL, and event sequence analysis for electronic health records.

Mica EndsleyPresident at SA Technologies and former Chief Scientist for the US Air ForceMica is President of SA Technologies and is the former Chief Scientist for the US Air Force. Dr. Endsley is a Fellow and Past-President of the Human Factors and Ergonomics Society. She is a world leader in the

design, development and evaluation of systems to support human situation awareness (SA) and decision-making, and the integration of humans and automation. She has authored over 200 scientific articles and is the co-author of Analysis and Measurement of Situation Awareness and Designing for Situation Awareness.

Wendy JephsonHead of Behavioral Science for Market Technology Risk and Surveillance Solutions at NASDAQWendy is Head of Behavioral Science for Market Technology Risk and Surveillance solutions at Nasdaq. She is responsible for leading a unique team of experts that combines behavioral science, financial domain knowledge, and advanced analytics to bring diverse thinking and cross-industry experience to solving some of the biggest challenges in financial services. Dual qualified as a commercial solicitor and Business Psychologist, in her role as Co-Founder, Wendy was instrumental in the original vision, growth phase and the recent acquisition of Sybenetix by the US exchange and technology giant, Nasdaq. Wendy’s behavioral science & legal expertise was developed in major corporations such as Eli Lilly and Xerox and through her MSc in Business Psychology. The British Psychological Society and leading psychologists have recognized Wendy’s expertise in enhancing decision-making skill, applying psychology to the management of risk and the understanding of how physiological factors affect investment behavior.

14

INVITED SPEAKERSJudith OrasanuJudith Orasanu has an been an integral part of the NDM community since its inception. During the 1980s, she served as the program manager for the Basic Research Office at the Army Research Institute where she was the director for the research program that was the precursor for the Naturalistic Decision Making movement. Her studies of aircrew decision making at NASA-Ames furthered our understanding of how teams make decisions in dynamic environments. She helped found the Cognitive Engineering and Decision Making technical group of the Human Factors and Ergonomics Society.

Daniel SerfatyAs Aptima’s Founder and CEO, Daniel Serfaty has led Aptima to become the premier Human-Centered Engineering business in the world. His work optimizes the integration of humans with intelligent technologies in defense, healthcare, aviation, and education. His keynote addresses around the world are encouraging his audiences to imagine a future in which human and artificial intelligences work together in the service of humankind. Daniel’s interdisciplinary background includes degrees in mathematics, psychology, and aerospace

engineering from the Université de Paris, the Technion, Israel Institute of Technology, and UConn. He is the recipient of the University of Connecticut Distinguished Service Award and has been inducted in its Engineering Hall of Fame. He remembers fondly the spirited fights between engineers and psychologists at the very first NDM Conference thirty years ago.

Lois JamesLois James is an assistant professor in the Washington State University (WSU) College of Nursing, where she focuses on bias, stress, sleep, and performance in “high stress” populations such as police officers, military personnel, nurses, and top tier athletes. She has received multiple honors and awards for her work, and is internationally recognized as a leading expert in her field. Dr. James’s work has been published in academic journals, practitioner magazines, and mainstream media such as the New York Times and the Washington Post. During her time at WSU, James has brought in approximately $5,000,000 of extramural funding.

15

Rhona FlinRhona Flin is Professor of Industrial Psychology at Aberdeen Business School, Robert Gordon University and Emeritus Professor of Applied Psychology at the University of Aberdeen. Her research examines human performance and safety in high risk industries and acute healthcare.

Paul FalzerPaul R. Falzer, Ph.D., is a Research Scientist at the VA Connecticut Healthcare System. His research focuses on clinical decision making, treatment interventions and outcomes, implementation science, and evidence-based medicine. Paul will be speaking about problems that have resulted from sweeping attempts to improve the delivery of health care, and how initiatives to optimize decision making have been integral to these efforts. Paul will discuss how naturalistic decision making concepts and approaches enable us to appreciate the current situation and address some of its intractable complications.

16

NOTES:

17

NOTES:

18

NOTES:

19

NOTES:

20

CONFERENCE ACCESSHOTELMarines’ Memorial Club & Hotel609 Sutter StreetSan Francisco, CA 94102(415) 673-6672

NDM 14 will be held at the Marines’ Memorial Club & HotelThe flagship Marines’ Memorial Club is located in the heart of downtown San Francisco, just off Union Square and the cable cars. This handsome California Spanish Revival hotel offers an inviting club atmosphere that is rich with history, honor and pride. Owned and operated by the Marines’ Memorial Association, Marines’ Memorial Club is the first ever “Living Memorial” in the United States, offering historic exhibits featuring American military history from all eras, and a library and museum with artifacts and books donated by grateful members, Veterans and their families.

Aquarium of the BayThere will be a bus available for interested attendees that leaves the hotel by 6:00pm and leaves the Aquarium to return to the hotel at 9:30pm.

Directions:Head west on Sutter Street toward Taylor StreetTurn right at the first cross street onto Taylor StreetTurn right onto Filbert StreetTurn left onto Powell StreetTurn right onto North Point StreetTurn left onto Grant AvenueTurn left onto Beach Street/The EmbarcaderoDestination will be on the right.

21

INFORMATIONREGISTRATION & BADGESThe NDM 14 Registration Desk is located on the 10th floor in the foyer. Onsite registration, badge, and conference materials are located there.

Registration hours:Monday: 8:00-5:00Tuesday: 8:00-5:30Wednesday: 8:00-3:00Thursday: 8:00-5:00Friday: 8:00-12:00

Please wear your badges for admission to all conference sessions. For any special requirements you may need, please refer to the registration desk at the earliest time as possible.

22

WIFIWireless internet is complementary throughout the hotel. Network ID: MMC_BallroomPassword (case sensitive): LIVINGMEMORIAL1946 Please see the registration desk for any questions.

HOTEL MAP

The conference will be held on the 10th and 11th floors of the hotel.23