a ssessment of communication performance in simulated volcanic crises jackie dohaney, e. brogt, t.m....

ASSESSMENT OF COMMUNICATION PERFORMANCE IN SIMULATED VOLCANIC CRISESJackie Dohaney, E. Brogt, T.M. Wilson, B. Kennedy

Postdoctoral Fellow, Geoscience Education

jdohaney@gmail.com

Why teach and assess Communication Skills?

1. We typically don’t teach communication skills

2. These are often adhoc, embedded, implicit (hidden) learning outcomes

3. Effective communication is central to Disaster Risk Reduction principles

Project Outcome:

Build and assess curricula that improves students’ communication skills.

Research Objective:

Test & create evidence-based measures of communication.

1) ‘Unpack’ Communication Performance

2) Variables/Proxies of Communication

3) Qualitative assessment of communication (via student pre-post interviews)

4) Compare proxies to interviews.

ASSESSING COMMUNICATION

1) ‘Unpack’ Communication Performance

How can we measure communication?

Effectiveness of communication is highly contextual, and receiver-dependent.

Anecdotal, holistic, inherited and conflicting views on teaching communication.

-> Little educational research is dedicated to the rigorous assessment of communication.

ASSESSING COMMUNICATION

Different perspectives

1) ‘Unpack’ Communication Performance-> Definition of “Best Practices” in Crisis Communication

7 C’s of Communication

Comprehensible – simple, clear, jargon-free

Contextualized – diversity, cultures, differences

Captivating – engaging, relevant

Credible – open, frank, acknowledges uncertainty

Consistent – backed by evidence, confirmable

Courteous – compassionate, empathetic, respectful

(Addresses ) Concerns – empowers action/response

Vivienne Bryner (University of Otago, NZ)

http://www.youtube.com/watch?v=grhrLT8tfjg

ASSESSING COMMUNICATION

Curricula: Role-plays & Simulation

ASSESSING COMMUNICATION

To acquire and perfect communication skills, research suggests students need to practice them in authentic scenarios. (Cox et al. 2012).

Role-playing, modelling, coaching, rehearsal, reinforcement and feedback, enable individuals to improve their communication competence. (Richmond et al. 2013).

2) Variables that contribute to communication?

i.e., confidence

ASSESSING COMMUNICATION

Student Interviews

Actual performance

Variable: Communication ExperienceMeasure: Communication Experiences Survey

(Validation Process; Dohaney et al 2015)

Experiences:Oral presentations

Poster presentations

Debates & Speeches

In the classroom (Teaching)

Group Discussions & Meetings

Professional Media

Setting/style:- At conferences, in your department, during your education- Provided, received or self-evaluative feedback for

communication?

ASSESSING COMMUNICATION

Estimated number of experiences

NoneFew (1-10)

Some (11-20)Many (21-30)

A lot (>30)

PLEASE COMPLETE THE SURVEY

Variable: Communication EfficacyMeasure: Self-perceived communication competence

SPCC Instrument (McCroskey and McCroskey 1988)

Statements:

1. Present a talk to a group of strangers.

2. Talk with an acquaintance.

3. Talk/Discuss at a large meeting of friends.

4. Talk in a small group of strangers.

5. Talk with a close friend.

6. Talk/Discuss at a large meeting of acquaintances.

7. Talk with a stranger.

8. Present a talk to a group of friends.

9. Talk in a small group of acquaintances.

10. Talk/Discuss at a large meeting of strangers.

11. Talk in a small group of friends.

12. Present a talk to a group of acquaintances.

ASSESSING COMMUNICATION

Likert Choices:Very strong ability

Strong abilityAverage ability

Poor abilityVery poor ability

Can look at gender, age, nationality, curricula-type, etc to assess any factors that might influence pre- scores and “changes” (post- pre).

Perceptions of Crisis Communication:

Content Language & Terms Amount of content Level of detail Equity & Diversity

Delivery Visual Aids Media-type Tone Intent of message/Purpose

Different Audiences Scientists, Emergency Managers The public and communities

ASSESSING COMMUNICATION

Variable: Science Communication Best Practices

Measure: Perceptions of Science Communication Best Practices (Development process; Dohaney et al 2015/16)

ASSESSING COMMUNICATION

Likert Choices:Stongly Agree

AgreeNeutral

DisagreeStrongly Disagree

Examples of Statements:1. In a crisis, scientific information presented to the PUBLIC should appear open and completely transparent.

2. In a crisis, scientific information presented to the PUBLIC should be comprehensive (including all scientific facts discussed).

3. In a crisis, when communicating to the PUBLIC, it is appropriate to show your emotions

ASSESSING COMMUNICATION

Perceptions: Better Communicators

ASSESSING COMMUNICATION

Perceptions: Scientific Responsibilities

3) Qualitative assessment of communication (via student pre-post interviews) -> Direct Measure of Communication Performance

Instrument- Presentation Skills Protocol for Scientists (2PS)

Rubric to guide coding

of pre- and

post-interviews

ASSESSING COMMUNICATION

Pre-Interview Script:

Question: How do scientists actually know that a volcano is going to erupt?

“ That goes back to the monitoring that we are always doing.

Our scientists know when something is happening, that is different from what we usually see. That’s when we will investigate further.

And ya know, break it down into what that actually means.”

ASSESSING COMMUNICATION

Post-interview script:

Question: How do scientists actually know that a volcano is going to erupt?

“So we monitor the volcanoes constantly. And we have different methods of doing this. Scientists know what to look out for.

We monitor it so frequently that, anything out of character, of the background level, so that we know something is a different kind of activity.

So we will look into that further. And from there we can figure out if that is a normal process, or an increase in activity”

ASSESSING COMMUNICATION

Future Work: (4) Compare proxies to interviews.

(5) Incorporating the qualitative coding into a robust rubric for assessment of communication performances that is appropriate to crisis contexts

Excellent Average Poor

ASSESSING COMMUNICATION

Thank You! Any Questions?

Jackie Dohaney

Postdoctoral Fellow, Geoscience Education

jdohaney@gmail.com

Perceptions (please see me!)

Communication Experiences Survey

http://canterbury.qualtrics.com/SE/?SID=SV_9Xrc0QfnzZnyN6d

SPCC Study ParticipantsStudent participants (n=37) were recruited from third and fourth year

physical volcanology and hazards management courses which hosted the volcanic hazards simulation as part of their curricula.

Two iterations of the simulation were tested; One was embedded at the end of a 7-day field course (took place in January, 2012; n=19) the other was embedded within a lecture-based course (took place in August 2012; n=18).

They were mixed cohorts of American study-abroad students and New Zealand students who attended the University of Canterbury. Students ranged in gender (female (13) and male (24)), nationality (New Zealand (24), United States of America (11), Netherlands (1) and India (1)), and age (aged 19 – 22 (20) and - 23 years old (17)).