Large-scale Discourse Analysis of Counseling Conversations

Kevin Clark, Tim Althoff, Jure Leskovec

Mental Health by the Numbers

• 43.8 million adults (18.5%) in the U.S. experience mental illness each year

2

Statistics provided by the National Institute of Mental Health: http://www.nimh.nih.gov/

• Suicide is leading cause of death for people aged 15-24

• Nearly 50% of youths (aged 12-18) with mental illness didn’t receive treatment in the previous year

Counseling

• Treatments like psychotherapy and counselingcan help!• Lots of great research on how to counsel

effectively • Typically small scale and qualitative

• Technology-mediated counseling has greatly broadened access to counseling resources• Also allows for large scale quantitative studies

3

This Talk

• What makes a good counselor?• How do you help someone feel better?

• Various techniques from NLP to discover effective conversation strategies

• Largest quantitative study of crisis counseling to date

4

The Data

5

• Collaboration with a nonprofit supporting teenagers in crisis through text messaging

• Texters matched with extensively trained volunteer counselors

• Counseling conversation via SMS• Texter gets follow-up survey

• So data includes conversation outcomes

Dataset Statistics

6

• 80,855 conversations, 3.2 million messages• 15,555 (19.2%) of conversations have survey responses• 408 counselors

• 130 counselors with over 15 conversations with survey responses

6

Split by success rate

Less successful (~50% SR) More successful (~75% SR)

Counseling “Strategies”

1. Adapt to the conversation2. Be creative in responses3. Work towards making progress4. Facilitate perspective change

7

Adaptability

• Are counselors aware of how conversations are going? How do they react?

• Compute distance between counselor language in positive/negative conversation• Represent language with TF-IDF vector of word occurrences• Cosine similarity for distance

• Observe how this changes over time

8

Adaptability

9

Time

Difference in languagebetween positive and negative conversations

Not adapting

Adapting

1. Adapt to the conversation

10

0-20 20-40 40-60 60-80 80-1003ortLon of conversDtLon (% of PessDges)

0.012

0.013

0.014

0.015

0.016

0.017

0.018

0.019

DLs

tDnc

e be

twee

n po

sLtLv

e D

nd n

egDt

Lve

conv

ersD

tLons 0ore successful counselors

Less successful counselors

Not adapting

Adapting

Creativity & Generic Responses

11

• Do counselors use generic or “templated” responses?• “How does that make you feel?” vs.

“Thanks for sharing that with me. That sounds really challenging. How do you feel about X, Kevin?”

• Measure “creativity”• Compute the number of close neighbors to each response• Cosine distance in TF-IDF space is below a threshold

Message with few neighborsMessage with many neighbors

2. Be creative in responses

12

0ore successful Less successfulCounselor qualLty

8

10

12

14

16

18

20

22

24# 6LPLlar counselor reactLons ConversatLon qualLty

PosLtLve1egatLve

Finding: More successful counselors use more creative responses than less successful counselors

Creative

Generic

How do more and less successful counselors talk differently?

• More successful counselors …• writing longer messages• use more check questions• “it sounds like…”

• use more hedges (lessen the impact of an utterance)• “maybe”, “fairly”

• avoid “why” questions

• Many more examples in the paper

13

Conversation Progress

14

• Is there a general higher-level structure to counseling conversations?

• How do counselors navigate this structure?

• Use techniques from unsupervised conversation modeling to learn ordered sequence of conversation stages

Conversation Model

15

• Assign each message in each conversation a stage using a variant of Hidden Markov Models • Force stages to be in increasing order

Conversation as sequence of text messages m1

m2

m3

m4

m5

m6

m7

Model assigns a stage to each messagem1 stage 1m2 stage 2m3 stage 2m4 stage 3m5 stage 4m6 stage 4m7 stage 5

Conversation Stages

16

Stage Interpretation Texter top words Counselor top words1 Introduction hi, hello, name, listen, hey hi, name, hello, hey, brings

2 Problem introduction

dating, moved, date, liked, ended

gosh, terrible, hurtful, painful, ago

3 Problem exploration

knows, worry, burden, teacher, group

react, cares, considered, supportive, wants

4 Problem solving write, writing, music, reading, play

hobbies, writing, activities, distract, music

5 Wrap up goodnight, bye, thank, thanks, appreciate

goodnight, 247, anytime

Conversation Stages

17

Stage Interpretation Texter top words Counselor top words1 Introduction hi, hello, name, listen, hey hi, name, hello, hey, brings

2 Problem introduction

dating, moved, date, liked, ended

gosh, terrible, hurtful, painful, ago

3 Problem exploration

knows, worry, burden, teacher, group

react, cares, considered, supportive, wants

4 Problem solving write, writing, music, reading, play

hobbies, writing, activities, distract, music

5 Wrap up goodnight, bye, thank, thanks, appreciate

goodnight, 247, anytime

1 2 3 4 5Stage

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Sta

ge d

urat

Lon

(Ser

cent

of c

onve

rsat

Lon)

0ore successful counselorsLess successful counselors

3. Work towards making progress

18

1 2 3 4 5Stage

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Sta

ge d

urat

Lon

(Ser

cent

of c

onve

rsat

Lon)

0ore successful counselorsLess successful counselors

3. Work towards making progress

19

• More successful counselors are quicker to know the problem and spend longer on the problem solving stage

Perspective Change

• Prior research on depression finds• Focusing on others instead of yourself can help (Campbell and

Pennebaker, 2003)• Having a positive view of the future can help (Pyszczynski et al.,1987)

• We quantify perspective change by tracking the frequency of LIWC markers (Tausczik and Pennebaker, 2010)• “I, me, myself, …” vs “he, she, they, …”• Past vs Present vs Future

20

Perspective Change: Self-Focus

21

• Texters who talk less about themselves and more about others tend to have successful conversations

0-20 20-40 40-60 60-80 80-100Portion of conversation (% of Pessages)

0.70

0.75

0.80

0.85

6el

f rel

ativ

e fre

quen

cyPositive conversations1egative conversations

Perspective Change: Future

22

0-20 20-40 40-60 60-80 80-1003ortion of Fonversation (% of Pessages)

0.060.070.080.090.100.110.120.130.140.15

)utu

re re

lativ

e fre

quen

Fy 3ositive Fonversations1egative Fonversations

0-20 20-40 40-60 60-80 80-100Portion of conversation (% of Pessages)

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

0.76

Pre

sent

rela

tive

frequ

ency Positive conversations

1egative conversations

0-20 20-40 40-60 60-80 80-1003ortion of conversation (% of Pessages)

0.120.140.160.180.200.220.240.260.280.30

3as

t rel

ativ

e fre

quen

cy

3ositive conversations1egative conversations

• Texters who talk less about the present and more about the future tend to have successful conversations

Past Present Future

• Simple hypothesis: The texter will talk more about something (e.g., the future) if the counselor talks about it first

• Linguistic coordination• Use coordination measure from

(Danescu-Niculescu-Mizil, 2012)

• We find significant coordination of texter towards counselor for all perspective change markers (e.g. future)• Counselor can help facilitate perspective change

23

4. Facilitate perspective change

Conclusion

24

Applying NLP techniques can teach us a lot about counseling1. Adapt to the conversation

2. Be creative in responses

3. Work towards making progress

4. Facilitate perspective change

Full study in paper: Large-Scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

Since This Research…• Project no longer active at Stanford (no data access), but…

• The counseling organization has a research fellowship program• Looking for AI/ML/NLP Experts, 3-6 months working on site

• Lots of new research on applying NLP to mental health• Especially on identifying/risk-asessing mental illness, depressions, etc.• CLPsych: Computational Linguistics and Clinical Psychology Workshop

• Also a growing number of startups in this space

25

26

Thanks!Any Questions?

New Research on the data

27

• Outside event causing increased volume

New Research on the data

28