thinking hard together: the long and short of collaborative idea generation in scientific inquiry

Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University

Thinking Hard Together: Thinking Hard Together: the Long and Short of the Long and Short of

Collaborative Idea Generation in Collaborative Idea Generation in Scientific InquiryScientific Inquiry

Hao-Chuan Wang, Carolyn P. Rosé, Yue CuiCarnegie Mellon University

Chun-Yen ChangNational Taiwan Normal UniversityChun-Chieh Huang, Tsai-Yen Li

National Chengchi University

Funded through the National Science Foundation, The Pittsburgh Science of Funded through the National Science Foundation, The Pittsburgh Science of Learning Center, and the Office of Naval ResearchLearning Center, and the Office of Naval Research


Why Collaborative Idea Generation?Why Collaborative Idea Generation?

Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266

• Supported by common CSCL environments

• Idea generation (problem finding) is recognized as preparation for problem solving (e.g., Hmelo-Silver, 2004)

• Its value as a learning task is not well established


Problem:Problem: Process Losses in Group Process Losses in Group Idea GenerationIdea Generation

• Well known problem (Connelly, 1993; Diehl & Stroebe, 1987; Kraut, 2003)

• Non-interacting groups may produce more and better ideas (Hill, 1982; Diehl & Stroebe, 1987)*How many people does it How many people does it

take to screw in a light bulb?take to screw in a light bulb?


Research QuestionsResearch Questions

• Can we design support for collaborative idea generation that mitigates process losses?

• Are there long term learning benefits of collaborative idea generation that outweigh the short term productivity losses?


OutlineOutline• Vision: Using language technologies to

offer context sensitive support for collaborative idea generation

• Experimental Study (2X2 factorial design)– Pairs versus Individuals– With versus without support from an

automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions


OutlineOutline• Vision: Using language technologies

to offer context sensitive support for collaborative idea generation




The Debris-FlowThe Debris-FlowHazard Task Hazard Task (Chang & Tsai, 2005)(Chang & Tsai, 2005)

Task1: “What are the possible factors that might cause a debris-flow hazard to happen?”

Task 2: “How could we prevent it from happening?”

Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266


The Constructive Process of The Constructive Process of Idea Generation Idea Generation (Brown & Paulus, 2002)(Brown & Paulus, 2002)

• Idea Generation Prompt– What are the possible factors that might cause a debris-flow hazard to

happen?

• Domain Knowledge– Debris flow refers to the mass movement of rocks and sedimentary

materials in a fluid like manner– There are many typhoons (i.e. hurricanes) in Taiwan during the

summer• Bridging Inferences

– Heavy rain implies the presence of massive amounts of water– The presence of massive amounts of precipitation is likely to cause the

fluid movement of rocks • Idea

– Typhoons could be a factor causing a DFH to happen

IdeasDomainKnowledge

BridgingInferences


http://www.cs.cmu.edu/~cprose/TagHelper.html

Automatic Support for Automatic Support for Collaborative Idea GenerationCollaborative Idea Generation

• Trigger dialogue agents with an automatic analysis of a collaborative learning interaction

TagHelper

Labeled Texts

Unlabeled Texts

Labeled Texts

A Model that can Label More Texts

Time

Idea

Lab

els

Student 1 People stole sand and stones to use for construction.

Agent Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?

Student 2 Development of mountain areas often causes problems.

Student 1 It is okay to develop, but there must be some constraints.


2 Part Feedback Generation2 Part Feedback GenerationStudent 1People stole sand and stones to use for construction.

Agent Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable.

Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?

Comment: acknowledges the idea the student just contributed

Tutorial: points the student in a direction for moving on with brainstorming

Designed based on “category labels” shown to be effective in previous studies of idea generation (Dugosh et al., 2000; Nijstad & Stroebe, 2006)




• Experimental Study (2X2 factorial design)

– Pairs versus Individuals– With versus without support from an



MethodMethod• Experimental Design: 2X2 Factorial design

– Working in Pairs vs Working Individually– Feedback from VIBRANT vs No Feedback

• Participants: 42 10th grade students from a central Taiwan high school were randomly assigned to 4 conditions

– Individual+NoFeedback (7 students)– Individual+Feedback (7 students)– Pair+NoFeedback (7 pairs)– Pair+Feedback (7 pairs)


Collaborative Idea Generation Support Collaborative Idea Generation Support Using Dialogue AgentsUsing Dialogue Agents

Conferencing Mode:Student 1, Student 2 &Dialogue Agent

Task Description

Student 1’sContribution

Student 2’sContribution

Agent’sFeedback


Experimental ProcedureExperimental Procedure

1. Background reading (10 min)2. Pretest (15 minutes)3. Brainstorming 1 (30 minutes)

• Experimental manipulation takes place during this phase

4. Brainstorming 2 (10 minutes)• Serves as a “transfer task”

5. Post test (15 minutes)


What counts as a good idea?What counts as a good idea?• The Debris-Flow Hazard task was designed by a

panel of science educators as an assessment of creative problem solving ability

• It has been used in a series of classroom studies in Taiwan

• The student responses from these assessment studies have been evaluated by the panel

• Based on this data, the panel decided on a set of “valuable ideas” for each task, which are used to measure idea generation success


Analysis of Verbal Data: First Analysis of Verbal Data: First Brainstorming TaskBrainstorming Task

• Conversation logs segmented into idea units– Typically at contribution boundaries– Contributions containing more than one idea broken up– Agreement of 2 coders on 10% of data was satisfactory

(Kappa .7)• Idea units coded for one of 19 specific idea

labels, other ideas related to correct solutions, and other ideas

– Agreement of 2 coders on 10% of data was good (Kappa .82)


Analysis of Verbal Data (Second Analysis of Verbal Data (Second Brainstorming Task)Brainstorming Task)

• Idea units from second brainstorming task already segmented

– Coded with 15 pre-defined “valuable” ideas defined by experts or other

– Agreement of 2 coders on 10% of the data was acceptable (Kappa .74)


Expected Productivity loss in pairsExpected Productivity loss in pairs

• D.V.: Number of unique ideas• Results of ANOVA:

- Nominal Dyad > Real Dyad(p<.001, d=1.51)

• Productivity loss is evidenced

RealDyad

Nominal Real

Conditions

Num

. Uni

que

Idea

(Bra

inst

orm

ing

1)

02

46

810

1214

NominalDyad


Students in Pairs Learned Less Students in Pairs Learned Less

IndividualsPairs

• D.V.: Post test score• Covariate: Pre test score• Results of ANCOVA:

Individuals > Pairs(p<.01, Cohen’s d=.1.68)*

• Effect mediated by process loss

– R2=.48, p< .005, N=42

*Based on conversion formula by Cohen(1988): d = 2 * f for k=2

Ind Pair

Conditions

Adj

uste

d P

ostte

st S

core

02

46

810


SystemSystem Feedback as Learning Support during Feedback as Learning Support during BrainstormingBrainstorming

• D.V.: Post test score• Covariate: Pre-test

score• Evaluating the influence

of system effect on domain learning (pre to post learning increases)

• Result: System Feedback > No Feedback (p<.05, d=.70)

NoF F

Conditions

Adj

uste

d P

oste

st S

core

02

46

810

No Feedback

System Feedback


Effect of Working in Pairs on Task 2 Effect of Working in Pairs on Task 2 SuccessSuccess

Individuals

Pairs • D.V.: Number of unique ideas in the subsequent idea generation task

– Note: All students worked alone on this task

• Results: Pairs > Individuals(p<.01, d=.92)*

– No main effect of System Feedback

Ind Pair

Conditions

Num

. Uni

que

Idea

s (B

rain

stor

min

g 2)

01

23

45

67


• Sig. Interaction Effect(p<.05, d=.78)*

– Best: Pairs with Feedback

– Worst: Individuals with Feedback

Ind+NoF Ind+F Pair+NoF Pair+F

Condition

Num

. Uni

que

Idea

s (B

rain

stor

min

g 2)

01

23

45

67

IndividualsPairs

Effect of Working in Pairs on Task 2 Effect of Working in Pairs on Task 2 SuccessSuccess


• Main effect on Pair/Ind– F(1,38)=4.19, p<.05, d=.6– Pair > Individual

• No interaction

• Evidence of mediating success on Task 2– Significant correlation with

success on task 2– R2=.49, p < .0001, N=42

Ind Pair

Condition

P(ta

sk2

| rel

ated

task

1)

0.0

0.1

0.2

0.3

0.4

0.5

Connection between Task 1 and Connection between Task 1 and Task 2 in Pairs ConditionsTask 2 in Pairs Conditions


““Off-task Ideas”Off-task Ideas”• Solutions were mentioned

more frequently in the pairs conditions and in the feedback conditions

– Note: Only marginal• Mention of solutions

during the first task correlated with success during the second task

– R2=.13, p< .05, N=42

NoF FNo-Feedback/Feedback

Mea

n (v

alua

ble

solu

tions

)0.

00.

51.

01.

52.

02.

5

NoF FNo-Feedback/Feedback

Mea

n (v

alua

ble

solu

tions

)0.

00.

51.

01.

52.

02.

5

Nominal Real

Type of Dyad

Mea

n (v

alua

ble

solu

tions

)0.

00.

51.

01.

52.

02.

5M

ean

(val

uabl

e so

lutio

ns)

0.0

0.5

1.0

1.5

2.0

2.5


Connection between Task 1 and Connection between Task 1 and Task 2 in Pairs ConditionsTask 2 in Pairs Conditions

Speaker Text

Student 1 People stole sand and stones to use for construction.

VIBRANT

Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?

Student 2 Development of mountain areas often causes problems.

Student 1 It is okay to develop, but there must be some constraints.

Dialogue from Task1Dialogue from Task1


Pair+NoF Pair+F

Condition

LSA

-bas

ed S

imila

rity

0.0

0.1

0.2

0.3

0.4

0.5

• t(38)=2.57, p<.05, d=.76• Individuals in the Pair+F

condition tended to stay in the same topic more

• t(24)=1.994, p<.05, d= .77• Pair+F > Pair+NoF• Peers in the Pair+F condition

‘talked’ more similarly

Greater Cohesiveness in Pairs Greater Cohesiveness in Pairs with Feedbackwith Feedback

Pair+NoF Pair+F

Condition

Pro

potio

n of

Sta

ying

in th

e S

ame

0.0

0.1

0.2

0.3

0.4

0.5


Pairs versus IndividualsPairs versus Individuals• Pairs approached the problem from a broader

perspective– Better preparation for problem solving

• Feedback increased the intensity of Task 1 performance

– Trend of effect on Task 2 consistent with the overall positive effect of working in pairs

– Negative effect on Task 2 only in the Individual condition where focus was narrowly on Task 1

• But how can we balance success at learning/Task 1 with success at Task 2?


Process AnalysisProcess Analysis• Conjecture: Process losses occurred at early

stage of group idea generation (Diehl&Stroebe, 1991)– Never tested directly– Folk wisdom: brainstorm alone before group

brainstorming– Note: Opposite has been proven effectively

previously (Brown & Paulus, 2002)• Finding: about half of unique ideas contributed

in first 5 minutes• Cognitive interference strongest during that time• Feedback will show an effect after 5 minutes


Process AnalysisProcess Analysis

Process loss Pairs vs Individuals:F(1,24)=12.22, p<.005, 1 sigma

Process loss Pairs vs Individuals: F(1,24)=4.61, p<.05, .61 sigma

Negative effect of Feedback:F(1,24)= 7.23, p<.05, -1.03 sigma

Positive effect of feedback:F(1,24)=16.43, p<.0005, 1.37 sigma

0 5 10 15 20 25 30

02

46

810

12

Time Stamp

#Uni

que

Idea

s

Unique Ideas

Nom+NNom+FReal+NReal+F Pairs+Feedback

Individuals+NoFeedback

Pairs+NoFeedback

Individuals+Feedback

Pairs+Feedback

Individuals+NoFeedback

Pairs+NoFeedback

Individuals+Feedback


What did we learn?What did we learn?• Good News

– Interaction with dialogue agents during brainstorming increases learning

– Feedback increases idea generation in pairs and individuals after 5 minutes

• Bad News– Process losses in group brainstorming may

hinder learning– Feedback decreases idea generation in the

first five minutes


Current DirectionsCurrent Directions• Follow-up study

– Students brainstorm alone for five minutes without feedback

– Then work together for 25 minutes with feedback• Continuing to investigate automatic forms of

collaborative learning support– Thermodynamics (Kumar et al., 2007)– Middle school math (Kumar et al, to appear)

• Continued work on automatic collaborative learning process analysis (Rosé et al, Under Review)


Any Questions?Any Questions?

Htpp://www.cs.cmu.edu/~cprose/TagHelper.htmlHtpp://www.cs.cmu.edu/~cprose/TagHelper.html

thinking hard together: the long and short of collaborative idea generation in scientific inquiry

Documents