DESIGNING AN INTERACTIVE TEACHING TOOL WITH ABML

KNOWLEDGE REFINEMENT LOOPenabling arguing to learn

1 Faculty of Education, University of Ljubljana, Slovenia

2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia

Matej Zapušek1, Martin Možina2, Ivan Bratko2, Jože Rugelj2, Matej Guid2

12th International Conference on Intelligent Tutoring SystemsITS 2014: Honolulu, Hawaii 2014

SOME CONCEPTS ARE DIFFICULT TO EXPLAIN...

How to distinguish edible from toxic mushrooms?

INTRODUCING DOMAIN EXPERTS

Even for domain experts it is hard to articulate their knowledge!

MACHINE LEARNING: HOW TO INVOLVE A DOMAIN EXPERT?

The expert can state constraints and the domain knowledge in advance...

… verify, evaluate, and correct results of machine learning…

… or the expert and the computer iteratively improve the model.

ABMLargument-based machine learning

ARGUMENT-BASED MACHINE LEARNING

given

set of labeled learning examples ei

described with attribute values Di

where Ci is classification of learning example ei goal

learn prediction model (hypoteshis) H

IF ... THEN ...IF ... THEN ......

HCiei: Di

ai

example ei may have argument ai

IT IS MUCH EASIER TO EXPLAIN INDIVIDUAL CASES!

Is this mushroom toxic? Why?

ABML KNOWLEDGE REFINEMENT LOOP

Step 1: Learn a hypothesis with ABML

Step 2: Find the “most critical” example (if none found, stop)

Step 3: Expert explains the example

Return to step 1

critical example

learn data set

ArgumentABML

ABML KNOWLEDGE REFINEMENT LOOP

Step 1: Learn a hypothesis with ABML

Step 2: Find the “most critical” example (if none found, stop)

Step 3: Expert explains the example

Return to step 1 Step 3a: Explaining a critical example (in a natural language)

Step 3b: Adding arguments to the example

Step 3c: Discovering counter examples

Step 3d: Improving arguments

Return to step 3c if counter example found

ILLUSTRATIVE EXAMPLE: LEARNING TO DIAGNOSE FLU

Pacient Temperature Vaccination Coughing Headache ... Flu

1 normal yes no no ... no

2 high no yes no ... yes

3 very high no no yes ... yes

4 high yes yes no ... no

... ... ... ... ... ... ...

The current model:

IF Temperature < very high THEN Flu = no

cannot explain well Pacient 2.

The question to the expert:

„What is the reason for Pacient 2 having the flu?“

Pacient Temperature Vaccination Coughing Headache ... Flu

1 normal yes no no ... no

2 high no yes no ... yes

3 very high no no yes ... yes

4 high yes yes no ... no

... ... ... ... ... ... ...

EXPERT‘S EXPLANATION

Expert‘s explanation:

„Pacient #2 has the flue because of a high temperature.“

Expert‘s argument

is attached to learning example #2. New model is built.

Flu = yes BECAUSE Temperature > Normal

Pacient Temperature Vaccination Coughing Headache ... Flu

1 normal yes no no ... no

2 high no yes no ... yes

3 very high no no yes ... yes

4 high yes yes no ... no

... ... ... ... ... ... ...

WHAT IF THE EXPERT‘S ARGUMENT IS NOT GOOD ENOUGH?

ML method now induced a rule consistent with argument:

IF Temperature > Normal THEN Flu = yes

The rule is inconsistent with data! Expert is presented with counter example:

„Compare pacients #2 and #4. Why Pacient #4 doesn‘t have the flu?“

Pacient Temperature Vaccination Coughing Headache ... Flu

1 normal yes no no ... no

2 high no yes no ... yes

3 very high no no yes ... yes

4 high yes yes no ... no

... ... ... ... ... ... ...

THE EXPERT MAY IMPROVE THE ARGUMENT

Expert finds the crucial difference between Pacients #2 and #4:

„Pacient 2 didn‘t get vaccinated against the flu.“

Pacient Temperature Vaccination Coughing Headache ... Flu

1 normal yes no no ... no

2 high no yes no ... yes

3 very high no no yes ... yes

4 high yes yes no ... no

... ... ... ... ... ... ...

IMPROVED RULES MAY EXPLAIN UNSEEN EXAMPLES AS WELL

ML method induces a new rule:

IF Temperature > Normal AND Vaccination = no

THEN Flue = yes

The new rule also explains diagnosis for Pacient #3:

„Has flu because of a high temperature and didn‘t get vaccinated against it.“

ABML KNOWLEDGE REFINEMENT LOOP: THE INNER LOOP

Step 3a: Explaining a critical example (in a natural language)

„Pacient #2 has the flue because of a high temperature.“

Step 3b: Adding arguments to the example

Step 3c: Discovering counter examples

Step 3d: Improving arguments with counter examples

IF TEMPERATURE > NORMAL AND VACCINATION = NO

Temperature > Normal

ABML REFINEMENT LOOP & KNOWLEDGE ELICITATION

IF ... THEN ...IF ... THEN ......

ABMLargument-based machine learning

explain single example easier for experts to articulate knowledge

“critical” examples expert provides only relevant knowledge

“counter” examples detect deficiencies in explanations

arguments

critical examplescounter examples

EXPERT CAN INTRODUCE NEW CONCEPTS (ATTRIBUTES)

Pacient ... Headache Fatigue SoreThroat Appetite Flu

1 ... no no yes normal no

2 ... no yes yes low yes

3 ... yes yes no low yes

4 ... no no no normal no

... ... ... ... ... ... ...

FluSymptoms

no

yes

yes

no

Possible rule with the new attribute:

IF Temperature > Normal AND FluSymptoms = yes

THEN Flu = yes

...

EXPERT CAN CORRECT CLASSIFICATION OF LEARNING EXAMPLE

... Headache Fatigue SoreThroat Appetite FluSymptoms Flu

... no no no normal no yes

Pacient Temperature Vaccination Coughing ...

37 normal no no ...

The question to the expert:

„What is the reason for Pacient 37 having the

flu?“

Expert corrects the classification of Pacient 37:

„Pacient 37 doesn‘t have the flu.“

no

KNOWLEDGE ELICITATION WITH ABML

IF ... THEN ...IF ... THEN ......

ABMLargument-based machine learning

inconsistencies in labels

are detected automatically

misclassificated examples are

easily recognized and corrected

arguments

critical examplescounter examples

experts introduce

new attributes

human-understandable models

suitable for teaching

IF ... THEN ...IF ... THEN ......

ABMLargument-based machine learning

arguments

critical examplescounter examples

How to use ABML in educational setting?

INTRODUCING STUDENTS

THE OUTER LOOP

Step 1: Learn a hypothesis with ABML

Step 2: Find the “most critical” example (if none found, stop)

Step 3: Student explains the example

Return to step 1

THE OUTER LOOP & THE INNER LOOP

Step 1: Learn a hypothesis with ABML

Step 2: Find the “most critical” example (if none found, stop)

Step 3: Student explains the example

Return to step 1 Step 3a: Explaining a critical example (in a natural language)

Step 3b: Adding arguments to the example

Step 3c: Discovering counter examples

Step 3d: Improving arguments with counter examples

Return to step 3c if counter example found

USING TEACHER‘S ATTRIBUTES!

ARGUING TO LEARN

Argumentation involves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999)

Participating in argumentation helps students learn about argumentative structures (Kuhn, 2001)

A NEW PARADIGM

arguing to learn

with

argument-based machine learning

Let‘s see how this works in practice

BASIC OR ADVANCED?

„basic“

„advanced“

LEARNING DATA SET

121 solutions of 62 different exercises

teacher labeled each solution

as „basic“ or „advanced“

learn data: 91 examples

test data: 30 examples

KNOWLEDGE ELICITATION FROM THE TEACHER

1. relevant description language: new attributes

2. consistently labeled examples

TEACHER‘S GOALS:

RESULTS OF KNOWLEDGE ELICITATION FROM THE TEACHER

9 iterations

9 new attributes

9 rules

only 1 out of 5 initial attributes remained

A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION

STUDENT‘S TASK

obtain rules for distinguishing „basic“ and „advanced“ solutions rules must consist of attributes in teacher‘s final model

use teacher‘s descriptive language

A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION

RECOMMENDATIONS TO THE STUDENT

use the most important features for explanations

use the smallest possible number of features in a single argument

try not to repeat the same arguments

THE FIRST „CRITICAL“ EXAMPLE

The question to the student:

„Why is this solution advanced?“

Student‘s argument:

„Because function zip is present and the number of rows is low.“

Solution = advanced BECAUSE Zip = True AND cRows = low

COUNTER EXAMPLE

IF Zip = True AND cRows = low THEN Solution = advanced

The rule is inconsistent with data! Student is presented with counter example:

„Compare these two solutions. Why is the second solution a basic one?“

IMPROVING ARGUMENT

IF Zip = True AND cRows = low AND LiCom = True

THEN Solution = advanced

Student‘s extended argument:

„Because function zip is present, the number of rows is low,

and a list comprehension occurs. “

no more counter examples next iteration

AT THE END OF THE INTERACTIVE SESSION

5 iterations

half an hour

90% accuracy on (previously unseen) testing data

several suggestions of new descriptive features

ASSESMENT

Results: experiment with 7 students

7.1 iterations

87.1% classification accuracy of obtained rule model

86.7% correctly „manually“ classified (previously unseen) examples

very positive qualitative feedback from the students

CONCLUSIONS

New paradigm:

arguing to learn with argument-based machine learning

Future work:

• applications in several domains

• assessing argument‘s quality for improved immediate feedback

• goal-oriented extension (see our ITS 2012 paper)

QUESTIONS & DISCUSSION

Thank you!

slides: ailab.si/matej

enabling arguing to learn