case-based reasoning – through 3 applications copyright © 2006 reich prof. yoram reich faculty of...

87
Case-Based Reasoning – through 3 Case-Based Reasoning – through 3 applications applications Copyright Copyright © © 2006 Reich 2006 Reich Prof. Yoram Reich Prof. Yoram Reich Faculty of Engineering Faculty of Engineering Tel Aviv University Tel Aviv University Israel Israel

Post on 20-Dec-2015

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Case-Based Reasoning – through 3 Case-Based Reasoning – through 3 applications applications

Copyright Copyright ©© 2006 Reich 2006 Reich

Prof. Yoram ReichProf. Yoram Reich

Faculty of EngineeringFaculty of EngineeringTel Aviv UniversityTel Aviv University

IsraelIsrael

Page 2: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 2

Outline

Generic learning tasks The basic concept Past work

eCobweb, eProtos Present work

Conversational CBR (Gil Chen) CBR with weak causal knowledge (Adi Kapeliuk)

KDML – Knowledge Discovery Modeling Language

Page 3: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 3

Design of Cable-Stayed Bridges

Other examples of cable-stayed bridges

Page 4: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 4

Cable-Stayed Bridge Design: Specification Properties

8 specification properties

Page 5: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 5

Cable-Stayed Bridge Design: Product Description Properties

30 product description properties

Page 6: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 6

Cable-Stayed Bridge Design: Derived Properties

Aesthetics

Page 7: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 7

Cable-Stayed Bridge Design: Analysis Properties

Analysis according to American Bridge Design Code 15 analysis properties

Page 8: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 8

GLTs – Supporting Design Activities

Concept formation (unsupervised) – Synthesis

Concept learning (supervised) – Analysis, Redesign, Evaluation

Page 9: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 9

Mapping Design Tasks to GLTs and ML Programs

Page 10: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 10

Mapping problems into generic learning tasks

Y. Reich, Macro and micro perspectives of multistrategy learning, in Machine Learning: A Multistrategy Approach, Vol. IV (R. S. Michalski and G. Tecuci, eds.), (San Francisco, CA), pp. 379–401, Morgan Kaufmann, 1994.

Page 11: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 13

Bridger’s architecture

Page 12: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 14

eCobweb – Classification Hierarchy

Page 13: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 15

Example of COBWEB operators from 2nd application

Page 14: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 16

Case Structure & Representation

Case

ID Name DescriptionDate Version

Keyword1

Keywords

Keyword2...

KeywordN

Environment1

Environments

Environment2...

Environment4

Topic1

Topics

Topic2...

TopicM

Q/A1

Q/A

Q/A2...

Q/A J

Action1

Solution

Action2...

ActionK

Automaticallyadded by thesystem

Page 15: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 17

Incremental Conceptual Clustering – COBWEB

Operators

Put in Existing Class

Create a New Class

Merge

Split

C0

C1 C2 C3

C4 C5

p11

p1

p2

p3 p4 p5

p6 p7 p8 p9 p10

Page 16: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 18

Page 17: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 19

Page 18: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 20

Page 19: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 21

Page 20: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 22

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

Initial State:

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

Constrain sketch entities (1)Imported dimension's style (1)Import ACAD border (4)Import prof iles (5)OLE objects and bitmaps (2)

Change units of dif ferentformats (7)Wrong units and settings (9)

Missing 3D curve entitiesb (6)Missing solids (5)Missing sketch entities (7)

import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)

import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)

Drawing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16)Assembly (16)

Data Exchange (16)Settings/Defaults (16)

Part (18)Assembly (18)

Data Exchange (18)

Page 21: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 23

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)

import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)

Drawing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16)Assembly (16)

Data Exchange (16)Settings/Defaults (16)

Part (18)Assembly (18)

Data Exchange (18)

2.85Category

Utility =

Page 22: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 24

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)

import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)

Drawing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16)Assembly (16)

Data Exchange (16)Settings/Defaults (16)

Part (18)Assembly (18)

Data Exchange (18)

Category

Utility = 3.32

Page 23: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 25

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)

import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)

Drawing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16)Assembly (16)

Data Exchange (16)Settings/Defaults (16)

Part (18)Assembly (18)

Data Exchange (18)

Category

Utility = 3.13

Page 24: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 26

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

import (13) profile(3)dxf (10) part (3)dwg (10) dimension(3)sketch (6) wireframe (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16) VDA (3)units (16) ACIS (3)control (12) scale (2)change (10) VRML (2)incorrect (7) STL (2)IGES (3) STEP (5)

import (18) VRML (2)missing (17) image (2)curve (11) VDA (4)sketch (10) ACIS (4) solid (5) STEP (2)geometry (4) IGES (4)complete (3)

Drawing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16)Assembly (16)

Data Exchange (16)Settings/Defaults (16)

Part (18)Assembly (18)

Data Exchange (18)

3.05

New Category )1 case(

Import (1)STEP (1)Change (1)Units (1)Control (1)

Part (1)Assembly (1)

Data Exchange (1)Settings/Defaults(1)

Category

Utility =

Page 25: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 27

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

2.94Category

Utility =

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS" + "MISSING DATA"(34 cases)

import (13) prof ile(3)dxf (10) part (3)dw g (10) dimension(3)sketch (6) w iref rame (2)sheet (4) curves (2)border (4) object (2)template (4) bitmap (2)titleblock (4) constrain (2)

import (16 + 18 = 34) VDA (3 + 4 = 7) units (16)ACIS (3 + 4 = 7) control (12) scale (2)change (10) VRML (2 + 2 = 4) incorrect (7)STL (2) IGES (3 + 4 = 7) STEP (5 + 2 = 7)image (2) missing (17) curve (11)sketch (10) solid (5) geometry (4)complete (3)

Draw ing (11)Part (7)Assembly (2)

Data Exchange (13)2D Sketch (2)

Part (16 + 18 = 34)Assembly (16 + 18 = 34)

Data Exchange (16 + 18 = 34)Settings/Defaults (16)

"IMPORT"

"DXF/DWG"(13 cases)

"UNITS"(16 cases)

"MISSING DATA"(18 cases)

Constrain sketch entities (1)Imported dimension's sty le (1)Import ACAD border (4)Import prof iles (5)OLE objects and bitmaps (2)

Change units of dif ferentformats (7)Wrong units and settings (9)

Missing 3D curve entitiesb (6)Missing solids (5)Missing sketch entities (7)

Merge 2 highest

scores 2.85 3.32 3.13

Page 26: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 28

Incremental Conceptual Clustering – COBWEB

“Good”/”Bad” Category Utility Samples

Operators CU values

DXF/DW G Units New Merger2.6

2.8

3

3.2

3.4

Missing Data

CU

Page 27: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 29

eCobweb – CU – Category Utility

Y. Reich, “Constructive induction by incremental concept formation,” in Artificial Intelligence and Computer Vision (Y. A. Feldman and A. Bruckstein, eds.), pp. 191–204, Amsterdam: Elsevier Science Publishers, 1991.

Page 28: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 30

eCobweb characteristics

Property-value pairs that describe each category:

The probability to get an attribute value given that a case

belongs to a class

The probability to be in a class given that a case has a

particular attribute value

thresholdCVAP kiji |

thresholdP VAC ijik |

Page 29: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 31

eCobweb – Growth of Classification Hierarchy

Page 30: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 32

eCobweb – Prediction Methods

Page 31: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 35

CMLM – Contextualized ML Modeling

Page 32: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 38

Examples of Recent Systems

Conversational CBR – Help Desk (CRM) system Complex problem Requires multiple GLTs Evolution of methods Future enhancements

TeSAS (Technical Support Aiding System): CBR with weak knowledge – lesson learned system Complex problem Requires multiple GLTs Evolution of methods Future enhancements

Page 33: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 39

Cased-Based Reasoning

Case DataBase

NearestCases

FirstSolution

ImprovedSolution

NewCase

NewSolvedCase

2. Retrieve

5. Retain4. Revise3. Reuse

6. Store

7. MaintainNewCase

NewCase

NewCase

NewCase

NewCase

NewCase

1. Build

Page 34: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 40

Help Desk – CAD Software Reseller (Systematics/Solidworks)

Addressing customers questions on all aspects of a product

Manned or Automated History – Thousands heterogeneous complex cases Product evolves rapidly and continually Examples:

CAD software – Solidworks support website

Page 35: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 41

Solidworks Support Website

Page 36: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 42

Conversational CBR – Help Desk

Problem characteristics: Problems could be classified in categories.

Each category has attributes that are common to all (or most) problems of the category. There could be little overlap in attributes of problems belonging to different categories.

Problem categories could be organized in a hierarchy. Users (usually) can relate a problem to a general category or environment, and give

keywords that describe their problem. Users cannot supply all information needed to understand the exact problem.

They should be asked questions about the problem, until an accurate specification of the problem is defined.

The majority of the problems are repetitive or variants of such.

The proposed solution (based on these characteristics) includes three main steps: Automatic unsupervised category tree creation of all cases.

Derived from characteristics #1, #2, and #5. Finding in the category tree the category most similar to the new problem.

Derived from characteristics #2, #3, and #5. “Interactive dialog” with the user asking questions to reduce the number of candidate

solutions. Derived from characteristics #4 and #5.

Page 37: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 43

TeSAS System Design

Page 38: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 44

TeSAS Case Structure

Keywords: Words that describe the problem such as export, DXF, layer, or BOM. A user may define as many keywords as she likes.

Environments: There are four in SolidWorks: Part, Assembly, Drawing, and General. A case may be represented by a combination of environments. For example, a Bill Of Material problem may be defined as a Drawing related problem and/or an assembly one.

Topics: List was taken from SolidWorks Knowledge Base. A topic may be Data Exchange, Detailing, Features, Mates, Dimensions, Notes, etc. A case may have multiple topics. For example, a dimension problem of an imported DWG file may be associated to Dimensions and Data Exchange.

Page 39: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 45

TeSAS Category Node Structure

CategoryControl

Units

Options/Settings

What kind of anIMPORT problem is it?

Units

Keywords

Environments

Topics

Q/A

Page 40: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 47

TeSAS Session – # 1: Input New Case

Page 41: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 48

TeSAS Session – # 2: Category Selection

Page 42: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 49

TeSAS Session – # 2: Q&A Verification

C0

C30C44

Is it an import or export problem? --> ImportC28

C4C1

What file format is required/used? --> DXF/DWGC15

C52Is it a drawing related problem --> Yes

What kind of DXF/DWG import problem is it? --> TemplateC2 C10

Page 43: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 50

TeSAS Session – # 2: Q&A Verification

Page 44: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 51

TeSAS Session – # 3: Case Selection and Q&A Retrieval

Page 45: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 52

TeSAS Session – # 3: Case Selection and Q&A Retrieval

Under the hood Selecting the question that differentiates best between the

children of the present intermediate node Presenting the question and descending the hierarchy

based on answer

Page 46: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 53

TeSAS Session – # 4: Action Retrieval

Page 47: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 54

TeSAS Results – Users’ Performance

5(6+3+2) test cases Problem complexity – difficulty of characterization

Page 48: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 55

TeSAS Intermediate Summary

System implementation with 2 GLTs eCobweb “C4.5” – Entropy minimization

System design could be represented by a simple graph

Page 49: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 56

Attendance Officers Support System

Close to 9% (10.9%) of the students in Israel (US) drop out of school.

School dropout is a systemic problem with both human and social aspects.

Attendance officers, who enforce the education attendance laws, find themselves dealing with many different cases in a situation with limited resources.

Unfortunately, creative solutions that often arise in such situations, are not shared by the attendance officers’ community since there is no system that accumulates knowledge.

Develop a case management system for sharing lessons.

Page 50: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 57

CBR with Weak Causal Knowledge

The problem facing attendance officers has the following attributes: It is based on a many-to-many mapping. The domain is highly dimensional. The problem domain is inhomogeneous and context dependent. The solution is highly subjective. Problem solving is based on few (e.g., 3-4) characteristic case

attributes. No available decontextualized domain knowledge exists. Future auditing or quality control requires explaining solutions. Practitioners are the sole source of knowledge and they have little

motivation to spend effort beyond their usual work. Any solution devised must fit naturally into their present work practice.

There is high cost to solution failure.

Page 51: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 58

System Design

expanding baseline system

developing baselinequality, efficient, tested

system

structureimportance

few/nonecases

highsubjectivity

timedependency

context(place,

environment)dependency

high costfor failure

use casessolved by

practitioners

automaticprocedures

multiplesources

flexibility ofproblem

representation

matrix relationbetween

properties &solutions

problemsolving

based onpast

experience

diversecase

collection

solution'sfeedback

fast,effective,

qualitysolution

use ofsimple &

accessibletools

finding best casesfrom a cluster and

assessing theirquality

case qualitymaintenance bycommunity and

designated experts

inability toformalize

decisionrules

CBR

case matrixclustering

solution withminimal

input

augmentinput withknowledge

maintainperson in

decision loop

case qualitycontrol

reclustering

qualitymeasure

cluster'squality/index

systemexamination

need

activity

method/means

result

decomposeprocess

simplerep.

heterogenemousspace

clustering

2. usingcluster's cases

1. findingclusters

collectqualified cases

from fieldexpert

characterizeclusters

collaborativefiltering

derivedneed

poor preliminaryproblem

understanding

problemunderstanding

developappliedsystem

participatorydevelopment

literaturereview

clustering the casebase and findingthe most suitable

group or groups tonew case

continuous caseacquisition from

field practitioners

system architecture anddevelopment process

cluster correctionand reassessment

Legend

no codifiedknowledge

many fieldusers

Page 52: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 59

“Straight Forward” Clustering Failed

Attribute-value pair representation Tested with many clustering algorithms (traditional

statistical, NN, fuzzy) Clustering was unsuccessful

Case representation was missing critical information Search for alternative representation Influence graph

Easy to use Minimal time

Page 53: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 60

Influence Graph Representing Weak Knowledge

1

3

2

4

5

Output

Input

Ranking

1

M

N

1

Weight

Ranking

3

1

2

7

1

2

1

4

5

2

3

1

Output

Input

Ranking

1

M

N1

1 32 45

3

1

2

7

1

21

52

3

1

4

Page 54: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 61

Solution Algorithm

Stage I: Memory organization Given: n cases represented by an influence graph. Cluster cases (using E) into g clusters , (g is not specified

a priori). Characterize clusters.

Stage II: SolutionGiven: a new case described by a set of inputs and a set of g clusters,

: Identify key inputs. Retrieve b best matching clusters , . Retrieve c nearest neighbors from members of . Solve using the outputs of the c cases. Complete .

Remarks: a may be as low as 3, b would be 1 or 2 in most cases, c should be manageable, e.g., about 3 from each cluster. If b is larger than 2, c should be adjusted accordingly.

gGGG ,,1

1nC1nP

gGGG ,,1

1 nka gbest GGGG ,,1 gb

bestG

iS1nC

1nE

Page 55: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 62

Results – Users’ Performance

7(3+2+1) + 5(2+2+2)

Page 56: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 63

Results – Users’ Performance

Page 57: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 64

Intermediate Summary System implementation with 2 GLTs (clustering and supervised learning)

System design could be represented by an influence graph

A simple graph could be used as the basis for building decision support system

If it is good for attendance officers, it might be good for us!If it is good for attendance officers, it might be good for us! If we recorded the design of KDD systems with influence If we recorded the design of KDD systems with influence

graphs we could have used them to build a support system graphs we could have used them to build a support system that would assist us in future system development.that would assist us in future system development.

Y. Reich and A. Kapeliuk, “A framework for organizing the space of DSS with application to solving subjective, context dependent problems,” Decision Support Systems, 2004.

expanding baseline system

developing baselinequality, efficient, tested

system

structureimportance

few/nonecases

highsubjectivity

timedependency

context(place,

environment)dependency

high costfor failure

use casessolved by

practitioners

automaticprocedures

multiplesources

flexibility ofproblem

representation

matrix relationbetween

properties &solutions

problemsolving

based onpast

experience

diversecase

collection

solution'sfeedback

fast,effective,

qualitysolution

use ofsimple &

accessibletools

finding best casesfrom a cluster and

assessing theirquality

case qualitymaintenance bycommunity and

designated experts

inability toformalize

decisionrules

CBR

case matrixclustering

solution withminimal

input

augmentinput withknowledge

maintainperson in

decision loop

case qualitycontrol

reclustering

qualitymeasure

cluster'squality/index

systemexamination

need

activity

method/means

result

decomposeprocess

simplerep.

heterogenemousspace

clustering

2. usingcluster's cases

1. findingclusters

collectqualified cases

from fieldexpert

characterizeclusters

collaborativefiltering

derivedneed

poor preliminaryproblem

understanding

problemunderstanding

developappliedsystem

participatorydevelopment

literaturereview

clustering the casebase and findingthe most suitable

group or groups tonew case

continuous caseacquisition from

field practitioners

system architecture anddevelopment process

cluster correctionand reassessment

Legend

no codifiedknowledge

many fieldusers

Page 58: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 65

Towards better management of KDD processes

Question: What do we need to manage in order to improve KDD processes and their future understanding/reuse/accounting?

Present KDD process definitions – CRISP-DP

Integrating ideas from PDM KDD is equivalent to design or product development

Versions (of data source, software tools, needs) Linkages between different aspects of the solution Context …

KDMLKDML: Knowledge Discovery Modeling Language An evolving language describing the information used in KD processes

Page 61: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 68

CRISP-DM Use – KDnuggets Poll, July 2002

Page 62: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 69

Documents, DMS, …, PDM

AuthorNameCreated onLast modifiedFormat

AccessAccessrightsrights

Configuration: rules of creationLayout: presentation style (both based on user’s profile)

Check-in/Check-in/

Check-outCheck-out

Mark-Mark-upup

SharedSharedworkspaceworkspace

PersonalPersonalworkspaceworkspace

DocumentRepository

WorkflowWorkflow

Page 63: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 70

KDML

Y. Reich, Life cycle management of information and decisions for system analyses, Mechanical Systems and Signal Processing, vol. 15, no. 3, pp. 513–527, 2001.

Page 64: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 71

KDML Validation

Validation in KD processes - future Initial validation:

Own experience in developing systems Analysis of reported projects

Page 65: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 72

KDML – Preliminary Validation

Page 66: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 73

KDML – Preliminary Validation

Page 67: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 74

Vision

Reports of KDD processes will include influence graph representation of design decisions

A tool that realize KDML will be developed, disseminated, and evolved by the community or a vendor Improve KDD practice Increase reuse of previous processes Generate new research agenda

KDD Process reports will become as important as “theoretical” development

Page 68: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 75

Checkup – New Data Mining Model and Computation Tools for Extracting Knowledge from Databases and Predicting Time-Dependent Processes Goal

Design and develop a new machine learning model (Checkup Model) for the production of rules with high accuracy from databases.

The produced rules are in the form of a conjunctive list of mathematical and logical expressions.

The rules are designed for further use such as predictions, knowledge discovery, and decision-making.

Limitations of existing models Difficulty in handling unknown mathematical and logical

conjunctions correlations, especially numeric functions such as: ceil, floor, mod, etc.

Inability to handle various attribute types such as: values (nominal or numeric) and vectors (array of values) where the classification attribute could be either numeric or nominal.

Page 69: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 76

The Machine Learning Model

The model is unique since: It is capable of learning from datasets that other models cannot. The learning can be directed to investigating and finding relations, heuristic

correlations, and empirical functions. The error rate is equal or better compared to other machine learning algorithms

on a target set of relevant databases. The knowledge is shown as rule collections with every rule having a unique

structure of logical and mathematical functions. The produced rules are short and minimal as possible. It is possible to add limitations and known relations between attributes.

Uniqueness

Page 70: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 77

The Machine Learning Model

Current machine learning models can learn from datasets whose attributes are numeric (integer or float) or nominal (string). A vector is defined as a list of values where the value could be any type as float, integer, or string.

Checkup supports vector-type attributes.

Learning from datasets whose attributes have various types such as scalar or vector.

The vector attributes can have different lengths, for all attributes in all instances. Meaning that a vector which is a list of items could be in any desire list count, anywhere.

Produce rules that have logical and mathematical structure with vector functions.

Applications: vector attributes can be use in medicine follow-up for prediction the best medicine taking or decide the best process behavior during time like heat treatment, etc.

Vector support

Page 71: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 78

Example of DB with Vector Attributes

Page 72: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 79

Rule Set (example: UCI Labor Database)

IF ((duration-4*working_hours)*(-3*duration-working_hours)) < 7480.5 AND (3*duration+working_hours) < 46.5 AND

)3*Sqr(duration)+working_hours+3 < (66.5 AND )Sqr(duration)-4*Sqr(pension) < (2.5

THENclass = bad

IF ((duration-4*working_hours)*(-3*duration)-working_hours) < 6635 AND-)duration+3*working_hours < (112.5 AND

)working_hours < (39.5THENclass = good

Page 73: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 80

Checkup Model

Learning (Data to Rules) Iterative feedback optimization (Generation of new

attributes) Overfitting protection (Avoid learning noise and

mistakes) Optimize system settings (Parameters tuning)

Page 74: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 81

Checkup

InitialConditions

to Rules

Attributesto

Conditions

DecodeRules

Find Best(Conditionsand rules)

Build VirtualAttributes

ContinueLearning

OutputRules

PackRules

Discretization and Build Conditions

Overfitting and pruning

Translate rules to readable syntax rules

Initialize virtual attribute storage

Page 75: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 82

Attributes to Conditions

Discretization. transforms a continuous attribute’s values into a finite number of intervals and associates with each interval a

numerical, discrete value. Build conditions. build all relevant combinations of type

“Attribute Φ interval” where Φ could be greater, equal or smaller than. (ex. if a>3 then …)

Page 76: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 83

Conditions to Rules

Given a collection of conditions, we use it to create rules. The rules need to describe the training data part. Each rule is a set of conditions (ex. if (a>3) and (b<4) then …)

Incremental covering of positive examples Continue expansion as long as rule cover negative examples Order of conditions is based on # of covered positive examples

Rules are added incrementally to the list after excluding previously used conditions

Page 77: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 84

Overfitting and Pruning

The extracted rules may fit the training data, including the noise. To protect from overfitting and to reduce the number of rules in such a case so that only the most relevant rules are used, a pruning method has been implemented.

The pruning method is called a packing algorithm Packing generates various combinations of rule-sets. The best rules from the set are extracted based on their

contribution to increasing performance (reducing error). Consequently, every packing method is like pruning in a

different way. All packing rules combinations are checked, and the best

packing that produces a higher prediction score is chosen.

Page 78: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 85

Building Virtual Attributes

The virtual attributes are functions-of-current attributes or virtual-attributes with most-common attributes or last-most-common attributes. Three different methods for building the new attributes are present.

1) Mixing the original attribute with extracted original attributes (derived from the most common attributes).

2) Mixing the last extracted original attribute with current extracted original attributes (both derived from the most common attributes).

3) Mixing the last virtual attribute with current attributes.

Page 79: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 86

A sample of mathematical operations for creating virtual attributes

Y=k1*S1+k2*S2Y=S1*S2Y=min(S1,S2)Y=max(S1,S2)Y=power(S1,3)Y=abs(S1)Y=sign(S1)Y=sqr(S1)Y=sqrt(S1)Y=trunc(S1)Y=round(S1)

Y=floor(S1)Y=ceil(S1)Y=mod(S1,2)Y=mod(S1,3)Y=mod(S1,S2)Y=log(S1)Y=ln(S1)Y=sin(S1)Y=cos(S1)Y=tan(S1)

Simple operations: +, -, *, /

Si=Attribute (Simple or Vector)k=Random Constants

Page 80: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 87

A sample of mathematical operations for creating virtual attributes (vector)

Y=SumVec(D) {Series: +a1+a2+a3+a4+...}

Y=MinVec(D) {Series: min(a1,a2, a3, a4…)}

Y=MaxVec(D) {Series: max(a1,a2, a3, a4…)}

Y=MulVec(D) {Series: a1*a2*a3*a4*...}

Y=Ser1Vec(D) {Series: +a1-a2+a3-a4+...}

Y=Ser2Vec(D) {Series: 1*a1+2*a2+3*a3+4*a4+...}

Y=Ser3Vec(D) {Series: 1*a1+4*a2+9*a3+16*a4+...}

Y=Ser4Vec(D) {Series: a1^(1/n)+a2^(2/n)+a3^(3/n)+a4^(4/n)+... case n items}

Y=Ser5Vec(D) {Series: an-2+an-1+an; last 3 items in vector}

D=Vector Attribute

Page 81: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 88

Principle of Operation

data

Training data

Testing data

Train

Test

Cross validation k1-Fold

Cross validation k2-Fold

Rules Parameter

Tuning

Packing selection,

whatever needs to be selected…

Page 82: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 89

Principle of Operation

Parameter tuning System parameters that control learning are tuned in the process. The parameters are:

Discretization levels: find the optimal discretization splitting levels, or groups, for all attributes.

Error threshold: find the optimal value for the maximum error allowed in rules.

Search method: find the best search method, extracting rules from the best algorithm or complex algorithm.

Page 83: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 90

Search space Space of functions (logical and mathematical) Set of rule sets

Search strategy Depth first search (with a “front” rather than a single solution) Different reinforcement strategies for attribute creation

Principle of Operation

Page 84: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 94

Results and Significance DatasetC4.5Checkup (I=1)Checkup (I=10)

balance-scale78.2081.1888.27

breast-cancer72.9673.5272.47

breast-w94.5996.6096.71

bupa65.4065.9568.99

colic85.1783.7083.63

credit-a85.6284.7185.29

credit-g71.2972.4871.78

diabetes73.9475.5974.28

glass67.6568.3364.06

hayes-roth70.3670.2073.86

heart-c75.5078.3280.47

heart-h79.2179.1578.47

heart-statlog78.7278.1582.96

hepatitis79.2882.5881.78

ionosphere89.7785.6188.60

iris94.8994.5695.11

labor79.9290.3687.28

lymph76.2779.7377.54

vote96.2794.5496.32

wine93.8296.7395.04

zoo92.8693.4295.55Irvine Datasets

Page 85: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 95

Results and SignificanceModelScoreZeroR50

DecisionStump53DecisionTable50

HyperPipes49.5

IB157IBk57

C4.553.5RandomTree56.5

KStar51Logistic45.5

NaiveBayes44.5OneR51

SMO45VotedPerception51.5

VFI50.5NeuralNetwork52.3

LinearRegression49.5Checkup98

Checkup compared to various other machine learning models (random dataset with hidden mathematical functions) .

Page 86: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 96

Significance

We find that Checkup is at the top of the prediction accuracy list even in I=1

When increasing iterations only to three, the prediction score becomes much higher

Current existing models have difficulty handling the numeric functions in datasets compared to the Checkup model

The model is highly flexible enabling learning from a large collection of mathematical operation; adding newer functions and generic equations to the model (system) is simple

Page 87: Case-Based Reasoning – through 3 applications Copyright © 2006 Reich Prof. Yoram Reich Faculty of Engineering Tel Aviv University Israel

Copyright © 2004 Reich 97

References

Chandrasekaran, B. Generic tasks in knowledge-based reasoning: high-level building blocks for expert system design. IEEE Expert 1(3):23-30, 1986.

Y. Reich and S. J. Fenves, Integration of generic learning tasks, Tech. Rep. EDRC 12-28-89, Engineering Design Research Center, Carnegie Mellon University, Pittsburgh, PA, 1989. Available at http://www.eng.tau.ac.il/~yoram/topics/generic-learning.html.

Y. Reich, “Constructive induction by incremental concept formation,” in Artificial Intelligence and Computer Vision (Y. A. Feldman and A. Bruckstein, eds.), pp. 191–204, Amsterdam: Elsevier Science Publishers, 1991.

Y. Reich, Life cycle management of information and decisions for system analyses, Mechanical Systems and Signal Processing, vol. 15, no. 3, pp. 513–527, 2001.

Y. Reich and A. Kapeliuk, Case-based reasoning with subjective influence knowledge, Applied Artificial Intelligence, vol. 18, no. 8, pp. 735–760, 2004.

G. Chen and Y. Reich, A conversational case-based reasoning help-desk utility for complex products. Submitted, 2004.

Y. Reich and S. J. Fenves, The formation and use of abstract concepts in design, in Concept Formation: Knowledge and Experience in Unsupervised Learning (D. H. J. Fisher, M. J. Pazzani, and P. Langley, eds.), (Los Altos, CA), pp. 323–353, Morgan Kaufmann, 1991.

Y. Reich, Macro and micro perspectives of multistrategy learning, in Machine Learning: A Multistrategy Approach, Vol. IV (R. S. Michalski and G. Tecuci, eds.), (San Francisco, CA), pp. 379–401, Morgan Kaufmann, 1994.

Y. Reich, “Measuring the value of knowledge,” International Journal of Human-Computer Studies, vol. 42, no. 1, pp. 3–30, 1995.

Y. Reich and A. Kapeliuk, “A framework for organizing the space of DSS with application to solving subjective, context dependent problems,” Decision Support Systems, 2004.