assessment in spreadsheets

LWA 2009

What you understand is what you get: Assessment in SACHS

Andrea Kohlhasea,b Michael Kohlhaseb,a

aGerman Center for Artificial Intelligence (DFKI), BremenbSchool of Computer Science and Engineering, Jacobs University Bremen

Handles on Knowledge

Knowledge Management (Probst, Raub, Romhardt;1997)

“0“,“9“,“5“,“,“

Exchange rate

1 $ = 0,95 €

Market mechanisms concerning

exchange rates0,95

Character Set

Context NetworkingSyntax

Glyphs Data Information Knowledge

3

Semantic Technology: Motivation

Volume of Data Knowledge

What is available is what you get!

≠

4

Semantic Technology: Metadata

Volume of Data Knowledge≠DataaboutData

≠

What you want is what you get!

K n o w l e d g e

5

Semantic Technology: Services

Volume of Data K n o w l e d g e

DataaboutData

≠

Semantic

Service

s≠

What is delivered is what you get!

K n o w l e d g e≠

6

Semantic Technology: Interactions

Volume of Data K n o w l e d g e≠DataaboutData

≠

Semantic

Service

s

≠

Semantic

Interactions

What you engage with is what you get!

≠ K n o w l e d g e

7

Semantic Technology: Spirit

Volume of Data ≠DataaboutData

≠

Semantic

Service

s

≠Semantic

Interactions

What you understand is what you get!

≠

K n o w l e d g e

8

Semantic Technology: Paradigms

Volume of Data ≠DataaboutData

≠

Semantic

Service

s

≠Semantic

Interactions

Semantic Illustration

Semantic Web

≠

K n o w l e d g e

9

Semantic Illustration Architecture: SACHS

Interpretation MappingSemantic Document

Financial Controlling System

10

„What is available is what you get!“ SACHS

11

Background Ontology

12

„What is delivered is what you get!“: SACHS

13

„What you engage with is what you get!“: SACHS

14

SACHS: Semantic Navigation

15

„What you understand is what you get!“: SACHS

16

SACHS: Framing

17

The „Wizard of Oz“ Experiment

Users

System

Requirements?

18

The „Wizard of Oz“ Experiment: SACHS

Author

ExpertMP3

Recordings

Financial Controlling

SACHS Semantic Document

19

Analysis of MP3 Recordings„Knowledge Items“: #39

„Explanations“:#110

„Explanation Types“: #7

20

Explanation Types

• Definition

• Purpose

• Assessment of Purpose

• Assessment of Value

• Formula

• Provenance

• History

Cover ratio =

Coverage of necessary costsCover ratio is a performance figure.

Optional Actions: None More costs needed More equity capital

needed

21

Explanation Types

• Definition

• Purpose

• Assessment of Purpose

• Assessment of Value

• Formula

• Provenance

• History

Trigger optional actions:• „= 100%“ everything‘s perfect• „< 100%“ more costs needed• „> 100%“ more equity capital

Verification of understanding

Origin of values

Layout/content evolution

22

Explanation TypesHow many times was each explanation type

chosen for describing a knowledge item? # Explanation Type

# Knowledge Items

DefinitionPurpose

Assessment of Purpose

Assessment of ValueFormula

ProvenanceHistory

23

Relevance RankingHow many times was each explanation type

chosen for describing a knowledge item?

Definition

Assessment of Value

Purpose

Provenance

Assessment of Purpose

Formula

History

71.8%

51.3%

46.2%

43.6%

30.8%

23.1%

15.4%

# Explanation Type# Knowledge Items

24

Help Needed - but Where?How many times was an explanation type chosen? # Explanation Type

# Explanations

25

Our Conclusion

SACHS covers not enough dimensions of understanding …

Assessment of Value!

26

What is Assessment?

Row 6 looks good!The revenues look good!

... but that is a disaster!

I like this ...

27

Assessment

Model

Generic assessment theory with functions to

judge

28

Assessment

Model

Concrete assessment theories with judgement functions

29

Assessment

Model

Concrete theories with

domains

30

SACHS: Assessment Module

31

SACHS: Assessment Module

+ Explanations

32

SACHS: Assessment Module

33

Upper Management will be happy about the leftover funds, but the PI of the project will be angry that he got less work out of the projected than

expected. Not to mention the funding agency and their policy ….

Multicontext Assessment

34

Multicontext Assessment Model

Concrete variants of assessment

theories

35

SACHS Conclusion

More (semantic) coverage of

dimensions of understanding

is needed!

36

General Conclusions

GenericAssessment

ConcreteAssessmentConcreteAssessment

ConcreteAssessment

ConcreteAssessment

ConcreteAssessment

ConcreteAssessment

ConcreteAssessment

ConcreteAssessment

• Understanding includes several dimensions Not one ontology for all!

Content strategy!• System ontology and Domain ontologies

37

Thank you!