effects of using examples on structural model comprehension: a controlled experiment dina zayan,...

Effects of Using Examples on Structural Model Comprehension:

A Controlled Experiment

Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki

2014.06.06

http://gsd.uwaterloo.ca http://necsis.ca

http://gsd.uwaterloo.ca/



2

Many problems in SE are related to domain knowledge transfer

3

For example…

• Business analysts communicating to other stakeholders who are novices

• Industry standard which contains domain models applied by novices

• Specifications written by experts given to suppliers

4

Why do we have these problems?

Domain Knowledge

Complex in nature

Difficult to articulate

Difficult to communicate

5

Let’s run a mini experiment

6

In a “second degree price discrimination” strategy, price varies according to the quantity demanded. Larger quantities are available at a lower unit price. Additionally, sellers applying

this strategy are not able to differentiate between different types of consumers. Thus, the suppliers will provide incentives for the

consumers to differentiate themselves according to preference. This also applies to

product quality.

This is an Abstraction

Airlines often offer multiple classes of seats on flights, such as first class and economy class.

This is an Example

7

Examples + Abstractions

Effective Knowledge Transfer

Results in Cognitive Psychology and Software Engineering

8

Modeling

Model = Abstractions + Examples

Example-Driven

9

Of Course Examples Help!

+ Capture intuition+ Relate to existing knowledge+ Show the intended usage of an abstraction

10

However…

1. When exactly do examples help?2. How much can they help?3. What types of examples help?4. Which stakeholders benefit the most

from examples?5. Where is a PROOF that they help?

11

Provide the Missing Empirical Evidence and Guidelines

Objective

12

“Analyze the effects of using explicit examples

on structural model comprehension

in the context of knowledge transfer to model receptors

who are novices to the application domain.”

Controlled Experiment

13

Two Groups of Participants

EDM Control

Model Abstractions

Model Abstractions

Experimental Procedures & Requirements

List

Experimental Procedures & Requirements

List

Examples!

(13) (12)

14

Instrumentation & Actors Map

Experimental Procedures &

Requirements List

Model Abstractions

SME Model Creator

Examples!

15

Treatments and Metrics

Sobeys Object Diagram

EDM

Control

1723 classes, 34 associations, and 8 constraints

18

(EDM Only)

4 positive and 2 negative examples

19



1) Diagram Completeness

EDM

Control

20

Diagram Completeness

• Mean (Control) = 49.2 %• Mean (EDM) = 88.1 %• p-value < 0.0002• Maximum score for

Control is lower than the minimum score for the EDM

+ 37% increase

21



1) Diagram Completeness2) Diagram Mistakes

EDM

Control

22

Diagram Mistakes• Mean (Control) = 15.1

mistakes• Mean (EDM) = 3.5

mistakes• p-value < 0.0053• Maximum number of

mistakes for EDM is equal to the minimum number of mistakes for Control

- 80% decrease

23



1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence

EDM

Control

24

Participants Rated Themselves

Difficulty

1 (very easy) to 5 (very difficult)

– Control: 3.83– EDM: 2.46

-27% decrease

Confidence

1 (unsure) to 5 (very sure)

– Control: 2.33– EDM: 4.08

+ 35% increase

25


Study Questions



EDM

Control

28


Study Questions



1) Task 2 Completeness

EDM

Control

29

Task 2 Completeness

• Mean (Control) = 13.6• Mean (EDM) = 21.2• p-value < 0.0012

+ 31% increase

Max 24

30


Study Questions



1) Task 2 Completeness2) Efficiency

EDM

Control

31

Task 2 Efficiency

• Mean (Control) = 0.7• Mean (EDM) = 1.2• p-value = 0.00082

+ 71% increase

32


Study Questions



1) Task 2 Completeness2) Efficiency3) Perceived Difficulty4) Perceived Confidence

EDM

Control

33

Participants Rated Themselves

Difficulty

1 (very easy) to 5 (very difficult)

– Control: 3.25– EDM: 2.08

-23.5% decrease

Confidence

1 (unsure) to 5 (very sure)

– Control: 3.00– EDM: 4.15

+ 23% increase

34


Study Questions


Participants Questions



1) Number of domain-related questions

EDM

Control

35

Number of Domain-Related Questions

• Control: 10 questions/session

• EDM: 1 question/session

- 90% reduction

36


Study Questions


Participants Questions

Feedback



1) Number of Domain-related Questions

1) UML Class Diagram2) Examples

EDM

Control

37

Understanding Without Examples?

EDM: “The class diagram was huge, I didn’t know where

to start reading to understand, so I looked at the examples one by one while looking at the model abstractions simultaneously.”

38

Classes Most Difficult To Comprehend

Class Control EDM Mistakes Control/EDM

Misses Control/EDM

BonusMechanic ✔ ✔ 5/2 4/0Redemption Mechanic

✔ ✔ 9/0 3/0

“Partner” Offer ✔ x 6/0 3/0Bill ✔ ✔ 9/3 0/0

40

How Were the Examples Helpful?

• Example 2: • Regular product vs. product with an offer associated• Application of the BonusMechanic class

• Example 4: • Propagation of points across classes and the

sequence for accumulating points• Account’s “pointsBalance” updated with Bill’s

“totalPoints” attribute

42

How Were the Examples Helpful?

• Example 5:• Redemption process

• Example 6:• Violation of the constraint prohibiting accumulation and

redemption for the same bill instance

43

Are Both Positive and Negative Examples Needed?

• “Yes” 12/13 EDM participants

• No comments about the number and content of examples

44

Discussion

46

Examples are mostly needed by Novices

• FixedPriceOff vs. FixedPercentOff– familiar

• RedemptionMechanic, RedemptionChannel, and BonusMechanic – foreign

47

Having a Variety of Examples

Useful when1. Having more than one association linked to a

single class

49



single class2. Complex constraints

51



single class2. Complex constraints3. Propagation of attributes through classes

53



single class2. Complex constraints3. Propagation of attributes through classes4. A domain concept that is spread over several

classes and associations (Points Accumulation)

54

EDM participant who preferred valid examples only answered the related question incorrectly

55

Partial Examples

• 12 participants– partial examples, no complete object diagram

• One participant– partial examples and complete object diagram

56

Conclusions

57

Quantitative Analysis

Dependent Variable

Significant? (p-value)

Hyptheses Testing

% Improvement due to the use of

examplesDiagram Completeness

Yes (<0.0002) Accept H1 +39%

Diagram Mistakes

Yes (<0.0053) Accept H2 -80%

Task 2 Completeness

Yes (<0.0012) Accept H3 +31%

Task 2 Efficiency

Yes (=0.00082) Accept H4 +71%

58

Qualitative Analysis

Dependent Variable % Improvement due to the use of examples

Perceived difficulty in creating the object diagram.

-27%

Perceived difficulty in answering Task 2 questions.

-23%

Confidence in their created object diagrams.

+35%

Confidence in the answers they provided for Task 2.

+23%

59

Limitations & Future Work

• Partial examples as views on a complete example

• Variety of examples vs. positive only

• Cost of creating and maintaining the examples vs. the benefit

• Empirical studies in an industrial context

Thank You!

Replication packageis available for download

http://gsd.uwaterloo.ca http://necsis.ca

Examples!




61

Example-Driven Modeling Using

• Unified syntax for abstractions and examples + redefinition

• Partial examples + automatic completion and constraint checking

• Specialization + extension

http://clafer.org

http://clafer.org/

effects of using examples on structural model comprehension: a controlled experiment dina zayan,...

Documents

efficiency edm control

mean edm

controlled experiment

exampledriven slide

perceived difficulty

software engineering

difficult control

sure control