Experimental Design

Threats to Validity

Threats to Validity• Statistical—are results due to some systematic factor (hopefully the independent

variable) or are results due to chance variations• Threats—measures used to assess the dependent variable are unreliable• --researchers violations of the assumptions that underlie statistical tests• Construct—how well do the study results support the theory or constructs

behind the research—• Threats—poorly defined definitions, or research is based on unsound/non-

validated constructs• External—the degree to which we are able to generalize the results of a study to

other subjects, conditions, times, and places• Threats—poor sampling (non-random), failure to sample across different times,

places, and conditions ecological validity—refers to appropriate generalization from the laboratory to real-life or natural environmental situations

• Internal—was the independent variable and not some extraneous variable responsible for changes in the dependent variable

• Threats—confounding variables (independent variable varies with at least one other variable), maturation, regression to the mean, history

Confounding Variables• Maturation—the natural aging process leads to changes irrespective of

influences of the independent variable• History—events that transpire during the course of the experiment• Testing—effects of repeated testing—subjects gain proficiency through

repeated practice• Instrumentation—change in calibration of the measuring instrument• Regression to the mean—tendency for values of a variable to be less

extreme over time• Selection—groups not equivalent at outset of study• Attrition—loss of subjects during the course of the study• Diffusion of treatment—subjects gain information about other research

conditions from subjects in those other conditions• Sequencing effects—subject’s later performance is influenced by previous

conditions of the study

Experiment Effects

• Subject effects—when people know they are being observed and behave differently than they would in more familiar situations

• • Placebo effect—false results because subjects expect a

specific effect of an experimental manipulation• • Experimenter effects—expectations of researcher might

cause researchers to bias results—through data selection or by subtle influencing outcome (blinds can compensate some)

Traceability

Traceability Management for Evolution Support

• Traceability: documenting for evolution• Traceable – can fully figure out where an item comes

from, why it comes from there, and where it goes to.• Forward vs. Backward Traceability (need both!)• Vertical Traceability

– Business objectives to requirements, requirements elicitation documentation to RD

• Horizontal Traceability – Definitions, assumptions

Figure 6.3 – Traceability links: forward, backward, horizontal, and vertical traceability

Objectives, domain concepts, requirements, assumptions

Architectural components & connectors

Source code Test data User manual

horizontal

vertical

forward backward

Traceability Links

• Include info such as:– Rationale– Date– Author– Contributors, Stakeholders– Status (proposed, rejected, accepted, approved, deferred, etc.)

• Can be between:– Requirements– Assumptions– Definitions– Etc.

Traceability Link Types

• Dependency – A depends on B• Variant – B is the variant of master A• Revision – B is the next version of previous

version A• Use – B uses A; A is used by B• Derivation – A is met by B; B is derived from A

Figure 6.4 – A taxonomy of traceability link types

Dependency link

Inter-version link Intra-version link

Revision Variant Derivation Use

Subtype

Figure 6.5 – Dependency link type

A dependsOn

Dependency B affects

Figure 6.6 – Variant link type

A variantOf

Variant B

master

Figure 6.7 – Revision link type

A next

Revision B

previous

Figure 6.8 – Use link type

A uses

Use B

usedBy

Figure 6.9 – Derivation link type

A derivedFrom

Derivation B

metBy

Evolution Example• The software to be designed will control a simulated automated teller machine (ATM) having a magnetic stripe reader for

reading an ATM card, a keyboard and display for interaction with the customer, a slot for depositing envelopes, a dispenser for cash (in multiples of $20), a printer for printing customer receipts, and a key-operated switch to allow an operator to start or stop the machine. The ATM will service one customer at a time. A customer will be required to insert an ATM card and enter a personal identification number (PIN) - both of which will be sent to the bank for validation as part of each transaction. The customer will then be able to perform one or more transactions. The card will be retained in the machine until the customer indicates that he/she desires no further transactions, at which point it will be returned - except as noted below.

• A customer must be able to make a cash withdrawal from any suitable account linked to the card, in multiples of $20.00. Approval must be obtained from the bank before cash is dispensed.

• A customer must be able to make a deposit to any account linked to the card, consisting of cash and/or checks in an envelope. The customer will enter the amount of the deposit into the ATM, subject to manual verification when the envelope is removed from the machine by an operator. Approval must be obtained from the bank before physically accepting the envelope.

• A customer must be able to make a transfer of money between any two accounts linked to the card. • A customer must be able to make a balance inquiry of any account linked to the card. • The ATM will communicate each transaction to the bank and obtain verification that it was allowed by the bank. In the case

of a cash withdrawal or deposit, a second message will be sent after the transaction has been physically completed (cash dispensed or envelope accepted).

• If the bank determines that the customer's PIN is invalid, the customer will be required to re-enter the PIN before a transaction can proceed. If the customer is unable to successfully enter the PIN after three tries, the card will be permanently retained by the machine, and the customer will have to contact the bank to get it back.

• If a transaction fails for any reason other than an invalid PIN, the ATM will display an explanation of the problem, and will then ask the customer whether he/she wants to do another transaction.

• Source: http://www.cs.gordon.edu/courses/cs320/ATM_Example/Requirements.html

Evolution Example• The software to be designed will control a simulated automated teller machine (ATM) having a magnetic stripe reader for

reading an ATM card, a keyboard and display for interaction with the customer, a slot for depositing envelopes, a dispenser for cash (in multiples of $20), a printer for printing customer receipts, and a key-operated switch to allow an operator to start or stop the machine. The ATM will service one customer at a time. A customer will be required to insert an ATM card and enter a personal identification number (PIN) - both of which will be sent to the bank for validation as part of each transaction. The customer will then be able to perform one or more transactions. The card will be retained in the machine until the customer indicates that he/she desires no further transactions, at which point it will be returned - except as noted below.

• A customer must be able to make a cash withdrawal from any suitable account linked to the card, in multiples of $20.00. Approval must be obtained from the bank before cash is dispensed.

• A customer must be able to make a deposit to any account linked to the card, consisting of cash and/or checks in an envelope. The customer will enter the amount of the deposit into the ATM, subject to manual verification when the envelope is removed from the machine by an operator. Approval must be obtained from the bank before physically accepting the envelope.

• A customer must be able to make a transfer of money between any two accounts linked to the card. • A customer must be able to make a balance inquiry of any account linked to the card. • The ATM will communicate each transaction to the bank and obtain verification that it was allowed by the bank. In the case

of a cash withdrawal or deposit, a second message will be sent after the transaction has been physically completed (cash dispensed or envelope accepted).

• If the bank determines that the customer's PIN is invalid, the customer will be required to re-enter the PIN before a transaction can proceed. If the customer is unable to successfully enter the PIN after three tries, the card will be permanently retained by the machine, and the customer will have to contact the bank to get it back.

• If a transaction fails for any reason other than an invalid PIN, the ATM will display an explanation of the problem, and will then ask the customer whether he/she wants to do another transaction.

Source: http://www.cs.gordon.edu/courses/cs320/ATM_Example/Requirements.html

Change: The ATM must be able to service multiple customers and be able to use any bill denominations $100 and under.

Figure 6.10 – Derivational traceability links implied by satisfaction arguments

SysReq: system requirement ASM: environment assumptions

DOM: domain properties

SOFREQ: software requirements

G: system objective ASM: environment assumptions

DOM: domain properties

REQ: requirements

G: system objective OP: system operations

Derivation

Derivation

Derivation

0..*

MasterVersion

VariantOf

0..1

0..1

PreviousFrom

NextTo

0..1

Figure 6.11 – Item traceability: an entity-relationship model

1..*

Derivation

DerivedFrom

MetBy

0..* Use Uses

UsedBy

RD item

RefName Spec

SW lifecycle item

RefName Spec

DerivedFrom

0..*

1..*

0..*

MasterVersion

VariantOf

0..1

0..1 PreviousFrom

NextTo

0..1

Revision

CM attributes

Variant CM attributes

Revision

CM attributes

Variant CM attributes

Inter-version dependencies Intra-version dependencies

Figure 6.12 – Item traceability: model instantiation to meeting scheduling

Derivation

DerivedFrom

Revision

Uses

Participant notification

requirements

Notifier Module Specs

RefName: NMS Spec: ….

DerivedFrom

NextTo

Notification Test Data

RefName: NTD Spec: …

Notification requirements + SMS notification

MS interface requirements

MS interface reqs + GSM interface reqs

Revision

Objective: Anywhere anytime notification

Derivation Use

User Manual Sect.: Notification

RefName: UMN Spec: …

Use

…

RefName: AA-Notif Spec: ….

RefName: Notif+SMS Spec: ….

RefName: MSI/GSM Spec: ….

Inter-version dependencies Intra-version dependencies

Figure 6.13 – Traceability management

Establish traceability links

Exploit traceability links

Maintain traceability links

Define traceability policy

Figure 6.14 – Single-relation traceability graph represented by the matrix in Table 6.2

T1 T2

T3

T4

T5

Figure 6.15 – Feature diagram for variants of the meeting scheduling system

MeetingScheduling

MeetingInitiation

ConstraintsAcquisition Planning RuleBased ConflictResolution

MeetingNotification

byEmail byE-agenda Dates Dates&Locations byEmail bySMS

WithPreferences WithoutPreferences

Figure 6.16 – Change control

Change initiation Change evaluation and prioritization

Change consolidation

Retro Demo

Integrating Multiple System Views

Chapter 14Requirements Engineering: From System Goals to

UML Models to Software SpecificationsAxel van Lamsweerde

Meta-Models

• Meta-model – defines and interrelates conceptual abstractions in terms of which other models are defined.

• 3 Levels of Meta-Models:– Meta Level – domain-independent abstractions– Domain Level – concepts specific to the modeled

system.– Instance Level – specific instance of domain-level

concepts in the running system

Roles of a Meta-Model

• Define structure of modeling language; provides meta-language

• Provides framework where intra- and inter-view consistency rules can be defined.

• Yields logical schema.• Allows modeling to be defined as strategies for meta-

model traversal.• Summarizes features of modeling language; allows for

variants and easy comparison.• Provides a basis for tool support and integration.

Chapter 14

Figure 14.1 – The meta, domain, and instance levels

CopyReturnedOnTime If Borrowed

Patron Borrow

Book BookCopy Copy

Input

Performs

Responsibility

Agent Entity

Association

Operation Goal Responsibility

Performance

Input

Link

CopyReturnedBy 11/11/07 Axel

Borrow

L.Dupré, Bugs in Writing

info/40/123 Copy

Input

Performs

Responsibility

meta-concept

meta-relationship

Instance

concept

concept instance

Meta-Model Structure

• Meta-Attributes:– Name – unambiguous reference to corresponding

instances at the domain level.– Def – unambiguous definition of corresponding

instances at the domain level.– Issue – (optional) records process information

when corresponding instances are elaborated.

Meta-Models

• Goal• Object• Agent• Operation• Behavior

Figure 14.2 - Overall metamodel structure

SystemModel

ObjectModel GoalModel AgentModel OperationModel BehaviorModel

Figure 14.3 – The goal metamodel

Obstacle

DomDescript

ObstructedBy Divergence

OR-Ref

1..*

GoalModel

LeafGoal

Achieve

Maintain/Avoid

Goal

Name Def [Category] [Priority] [Source] [Stability]

Refinement

[Status] [Tactic] [SysRef]

BehavioralGoal

[FormalSpec]

SoftGoal

[FitCriterion] BoundaryCondition

[Category] [Likelihood] [Criticality] [FormalSpec]

AND-Ref

1

*

*

*

*

1..*

O-Refinement

[Status]

OR-ref

AND-ref

Resolution

Requirement

Expectation

*

*

Figure 14.4 – The object metamodel

Agent

Goal 1..*

ObjectModel

Attribute

DomInvar

DomHyp Object

Name Def InstanceOf

DomDescript Name Def [FormalSpec]

Range

2..*

1..*

Concern

ValuesIn

Rigidity Multiplicity

Link

Role Multiplicity Position

DomInit

Association Entity Event

ApplicationSpecific Built-In

Specialization

Aggregation

Figure 14.5 – The agent metamodel

Monitoring

Goal

AgentModel

Attribute

Agent Name Def [Load]

1

0..1

OR-Ass

* Assignment

[SysRef]

Association

SoftwareToBeAgent

Control

stateVar

EnvironmentAgent Requirement Expectation

Dependency depender

dependee

dependum

Responsibility

Responsibility

1

1..*

1..*

1

Wish

1 1..*

LeafGoal Ass

1..*

Figure 14.6 – The operation metamodel

Input

LeafGoal

OperationModel

Attribute

Operation

Name Def DomPre DomPost [Category]

1

Operationalization

[ReqPre] [ReqTrig] [ReqPost]

Association

Output

stateVar Agent Performance

1..*

1..*

1..*

Instance

InternalEvent

Figure 14.7 – The behavior metamodel

Goal

BehaviorModel

State

1

Scenario

Controlled Variable

Attribute

Association

Agent BehaviorOf

1..*

1

1..*

AgentSM

Transition

1..*

Label

Guard

Event

Operation

Input

Output

Sequential Composition

Parallel Composition

1..*

InstanceCoverage

Goal

TimelineSlice

1..*

0..1

*

0..1

Interaction

0..*

Episode

Source

Target

InteractionEvent

1

1 Instance

1

1..*

InternalEvent ExternalEvent

Class Coverage

1..*

History

StateMachine

Coverage

Path

1

1

1..*

1..*

Inter-view Consistency Rules• Two views of a system model are said to be structurally consistent if they

satisfy a set of rules constraining their respective elements for compatibility and complementarity.

• Patterns for consistency/view integration:

for every item it1 satisfying some property P1(it1) in view V1,there exists a corresponding item it2 in view V2 that satisfies some property

P2(it1, it2) linking it1 and it2

for every instance ins1 of metaconcept C1 in metamodel component MC1,there exists a corresponding instance ins2 of metaconcept C2 in metamdel

component MC2, linked to ins1 by an instance of the inter-view metarelationship IR.

Rules

• See page 494-496

Grouping Related View Fragments into Packages

Grouping Related View Fragments into Packages

Considerations:• A package might regroup model parts

pertaining to the same view• It might contain model elements related to

the same goal category, the same goal, the same functionality, or the same agent

• It should regroup elements that are likely to evolve together

Figure 14.8 - Model packaging

GoalModel

Functional Goals

Usability Goals

Performance Goals

Interoperability Goals

Development Goals Goals

OperationlModel

Software Operations

Legacy Components

Staff Tasks

package containing other packages

package LibraryModel

GoalModel ObjectModel

Operation Model

Agent Model

Behavior Model

import dependency

LibraryModel

Catalog System

BiblioSearch System

Journal Management

Loan System

BookAcquisition System

<<import>>

dependency