creating a software factory · creating a software factory accelerate development, augment code...

V1.0 | 2019-04-24

Steve Barriault – Director Sales and Field Engineering APAC – VectorCAST – Vector Informatik

Creating A Software FactoryAccelerate development, augment code quality

2

u Motivation

Software and Testing

Accelerating Development and Testing

How To Build the Ultimate Test Environment

Conclusion

Agenda

3

Who we are?

Motivation

Aerospace

Automotive

Medical

Industrial

Business-Critical

Railway

Founded 1990, Rhode Island, USA

Founded 1988 , Stuttgart, Germany

Since 2017 there is only one

4

‘Conforming to a rule’

Did we build it right?

u Not necessarily did we build the right thing: requirements!

In this presentation, we will just concentrate on developing software

Compliance

Motivation

ISO 26262-6 2011; Page 5

5

u “80% of embedded systems are delivered late” *

u “New code generally has 50 to 100 bugs per thousand lines” *

Software has bugs

Motivation

* Jack Ganssle, embedded systems commentator

Quality

Speed Price

Idealbut

possible?

Good and fast

Good and cheap

Cheap and fast

Expensive Slow

Poor

6

In many projects: test == buffer

Motivation

time

Design and Requirements

Coding TestConcept and

Planning

Releasedate

Startdevelopment

Plan

Reality 1

Reality 2

Design and Requirements

Coding Test

Design and Requirements

Coding Test

Releasedate

Commercial and reputational risk: delay or bugs

Release and Maintenance

7

Motivation

u Software and Testing

Accelerating Development and Testing

How To Build the Ultimate Test Environment

Today’s Demo

Conclusion

Agenda

8

What should a Software Developer be?

Software and Testing

Software is critical…

What we need is a

Software Factory

Would you put a house together the way we put software together?

What are carpenters always have in their toolbox – and constantlyusing?

Answer: a level and a meter tape!

9

Motivation

Software and Testing

u Accelerating Development and Testing

The Ultimate Testing Environment

Conclusion

Agenda

10

Key defining principles to build the ultimate software factory

Accelerating Development and Testing

Nbr Key principle

1 Leverage tried and tested manufacturing standards

2 Understand and track the requirements

3 Create a continuous testing environment

4 Work smarter not harder

5 Build a scalable constant integration engine

11

u Understand the System

u Just-In-Time delivery

u Find bottlenecks

u Boost communication and collaboration

u Stop the production if there is an issue

u Everyone helps if there is a major issue

#1 Leverage tried and tested manufacturing standards

Accelerating Development and Testing

An example of a Kanban board applied to Software development process

Managing the ‘work in progress’

12

#2 Understand and track the requirements

Accelerating Development and Testing

Coding

System Testing

Requirements Analysis

High LevelDesign

DetailedDesign

Integration Testing

UnitTesting

System Requirements

High-Level Requirements

Low-Level Requirements

Run Test CasesLink

Run Test CasesLink

Run Test CasesLink

Customer

Request

▪ Basis for SW Development

▪ Basis for SW Tests (functional)

▪ Allows Traceability

• Requirement <-> Test Case

• Requirement <-> Source Code

▪ Allows Change Impact- Analysis

• When source code changes:

• -> what requirement might be affected

• When a requirement changes:

• -> what source code might be affected

• -> what test case might be affected

Write Test

Cases

Write Test

Cases

Write Test

Cases

13

#3 Create a continuous testing environment

Accelerating Development and Testing

u Test as early as possible

u Use simulation

u Define test cases with the requirements

u Regression Testing on each code change

u Change-based testing –only run tests affected by source code changes OR requirement changes

u "Baseline test cases“ option for legacy code

u How? Code coverage tells us which tests go through which functions.

void func1(void)

{

/* I have changed! */

}

void func2(void)

{

/* I have NOT changed! */

}

Tests A, B, D, …

Tests A, C, E, J, Z

14

#4 Work smarter not harder

Accelerating Development and Testing

An example?

Early Detection

1.Engineer modifies code & injects a BUG!

2.Engineers commits code

3.Continuous Integration server runs impacted test casesERROR DETECTED IMMEDIATELY

4.Notification is given to the team, correction is brought to the code

Error Prevention

1.Engineer modifies code & injects a BUG!

2.Engineer runs tests affected by the change immediately on his own PC

3.ERROR DETECTED – no one is impacted except the engineer

4.The code is committed when no tests are failing(Quality gate check)

Both approaches reduce debugging time significantly but error prevention completely eliminates the (negative) impact

of bug introduction onto the other team members

Now?

1.Engineer modifies code & injects a BUG!

2.Engineers commits code

3.Tests run once in a while (2 weeks?) and maybe find the trouble

4.DEBUG PHASEWork until found and fixed!

15

u Continuous integration in an assembly line process

u A fully tested source code base should always be available to produce a product

u Each stakeholder can run all tests

u Developer

u Tester

u CI Server

u And what about different code configurationsand calibrations?

u Tests should be recyclable as much as possibleacross all possible configs and calibrations

#5 Build a scalable constant integration engine

Accelerating Development and Testing

Tests from sports-car configuration being used (and specialized, if need be) by SUV, at a click of a mouse

16

Motivation

Software and Testing

Accelerating Development and Testing

u How To Build the Ultimate Test Environment

Conclusion

Agenda

17

Test Cases grow during project development

The Ultimate Testing Environment

System Requirements

System

Test Cases

High-Level Requirements

Integration Test Cases

Low-Level Requirements

Unit

Test Cases

Test Cases

Source Code

Coding

18

What can change?

u Requirement

u Invalidate Test Cases linked to that Requirement

u Source Code Traceability can give a clue what code needs to be changed

u Test Case

u Re-run the changed Test Case

u Source Code

u A real challenge

u Do we need to execute all test cases?

u Tools can help reducing the test execution times with an intelligent Test Case Selection

Change Impact Analysis

The Ultimate Testing Environment

19

Continuous Testing of Code Changes

The Ultimate Testing Environment

System Requirements

System

Test Cases

High-Level Requirements

Integration Test Cases

Low-Level Requirements

Unit

Test Cases

Test Cases

Source Code

Coding

Code Change

20

Change-Based Testing Integrated Into Continuous Integration

Simulator / Target

VectorCAST/AnalyticsWeb dashboard analysis

Simulator / Target

Managers

VectorCAST Integration/functional automation (CANoe, scripts, etc)

VectorCAST Unit/module testing

High-level requirement black-box testing

Low-level requirement White-box testing

The Ultimate Testing Environment

VectorCASTand other tools, reqs

21

Parallel Testing

The Ultimate Testing Environment

Independent Test Environments

Test Environment

Full Run

Test Environment Full Run

Test Environment Full Run

Test Environment Full Run

Test Environment

Full Run

Test Environment

Full Run

timeTest Environment

Full RunTest

Environment Full Run

Test Environment Full RunTest Environment

Full Run Test Environment Full

Run

Test Environment Full Run

Sequentialexecution

Parallelexecution

22

u It is important to visualize project metrics in charts, graphs and heat maps

Visualization

The Ultimate Testing Environment

23

u Define and use quality gates

u Build documented workflows for each Activity

u SCM commit requires > Clean status analysis report?

> Completed peer review?

> All tests OK on home machine? Some tests OK?

> +More?

u Branch cannot go to QA until all existing tests pass

u Quality gates Must be Automated (factory!)

Potential work flow: Software change

The Ultimate Testing Environment

24

Motivation

Software and Testing

Accelerating Development and Testing

How To Build the Ultimate Test Environment

u Conclusion

Agenda

25

SW Factory

How will you test in your next project?

Conclusion

Weeks Minutes

Time to Test

Manual Automatic

Level of Automation

Per Release Per Change

Frequency of Test

QA Everyone

Who Runs Tests