creating a software factory · creating a software factory accelerate development, augment code...

V1.0 | 2019-04-24

Steve Barriault – Director Sales and Field Engineering APAC – VectorCAST – Vector Informatik

Creating A Software FactoryAccelerate development, augment code quality

2

u Motivation

Software and Testing

Accelerating Development and Testing

How To Build the Ultimate Test Environment

Conclusion

Agenda

3

Who we are?

Motivation

Aerospace

Automotive

Medical

Industrial

Business-Critical

Railway

Founded 1990, Rhode Island, USA

Founded 1988 , Stuttgart, Germany

Since 2017 there is only one

DO-178.pptx

ISO-26262.pptx

IEC-62304.pptx

IEC-61508.pptx

BIZ-IOT.pptx

EN-50128.pptx

4

‘Conforming to a rule’

Did we build it right?

u Not necessarily did we build the right thing: requirements!

In this presentation, we will just concentrate on developing software

Compliance

Motivation

ISO 26262-6 2011; Page 5

5

u “80% of embedded systems are delivered late” *

u “New code generally has 50 to 100 bugs per thousand lines” *

Software has bugs

Motivation

* Jack Ganssle, embedded systems commentator

Quality

Speed Price

Idealbut

possible?

Good and fast

Good and cheap

Cheap and fast

Expensive Slow

Poor

6

In many projects: test == buffer

Motivation

time

Design and Requirements

Coding TestConcept and

Planning

Releasedate

Startdevelopment

Plan

Reality 1

Reality 2


Coding Test


Coding Test

Releasedate

Commercial and reputational risk: delay or bugs

Release and Maintenance

7

Motivation

u Software and Testing



Today’s Demo

Conclusion

Agenda

8

What should a Software Developer be?


Software is critical…

What we need is a

Software Factory

Would you put a house together the way we put software together?

What are carpenters always have in their toolbox – and constantlyusing?

Answer: a level and a meter tape!

9

Motivation


u Accelerating Development and Testing

The Ultimate Testing Environment

Conclusion

Agenda

10

Key defining principles to build the ultimate software factory


Nbr Key principle

1 Leverage tried and tested manufacturing standards

2 Understand and track the requirements

3 Create a continuous testing environment

4 Work smarter not harder

5 Build a scalable constant integration engine

11

u Understand the System

u Just-In-Time delivery

u Find bottlenecks

u Boost communication and collaboration

u Stop the production if there is an issue

u Everyone helps if there is a major issue

#1 Leverage tried and tested manufacturing standards


An example of a Kanban board applied to Software development process

Managing the ‘work in progress’

12

#2 Understand and track the requirements


Coding

System Testing

Requirements Analysis

High LevelDesign

DetailedDesign

Integration Testing

UnitTesting

System Requirements

High-Level Requirements

Low-Level Requirements

Run Test CasesLink

Run Test CasesLink

Run Test CasesLink

Customer

Request

▪ Basis for SW Development

▪ Basis for SW Tests (functional)

▪ Allows Traceability

• Requirement <-> Test Case

• Requirement <-> Source Code

▪ Allows Change Impact- Analysis

• When source code changes:

• -> what requirement might be affected

• When a requirement changes:

• -> what source code might be affected

• -> what test case might be affected

Write Test

Cases

Write Test

Cases

Write Test

Cases

13

#3 Create a continuous testing environment


u Test as early as possible

u Use simulation

u Define test cases with the requirements

u Regression Testing on each code change

u Change-based testing –only run tests affected by source code changes OR requirement changes

u "Baseline test cases“ option for legacy code

u How? Code coverage tells us which tests go through which functions.

void func1(void)

{

/* I have changed! */

}

void func2(void)

{

/* I have NOT changed! */

}

Tests A, B, D, …

Tests A, C, E, J, Z

14

#4 Work smarter not harder


An example?

Early Detection

1.Engineer modifies code & injects a BUG!

2.Engineers commits code

3.Continuous Integration server runs impacted test casesERROR DETECTED IMMEDIATELY

4.Notification is given to the team, correction is brought to the code

Error Prevention


2.Engineer runs tests affected by the change immediately on his own PC

3.ERROR DETECTED – no one is impacted except the engineer

4.The code is committed when no tests are failing(Quality gate check)

Both approaches reduce debugging time significantly but error prevention completely eliminates the (negative) impact

of bug introduction onto the other team members

Now?


2.Engineers commits code

3.Tests run once in a while (2 weeks?) and maybe find the trouble

4.DEBUG PHASEWork until found and fixed!

15

u Continuous integration in an assembly line process

u A fully tested source code base should always be available to produce a product

u Each stakeholder can run all tests

u Developer

u Tester

u CI Server

u And what about different code configurationsand calibrations?

u Tests should be recyclable as much as possibleacross all possible configs and calibrations

#5 Build a scalable constant integration engine


Tests from sports-car configuration being used (and specialized, if need be) by SUV, at a click of a mouse

16

Motivation



u How To Build the Ultimate Test Environment

Conclusion

Agenda

17

Test Cases grow during project development


System Requirements

System

Test Cases


Integration Test Cases


Unit

Test Cases

Test Cases

Source Code

Coding

18

What can change?

u Requirement

u Invalidate Test Cases linked to that Requirement

u Source Code Traceability can give a clue what code needs to be changed

u Test Case

u Re-run the changed Test Case

u Source Code

u A real challenge

u Do we need to execute all test cases?

u Tools can help reducing the test execution times with an intelligent Test Case Selection

Change Impact Analysis


19

Continuous Testing of Code Changes


System Requirements

System

Test Cases


Integration Test Cases


Unit

Test Cases

Test Cases

Source Code

Coding

Code Change

20

Change-Based Testing Integrated Into Continuous Integration

Simulator / Target

VectorCAST/AnalyticsWeb dashboard analysis

Simulator / Target

Managers

VectorCAST Integration/functional automation (CANoe, scripts, etc)

VectorCAST Unit/module testing

High-level requirement black-box testing

Low-level requirement White-box testing


VectorCASTand other tools, reqs

21

Parallel Testing


Independent Test Environments

Test Environment

Full Run

Test Environment Full Run



Test Environment

Full Run

Test Environment

Full Run

timeTest Environment

Full RunTest

Environment Full Run

Test Environment Full RunTest Environment

Full Run Test Environment Full

Run


Sequentialexecution

Parallelexecution

22

u It is important to visualize project metrics in charts, graphs and heat maps

Visualization


23

u Define and use quality gates

u Build documented workflows for each Activity

u SCM commit requires > Clean status analysis report?

> Completed peer review?

> All tests OK on home machine? Some tests OK?

> +More?

u Branch cannot go to QA until all existing tests pass

u Quality gates Must be Automated (factory!)

Potential work flow: Software change


24

Motivation




u Conclusion

Agenda

25

SW Factory

How will you test in your next project?

Conclusion

Weeks Minutes

Time to Test

Manual Automatic

Level of Automation

Per Release Per Change

Frequency of Test

QA Everyone

Who Runs Tests

creating a software factory · creating a software factory accelerate development, augment code...

Documents