1

Embedded TDD Cycle –The First 3 Years

Markku Åhman and Timo PunkkaSchneider Electric

first.last at schneider-electric.com

Agenda for today:

• Embedded Characteristics

• Unit Testing

• Dual Targeting

• Acceptance Testing Using Simulation

• Testing in Target HW

• Results and Summary

2

Embedded characteristicsCross-compilationEvent-drivenRTOS dependenciesReal time constraintsLow level HW dependenciesLimited tools supportRobustness

7

+2Low resource budgetLimited user interface

150KLOC CLegacy Code from 10+yrs RTOS /w taskingModular multi-uCSafety critical (property andhuman)

Main BoardRenesas uC2M Flash2M RAM20MHz

3

4

Problem statement:

“We can’t automate the testing of this”

Our Cycles

Unit Tests indevelopmentenvironment

Acceptance testsusing simulation

Automated testsIn target HW

Explorative Testing

Adapted from James Grenning, Embedded Test Driven D evelopment Cycle, 2004

Lenght of feedback cycle LongShort

5

Unit Test Build

#include <unity.h>

#include "test/mock/code/loopdb/Mockaddr_alarm_interface.h"

#include "code/utils/lists.h"#include <something.h>

Test build is described in test file using keywords

6

Has worked:CMock (be careful)Small group workshopsHigher level tests (through pinch point)Coaches as visitorsWebinarsRunning tests on PC

Dual Targeting

DrawTextLine();

Main();

IndicateAlarm();

DrawAlarmDisplay();

DrawCharacter();

DrawPixel();

display memory

RealImplementation

?DrawTextLine();

Main();

IndicateAlarm();

DrawAlarmDisplay();

DrawCharacter();

DrawPixel();

display memory

DrawTextLine();

Windows GUIComponent

SimulationImplementation

7

Dual Targeting

DrawTextLine();

Main();

IndicateAlarm();

DrawAlarmDisplay();

DrawCharacter();

DrawPixel();

fakedisplay memory

DrawBitmap();

Windows GUIComponent

UpdateDisplay();

ConvertToBitmap();

SendTextLineTo TestDriver();

Dual Targeting

8

Benefits of Simulation

• Running in development environment

• Fast build-run cycles

• Better debugging environment

• More portable code

Challenges in simulation

• The real-time

• Multi-tasking (OS)

• Distributed systems

9

Timing multiple executables

Executable A

for (;;) {Run_A();Win32.Sleep(1 ms);

}

Executable B

for (;;) {Run_B();Win32.Sleep(5 ms);

}

Ideal execution

Run A, t=0 Run B, t=0

Run A, t=1

Run A, t=2

Run A, t=3

Run A, t=4

Run B, t=5Run A, t=5

Run A, t=6

Executable A Executable Brealtime

0 ms

1 ms

2 ms

3 ms

4 ms

5 ms

6 ms

10

Possible execution

Run A, t=0 Run B, t=0

Run A, t=1 Run B, t=5

Run B, t=10

Executable A Executable Brealtime

0 ms

10 ms

20 ms

30 ms

40 ms

50 ms

60 ms

Run A, t=2

Run A, t=3

Run A, t=4

Run A, t=5

Run A, t=6

Run B, t=15

Run B, t=20

Run B, t=25

Run B, t=30

Another possible execution

Run A, t=0 Run B, t=0

Run A, t=1

Run B, t=5

Run B, t=10

Executable A Executable Brealtime

0 ms

10 ms

20 ms

30 ms

40 ms

50 ms

60 msRun A, t=2

Run B, t=15

Run B, t=20

11

Simulation time

Executable A

for (;;) {Run_A();Simulation.Sleep(1 ms);

}

Executable B

for (;;) {Run_B();Simulation.Sleep(5 ms);

}

SimulationTime Controller

Sleep 1 Sleep 51 ms passed 5 m

s passed

12

Tested in Development Environment?

Can:• High level functionality

• Multi-tasking

• Distributed systems

• Communication of parts

• Communication stress

Cannot:• Low level functionality

• Real concurrency

• Compiler differences

• CPU stress

13

SD

CABLESPAN 23tellabs

SD

OK1

OK2

PS1

PS2

TEMP

100

/15,2

30v~

20/

30Hz,

8.0/1

.75 A

BA

NK1

1x

2x3x

4x

5x

6x7x

8x

BAN

K21x

2x

3x

4x

5x6

x7

x8

x

1 5

2 6

3 7

4 8

BAN

K21 5

2 6

3 7

4 8

BAN

K1

CON

SOL

E

SD

OK1

OK2

PS1

PS2

TEMP

100

/15,2

30v~

20/

30Hz,

8.0/1

.75 A

BA

NK1

1x

2x3x

4x

5x

6x7x

8x

BAN

K21x

2x

3x

4x

5x6

x7

x8

x

1 5

2 6

3 7

4 8

BAN

K21 5

2 6

3 7

4 8

BAN

K1

CON

SOL

E

SD

OK1

OK2

PS1

PS2

TEMP

100

/15,23

0v~

20/3

0Hz,

8.0/1

.75 A

BAN

K1

1x2x

3x4

x

5x6x

7x8

x

BAN

K21

x2

x3

x4x

5x

6x

7x

8x

1 5

2 6

3 7

4 8

BAN

K21 5

2 6

3 7

4 8

BAN

K1

CON

SOL

E

SD

OK1

OK2

PS1

PS2

TEMP

100

/15,23

0v~

20/3

0Hz,

8.0/1

.75 A

BAN

K1

1x2x

3x4

x

5x6x

7x8

x

BAN

K2

1x

2x

3x

4x

5x

6x

7x

8x

1 5

2 6

3 7

4 8

BAN

K21 5

2 6

3 7

4 8

BAN

K1

CON

SOL

E

SD

TEST ACT ALM PWR1 2 3 4 C T R I LI OKNGLINE ALM

1 2 3 4LINE MODE

ACPWR

ON

OFFDC

1 21 2 3 12 3 4 5CLKMODE RATE

2.0A 2.0A

AC/DCTEST

OFF ON

SD

P ower

Fault

1 2 3 4 5 6

7 8 9 10 11 12

Link

Mode

Mode

Link

13 14 15 16 17 18

19 2120 22 23 24

Mode S elect

FdxAct

1X 2X 3X 4X 5X 6X

7X 8X 9X 10X 11X 12X

13X 14 X 15X 16X 17X 18X

19X 20 X 21X 22X 23X 24X

10/10 0Base-T Por ts

100Reset Clear

Module Stat usSelf Test

Console

P roCurve Switch 2424MHP J41 22B

14

User ExperienceLoad / StressExplorative

QA’s Cool New Role

Regression

Developer 1

Developer 2

Version Control Server

SubVersion(SVN)

Backed Up

Commit/Update

Commit/

Update

Update

Continuous Integration Server

CruiseControl

Not Backed Up

Uses

Build Scripts

Rakefile (Ruby Rake)

Target Binaries

Test reports

Monitors for feedback

Compile for PC simulation

Compile target binaries

Execute acceptance tests

Execute unit tests

Run test coverage

Continuous Integration Daily (fast, smoke suite)

Nightly (full suite)

15

12 releases later…

January May September

At some point

Development

To: 1 week/0-5 defects

From: 4 months/150 defects

Final System Test

Testing

16

Survey results (+5 .. 0 .. -5)

50%

15%

21%

14%

+5 (Much better)

+4

+3

+2

Key Remaining Challenges

Acceptance tests in low level language

Adopt new design style for testability

Flaky tests

17

Nightly build trend

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 1213 14 15 16 17 18 19 20 21 22 23 24 25 26 2728 29 30 31Sprint #

Tes

t #

Acceptance Tests Unit Tests

23% line coverage

80% line coverage

18

Photo creditsRobot: beni_bbFire: winlordQuestion marks: immrchrisIce arrow: inyaFire Brigade: 13dede All @ stock.xchng