1

Toward Target Level Testing Toward Target Level Testing and Debugging for and Debugging for

Embedded SoftwareEmbedded Software

Koehnemann, LindquistKoehnemann, Lindquist

•Ritu Varma•Roshanak Roshandel•Manu Prasanna

2

IntroductionIntroduction• Identifies problems with the test case execution

for embedded systems and propose solutions• Testing is done to reveal errors whereas

debugging is done to locate and correct the cause of error.

• Embedded systems are usually constrained by Concurrent design Real time constraints Distributed hardware architecture Device control dependencies

• Execution visibility and control severely restricted

• Why testing in Embedded Systems necessary?

3

Software TestingSoftware Testing• Testing for embedded systems has four stages:

Module level Testing Integration Testing System Testing Hardware/Software Integration Testing

• Testing Methods which address the problem identified by the embedded systems

Concurrency Real-time constraints Embedded Environment

are of interest to this paper.

4

Testing Concurrent SystemsTesting Concurrent Systems• Concurrency increases testing difficulties:

Large set of execution sequences Subsequent Execution with the same input

different results Valid result as well as how the program arrived at

that result Effect of the environment on testing and its results

• Acceptable output more important than correct output- Control over program execution

• Research: Static analysis: must examine a large set of state

not scalable Dynamic testing: passive watch of execution and

replaying the execution in case of failure Runtime control of concurrent program

5

Non-intrusive TestingNon-intrusive Testing• Control and testing techniques must not interfere with

behavior of test program, timing issues • Why not instrumentation?

Insert probes to watch execution, link users object code with the rest of debugging system, Code interferes with execution behavior of the program

Intrusive, voids execution of system

• Non-intrusive debugger for Ada Separate processor executes testing system and

communicates with the target processor through specialized hardware

Good start but don’t deal with the high level activities (process scheduling ,fault handling) only low level ( monitoring bus activity)

• Other Real time embedded tools: ROM monitors, Emulators, Bus Monitors

6

Impact of Underlying SystemImpact of Underlying System• Problem: dealing with abstraction• Concurrency, IPC at high level and implementation

at complier level (on run-time system)• In Development phase abstraction is fine but in

Testing phase? Complication Without sufficient control over program , there is no way

to ensure that a test case is testing the code. Implementation difference in the host and the target

environment.

• Compilers provide runtime system operation to bridge the gap between language construct and hardware.

7

Current State of Embedded Current State of Embedded TestingTesting

• Concurrency causes Problems for testing and debugging• H/S integration testing reveal problems such as:

Incorrect handling of interrupts, Distributed Communication problems, Incorrect order of concurrent events, etc.

• Goal: reveal errors early as possible Problem: Target level testing tools don’t exist

• Two environments- Host : supports s/w development and tools which give user

control over program execution (high level interface) Target: no support for software development and tools as

minimum cost and space are the goals of the target.

Target level testing occurs late in the development cycle and a

small part is allocated for H/S integration testing. But,Target is the only location where errors can be revealed.

8

Current SolutionsCurrent Solutions• Hardware solutions

Attempts at gaining execution visibility and program control Bus monitors, ROM monitors and in-circuit emulator Minimal effectiveness for software development - low level

machine data gathered only Mapping b/w low level events and entities in the program

• Software solutions Attempts to reduce the cost and time spent testing on target Determining factors in software testing:

Level of Criticality of software module Test platform availability Test Classification

Criteria's that assign program modules to test platforms Type of software, Hardware requirements, Test classification,

Platform availability, Certification Requirements

• Current solution attempts to reduce the amount of target testing

9

Are these Techniques Are these Techniques Effective?Effective?

• Expensive• Low level of functionality• Available too late • Poor test selection criteria• What about new architectures?

10

ExpensiveExpensive

•Custom validation facilities•Little or no reuse•Hardware is late and erroneous Software must be malleable

11

Testing ProblemsTesting Problems• Host: high-level interface, language • Target: machine instruction and

physical address• Translation: Tedious and error prone • Migration to target

Hardware’s errors have to be mitigated by software

Late in the development process – costs

12

More ProblemsMore Problems• Test cases and scenarios depend on

tools and platforms instead of theoretical test criteria

• Guidelines not being followed• Hardware architecture changes rapidly• Solution:

Adding functionality to the underlying system to support debugging

Hardware and run-time system

Specialized Middleware for development and testing of embedded systems?

13

Model Debugging SystemModel Debugging SystemAdd functionality to targetAdd functionality to target

• Ada Semantic Interface Spec toolkit• Two communicating processor:

Running the test program Running the debugger

• Target processor can: Execute and debug code only at a break point Run the debugger as a separate processor Provide a separate execution unit to execute

the debugger

• Interface between internal and external debugger

14

Changes in HW ArchitectureChanges in HW Architecture

• Hardware partitioning of memory• Computational facilities for

debugger• Hardware break points• Dedicated bus

15

Memory PartitioningMemory Partitioning

• Software is infinitely malleable Any change requires several tests and

retests

• Late changes is expensive• Hardware should bound software• SW is usually partitioned based on critical

level instead of design factors

Hardware partitioning – protected address space for processes – restriction on access

16

Computational Facilities for Computational Facilities for DebuggerDebugger

• External debugger vs. Internal debugger • Internal debugger: located on target, executes from the

target without interfering with the application Runs as a regular process on the processor

Low-level or periodic high level process Architecture provides separate facility for debugger execution

Target processor provides computational facility

• Computation Facility utilize architecture to dump some representation of instruction being executed

Info must be parsed to create history. Implement state machines for messages sent by HW.

Program the (stateful) functional unit within the processor. At boot time, the internal debugger code gets loaded and reside there. Messages are passed between internal and external debuggers.

17

Hardware BreakpointsHardware Breakpoints• Software and conditional

breakpoints are not feasible for real time systems

• Breakpoint registers – data and instruction types

• State of a processor – all internal registers, pipeline and cache information must be saved when a break is encountered

• Conditional breakpoints – how are they dealt with?

18

Architecture Support for Architecture Support for AbstractionsAbstractions

• Utilities must be built into architecture to provide basis for emulation, testing and program visibility

• A new level of abstraction is needed: Basic type (int, float) Process, memory management, semaphore, etc

• Hardware should be more aware of programming elements (software)

• Awareness of the programming environment is important

19

Dedicated BusDedicated Bus• An interface between the processor and the

external world• Protocol used across the connection

data rate, volume, direction, etc.

• Master-slave relationship Internal and external debuggers Bus and debugger

• Determination of active element within the processor Processor is active – sends messages to debugger Use special debugger portion of RTS – both debugger

and RTS are active Perhaps a combination of the two

20

Runtime System AdditionsRuntime System Additions• Goal: to minimize data and computational

requirements of the internal debugger along with communication with the external debugger

• Standardize services that provide visibility into RTS abstractions

• Processes: concurrency is a common abstraction Need to support modifying task state,

communication and synchronization, and process scheduling

• Interrupt management, Time management, Memory management, Exception Handling

21

SummarySummary• Embedded system testing techniques are ill-

equipped• Architectural and RTS changes have to be made• Evaluate the changes and check feasibility• Definition of Embedded Systems??

What is the border between HW/SW? Software techniques more suitable for less resource

constraint system Need for classification of embedded system??

• Need for specialized middleware for both design and testing of embedded systems

• Need for software development techniques although not discussed in the paper