(c) 2011 dave stewart - inhand electronics - systems conference boston, sept 20111 unique software...

(c) 2011 Dave Stewart - InHand Electronics - www.inhand.com Embedded Systems Conference Boston, Sept 20111

Unique Software Considerations in Military and Avionic Embedded Systems

Embedded Systems Conference Boston 2011ESC-402

Dave Stewart, PhDDirector of Research and Systems IntegrationInHand [email protected]


Overview

Most software presentations on Military and Avionic Systems focus on two key items: Reliability and Fault Tolerance

Security

This presentation focuses on many other aspects of software that are somewhat unique to military and avionic systems, yet are rarely discussed


Outline

Is that REALLY the Requirement? Self Destruct! Development Environment != Deployed System Sorry, no debug I/O allowed, but debug it quick! Error detection: Why bother? It Uses Too Much Power!


Is that REALLY the Requirement?Understanding the Real Requirements

Hundreds of pages of requirements This is typical for military and avionic applications If only tens of pages, first step is likely a need to refine the

requirements!

Separate between “wish-list” and “really really required” Many requirement documents are the culmination of the wish-

lists of everyone involved in prior projects Given fixed cost contracts, understand which ones have

flexibility, and which do not The systems engineer should look closely for any conflicting

requirements; address those first!


Is that REALLY the Requirement?The Impossible Requirements

It is not unusual to find requirements that seem like they will be very difficult to meet

When this happens, investigate the requirement carefully, and make sure what the customer wants is properly captured

Understand every Use-Case Scenario. Often the documented requirement does not actually meet the needs of the use cases

Yet, a simpler requirement might actually meet the needs of all use cases

Review how that requirement was implemented in previous or competitor projects

Some requirements are designed to capture what was implemented on a prior system, whether in-house or by a competitor

Review those other implementations closely, as more often than not, the customer really wants the same thing, not something new that is substantially more complex


Self Destruct!Zeroization

Many requirements call for the zeroization, but the actual meaning is left to interpretation. The interpretation is often “erase flash” upon either user

activation or detection of tamper This is often overkill. The real requirement imposed by military

and security agencies is usually that an enemy who gains hold of the device is unable to access any classified data

Therefore unencrypted classified code and data must be erased quickly and reliably

Ask the customer Do you really need to erase an operating system that is readily

available in the public domain?


Self Destruct!Understand the Use Cases

If the customer insists they mean erase all flash, challenge their understanding of the implications This is often not what the end user really wants, even if they are

asking for it

During requirements reviews, understand ALL the use-cases.

For example, what if there is a false alarm? Erasing all flash will

erase the OS and Bootloader, meaning the device becomes

useless and must be returned to manufacturer.

This simply can’t happen if system is deployed in the war zone.

When such use-cases are discussed with the customer, better

(aka easier to implement) solutions are usually found.


Self Destruct!Erasing the Code and Data

Unencrypted classified code and data must be erased before an enemy could get to it

Erase Times: NOR Flash: Minutes, needs power NAND Flash: Tens of seconds, needs power DRAM: Under 5 seconds, does not need power Crowbar DRAM: Milliseconds, does not need power

Simplest solution to meet requirements: put unencrypted classified code and data in crowbar DRAM only!

Only store strongly encrypted data in flash Decrypt data into DRAM only

Which encryption to use: a topic beyond the scope of this talk. This is one of the topics that is covered often by others


Self Destruct!Crowbar DRAM

Cheapest, fastest, and most reliable means of erasing

Even if power is removed from system, method works reliably

Simple Solution: DRAM hardware is modified so that a switch turns off Vcc to DRAM, and at the same time shorts Vcc to ground.

In cases of “false alarms” does not render the device into a brick


Self DestructErasing Flash

A software-only solution will not work. All it takes is removal of power to stop the erasing, at which time

other techniques can be used, such as removing flash from PCB then reading it independently.

Some secure flash drives and memories will latch internal controller Once erasing begins, it cannot be stopped. Removing power

only pauses the erase. Any attempt to power the chip to read will only resume the

erase operations. Caveats:

These are expensive flash memory options!

Understand the implications: what happens to device if feature is triggered?


Development Environment != Deployed SystemWhat do you need for software development?

It is typical for hardware requirements to drive the selection, design, and fabrication of hardware

From the first revision, hardware will incorporate all the required features needed in the end product, such as no user-accessible debug ports, no simple unsecure high-speed access, no keyboard or mouse, etc.

How is software developed for these systems?


Development Environment != Deployed System Software Team: Be part of Hardware Development!

The software team must be part of the requirement reviews and early design meetings.

Assume that every required feature will have a bug and needs to be debugged and fixed. Ensure that common tools used to troubleshoot such

errors are available, otherwise software costs will go through the roof, and high levels of reliability will be unachievable

Coordinate with hardware designers a plan for scaling back development features as the product matures and gets closer to production


Development Environment != Deployed System “Back Door” Mechanisms

Software engineers like to put in “back doors” to allow easy access to troubleshooting and debug utilities.

Such features, however, are frowned upon in military and avionic systems, because they are prime entry points for hacking, security breaches, and often contain unreliable code.

Presence of these mechanisms will usually cause a system to not meet certification requirements.

But absense of the mechanisms can greatly increase software maintenance costs.


Development Environment != Deployed System “Front Door” Mechanisms

Instead, consider “front door” mechanisms. Don’t try to hide these entry points, where they tend to be

unmanaged and not well secured. Rather, design them into the system from the start, to undergo

the same review scrutiny as any other feature Share the planned usage and lockouts with the customer and

end user. Often, they will want to use the features too during early development.

Include the necessary features to disable them when necessary, and securely re-enable them when they are needed.

Include a means to componentize the front door mechanisms, so that it can also be easily removed in final production versions.


Development Environment != Deployed System Debug Boards

Work with hardware designers to bring all the desired interfaces that customer does not want to a high-density debug connector e.g. debug serial ports, USB ports, JTAG, spare GPIOs, LEDs,

inter-chip communication signals, or anything else that would be useful to the software developer

Create a separate “debug board” that expands the signals into standard connectors Hardware team might balk at cost of such a board, which is

probably in the $5000 range However, existence of such a board can save 10 to 100 times

that cost in reduced software development effort Standardize the high-density debug connector, thus making

debug board reusable across multiple projects


Sorry, No Debug Allowed!Once unit is packaged, say bye to the debug board

The debug board helps reduce initial development, and can be used for troubleshooting any issues that can be replicated on an open system

Once the system is packaged, that key development interface is no longer available. Alternate mechanisms are needed to troubleshoot issues.


Sorry, No Debug Allowed!Log Files are the key!

Continuously collect data on every aspect of the system

Budget the amount of space needed to store log files early during system design

Store data on flash, for retrieval either by opening unit, or through an existing customer communications link

Do not store any classified data in the log files!


Sorry, No Debug Allowed!Not all log files are equal

Quality of log files is of extreme importance! It can eliminate the reliance on customer descriptions of the

problems. Often, their observations are inaccurate, or they provide their

interpretation of the problem.

With just the right information in a log file, many issues occurring in the field can be identified This is absolutely essential, especially in avionics Avoids the need to have to replicate the issue to debug it

Carefully review what goes into the log file A log file contains poor information if every time a problem

occurs, the log file only provides minimal clues to the actual problem


Sorry, No Debug Allowed!Concise Log Files

Memory reserved for log files can fill up quick Optimize memory usage as part of the design.

Be cryptic, that’s fine as long as there is easy way to decipher the log file “Average of velocity for past 30 seconds = 3.49231”

uses 40 bytes of memory

“av30=3.49231” uses 12 bytes of memory

Both the above provide equal amount of information, but second one is more concise, and log is searchable by unique acronym “av30”

Store in binary: 0xaa30 0x405F8202 Uses 6 bytes of memory 16-bit number as a key allows for 65536 separate types of log entries 32-bit floating point number is only 4 bytes of data

Create scripts to quickly search or filter desired data If this is not easy, then the uniqueness of keys is not adequate


Sorry, No Debug Allowed!What information should go into Log Files?

The more information stored, the more likely a single capture of a log file is sufficient to pinpoint a problem that occurred in field

Timestamp EVERYTHING! Microsecond resolution is desirable.

Millisecond resolution is essential.

Relative time, stored as a 16-bit or 32-bit number is

usually acceptable. Don’t use more than 4 bytes per

timestamp, or memory will fill up too quickly.



Periodically log the “state” of the system, including the following: Any code that is built as a finite state machine,

include the ‘mode’ or ‘state’ The current value and setting of every GPIO pin Current value of every key register Current value of any key global variables Current value of any sensor reading Current value of any controller output



Summary of every communication message that is sent or received: Includes Ethernet, WiFi, USB, SPI, etc.

Save key information, such as sender or receiver,

size of message, command or message type.

For short messages (i.e. a few bytes) include entire

message.

For longer messages, include first few data bytes.



Every Output I/O Device Driver Call: E.g. Instrument the stream interface, so that every

Write operation is captured

Save timestamp, device, port, and value written

Don’t need to log inputs or reads unless it is a message (see previous slide). For sensors, the state data logs the values periodically, instead of every time the device driver is called.



Every Interrupt Timestamp of entry into and exit from ISR

IRQ number

Running counter of how many interrupts of this type

received

If this is a shared IRQ, which device was interrupting



Every Context Switch Timestamp of entry into each thread

Thread ID

Timestamp of any entry into idle state or thread

The line of code to log this event needs to be in the operating system scheduler, right before the context switch occurs


Sorry, No Debug Allowed!How can we guarantee all software logs what it needs?

Log files are most effective when there are no “gaps” in information, resulting from one or more modules not logging like they should, or disabling the logs if they have the ability

When possible, put code to log information into kernel code, such as drivers or operating system functions, where they cannot be touched or changed by application programmers.

Logging is then enabled always for everything, with no ability to disable it.


Sorry, No Debug Allowed!Wow, isn’t that too much to log!

Estimate log file size based on typical usage.Here is an example State of system, 400 bytes, once per second Messages, 16 bytes per message, 500 messages per second I/O Driver Writes, 12 bytes per call, 200 times per second Interrupts, 12 bytes per interrupt, about 800 times per second Context Switches, 8 bytes per switch, 200 times per second

Above scenario = 400+12*800+12*200+8*200+16*500= 22 kBytes per second= 1.3 Mbytes per minute= 80 Mbytes per hour= 2 GBytes per day


Sorry, No Debug Allowed!Wow, isn’t that too much to log!

In past systems, this was a problem. In today’s system, this is easy. E.g. add a 2 GB micro-SD card solely for logging, and the full log for

any 24-hour period is possible. This is worst case

might only need to log for 8-hour days, and only a fraction of the above, as system might only really be in use 10% of the time during the day and idle or in a sleep mode rest of the day, thus bringing actual log requirements down to under 100 Mbytes per day, or 20 days of storage with the same 2GB card.

Use a circular buffer discard oldest data when maximum data usage has been reached.

Store multiple small files instead of one big file. E.g. start a new file every 5 minutes. Makes it easier to retrieve only partial logs within specific time periods.


Sorry, No Debug Allowed!Can’t afford the performance loss!

With proper optimization of log code, the logged data should not use more than 2% of the processing power.

E.g. Can write SD at 1MByte/sec, only need to write 22KByte/sec worst case, thus SD write is active less than 2% of the time.

Use a DRAM FIFO most logging is done in kernel space, where file access is not available Write to DRAM FIFO, overhead is negligible. Once per second, have a high-priority system-level thread read FIFO

and write to log file. Only systems that are 98% loaded or more won’t be able to support

this added overhead. If the system is that loaded, use a faster processor, or optimize one thread to reduce CPU usage by 2%

Consider the cost of NOT having this data when a problem occurs in a field unit that needs to be fixed. The 2% overhead is well justified!


Error Detection, Why Bother?What is right amount of error detection and handling?

What percentage of your code is error detection and handling?

Do you think that is too little? Too much? Just right?

How is more or less error detection and handling code justified?


Error Detection, Why Bother?Do I need to detect all errors?

Don’t waste time detecting errors for which the error handling is “blue screen of death” More often than not, this elevates simple errors into fatal errors

Only detect errors when it is possible to create a handler that recovers gracefully from the error Otherwise, let the logging mechanism simply log the information

as part of its normal duties

Error handling code must be part of the design documents Adding ad-hoc error handling often causes more harm than

good, because the code is rarely tested adequately, and only gets used when there are already problems in the system.


It Uses Too Much Power!Battery and Power Management

Mobile military applications and avionic applications have one thing in common: Power consumption is ALWAYS too much,

so reduce it more!

Aircraft need low power to reduce energy needs, and often just as important to reduce heat production within the closed body

Military applications need devices to run on battery for entire missions, as the ability to recharge is often at best a few hours each night


It Uses Too Much Power!Battery Life

Capacity is specified in mAh mAh = milli-Amp-Hours

E.g. 1400 mAh means the battery can drain 100mA

for 14 continuous hours

Or it can draw 1.4A for one hour

By measuring current when battery is discharging, can compute power consumption Power = Battery_Voltage * Discharge_Current


It Uses Too Much Power!Battery Life

Battery capacity only guaranteed for “new” batteries After a battery has been recharged a few dozen times, it starts to age As it ages, capacity decreases. The faster you discharge the batteries, typically the shorter its lifespan

Check battery datasheet. Some batteries may have capacity reduced by 50% in just a few hundred recharge cycles.

This means less than a year if recharged daily.


It Uses Too Much Power!Battery Level

For Alkaline batteries, voltage is a reasonable gauge

of remaining capacity

For Lithium type batteries, voltage level of battery

decreases relatively slowly thus voltage is not indicative of remaining life


It Uses Too Much Power!Battery Level

Li-Ion and Li-Polymer batteries typically use a “fuel gauge” to measure battery level

The gauge counts coulombs When charging, the count increases When discharging, the count decreases

Battery level is only accurate when the capacity is “calibrated” Once a battery starts aging, a battery level algorithm may require the

battery to be fully discharged, then reset the algorithm to indicate a new battery (even though it is old), then fully charge. This allows the new capacity to be re-learned.

Some drivers auto-calibrate anytime a full charge occurs Actual algorithm is hardware dependent

If battery level does not seem to be accurate, verify that drivers are properly reading the fuel gauge, and that there is some form of recalibration for aging batteries


It Uses Too Much Power!Reduce Power: Go For Idle

Idle systems can offer best power savings Sometimes, running FASTER can reduce power, by getting

back to idle quicker E.g. SPI driver with busy-wait loops. Run SPI clock as fast as

possible, to get back to idle more quickly For processor, running faster usually means more power

Slow down periodic threads Read an A/D 50 times per second instead of 100 times per second,

if application allows it Eliminate unnecessary services and interrupts

If system is not being used for 1 minute, capture every interrupt that occurs or every service that executes.

Ask WHY they execute, if system is idle? Coming out of idle often is costly to battery life.


It Uses Too Much Power!Reduce Power: Go For Idle

Idle Implementations Idle Thread

Lowest-priority thread has an empty while(1) loop. Anything that becomes active preempts this thread. Hardware-independent, thus easily portable to any processor Power-hungry; processor is still running at full tilt

CPU Idle Mode Most modern processors have an Idle Mode By installing hardware-dependent support to use this mode, significant

power savings can be achieved when idle Deep Idle

Turn off the processor oscillator when idle May take 100 usec extra to wakeup But the power savings can be significant Extra care needed if DMA running in background, e.g. for Display If turning off oscillator not an option because it breaks some devices,

consider just slowing down clock when idle


It Uses Too Much PowerSuspend and Shelf Modes

Suspend (aka Sleep) Turn off processor and most peripherals Keep DRAM powered State of each I/O pin controlled by software Enables resuming within a second

Shelf (aka Deep Sleep) Turn off processor and all peripherals Turn off DRAM State of each I/O pin is undefined Resuming requires rebooting Used to maintain enough power for a “soft” power-switch, and perhaps maintain

an RTC Hibernate is a form of shelf mode, except that the contents of DRAM are first

saved to the file system



Suspend Mode Power Software has significant impact on power consumption during

this mode, even though CPU not running

Devices not properly turned off or put into low power modes

may continue to draw power

I/O pins not placed in the right state may cause power drain

Shelf Mode Power Software has no control of this.

If this power consumption is too high, hardware engineers need

to investigate



Manage entry into suspend mode Use the xxx_power_off() of the driver

Start by saving the context In addition to saving context:

if the device has a low-power mode, enter it; or if the device has a power-enable, disable it if turning off power, hardware engineers should verify there is no backfeed, as that will

unnecessarily consume power during the sleep mode Ensure that any output signal to that device is placed into its inactive state

Note that for active-low signals that typically have a pull-up on the line, this means set the output HI

When exiting suspend mode Do opposite steps in xxx_power_on(), in reverse order

Exit low-power mode or re-enable power End by restoring the context

If device was powered off instead of a low-power mode It might be necessary to re-initialize the device, as any internal state would have been

lost.


It Uses Too Much PowerDevice Low-Power Modes

When a device is not needed, it should be placed into a low-power mode In theory, the xxx_power_off() function of the driver is

what should be called to turn off that device However, that only gets called during a suspend

operation For high-powered drivers, create an IOCTL to

execute the commands of the power-off function under user control

Create a second IOCTL to re-enable the power


It Uses Too Much PowerDuty Cycling

Duty cycling high-powered items can lower average power consumption

A 50% Duty Cycle of a device can reduce consumption in half for that device

This means ON half the time, then either OFF or in low-power mode the other half

Look at each individual hardware item to determine if there is an opportunity.

LEDs Analog Inputs Communications


Summary

Is that REALLY the Requirement? Review requirements early! Unrealistic requirements can often be negotiated

Self Destruct! Don’t erase everything, that is overkill! Put unencrypted classified data into DRAM

Development Environment != Deployed System Just because end requirements means no debug does not

mean prototypes can’t have it Create a debug board with favorite signals going to a high-

density connector


Summary

Sorry, no debug I/O allowed, but debug it quick! Continuously generate log files

Design the content to manage memory needs

Error detection: Why bother? Only detect errors that can be handled gracefully

No blue screens of death!

It Uses Too Much Power! Understand how battery power really works

Leverage idle, sleep modes, and duty cycling

(c) 2011 dave stewart - inhand electronics - systems conference boston, sept 20111 unique software...

Documents