Scalable electronics driving autonomous vehicle technologies

Fernando Mujica, Ph.D.Director, Autonomous Vehicles R&DKilby LabsTexas Instruments

Scalable electronics driving autonomous vehicle technologies 2 April 2014

TI’s advanced electronic solutions and research help auto makers build scalable, evolving systems for advanced driver assistance and autonomous vehicle operation.

Vehicles capable of autonomous operation are in the early stages of development today for use on the roads in the near future. To move self-driving cars from vision to reality, auto manufacturers depend on enabling electronic technologies for sensing, sensor fusion, communications, high-performance processing and other functions. Autonomous vehicle control systems will have to be scalable to accommodate a multi-year evolution as car models change and data loads increase with additional features and improved sensors. Innovations that can result in scalability include the use of distributed processing and localized sensor fusion.

Texas Instruments understands these requirements, based on its automotive expertise and innovative solutions that address the full range of automotive systems needs. TI also conducts its own autonomous vehicle research to help car makers find scalable solutions for systems that will perform well today and in the future.

One of the most exciting technology advances

today is development of automobiles that can

control themselves in certain situations and,

ultimately, will drive themselves with minimal or

no human assistance. Auto makers continually

announce their plans for introducing automated

features in upcoming models, and the industry

estimates that fully self-driving vehicles will be

available in less than a decade. Some vehicles that

are currently available offer automated monitoring

and warning features. Some are even capable of

self-control in certain limited situations. Looking

ahead, semi-autonomous, then fully autonomous

vehicles will be phased in and driven on the

roads along with traditional vehicles. Eventually,

and sooner than we can easily realize, all new

automobiles will be able to drive themselves,

changing our lives almost as dramatically as the

earliest cars impacted the lives of our ancestors.

These innovations are the result of advanced

electronics that can sense, recognize, decide and

act upon changes in the road environment. Auto

makers, as they introduce new automated features,

will face the usual factors that affect decisions about

electronic systems and components: performance,

size, cost, power requirements, reliability, availability

and support. Add to these the importance of

scalability, since systems will have to evolve from

year to year, along with car model changes that

bring feature additions and improvements in

sensing technology. To stay on track in offering

new automated capabilities, car makers will rely

on technology suppliers whose scalable solutions

offer the best options for balancing among all these

requirements in the overall system design.

Scalable electronics driving autonomous vehicle technologies 3 April 2014

Benefits, timing of autonomous vehicle operation

Three words express the advantages offered

by vehicle automation: safety, convenience and

efficiency. Autonomous vehicle control will aid in

eliminating many of the human errors that cause

most accidents, helping to save lives, reduce

injuries and minimize property damage. In addition,

cars that drive themselves can chauffeur children,

the elderly and disabled, free drivers to do other

things while traveling, or even appear where and

when needed without a human driver. Autonomous

operation will also be more fuel-efficient and allow

more cars to travel safely together on busy roads,

saving on energy and infrastructure costs.

Of all the benefits, safety has been the top

priority and is supported by many of the initial

automated feature offerings. Termed Advanced

Driver Assistance Systems (ADAS), these features

are designed to help drivers avoid mistakes and,

therefore, save lives in the near term. ADAS features

will also serve as important elements of fully

autonomous operation in the future.

The introduction of automated driving features

will happen in phases and with increasing

levels of autonomy. Using the National Highway

Transportation Safety Administration definitions, these levels include:

• Level0 (no automation): In these vehicles, the driver is in

full control at all times.

• Level1 (function-specific automation): The vehicle takes

control of one more more vehicle functions, such as

dynamic stability control systems. Most modern vehicles

fall into this.

• Level2 (combined function automation): This involves

automation of at least two primary functions. For example,

some high-end vehicles offer active cruise control and

lane keeping, working in conjunction, which would classify

them as level two.

• Level3 (limited self-driving automation): The vehicle is

capable of full self-driving operation in certain conditions,

and the driver is expected to be available to take over

control if needed.

• Level4 (full self-driving automation): The vehicle is in

full control at all times and is capable of operation, even

without a driver present.

Because of the initial relatively high cost of

automated driving technologies, these features

are introduced in higher-end vehicles first, but

are expected to migrate soon to mid-range

and economy cars. More about the phases of

introduction can be found in the TI white paper

Making Cars Safer Through Technology Innovation,

which also discusses many of the challenges facing

the automotive industry and the larger society as we

transition to fully autonomous driving.

Advanced ICs provide the enabling technology

Automated features, during every phase of

introduction, are based on many components,

including electronic sensors that capture information

about the car’s environment; sensor analog front

end (AFE) devices that convert real-world data

from analog to digital data; integrated circuits

Scalable electronics driving autonomous vehicle technologies 4 April 2014

(ICs) for communications; and high-performance

microprocessors that analyze the massive amount of

sensor data, extract high-level meaning, and make

decisions about what the vehicle should do. Add

microcontrollers (MCUs) to activate and control brakes,

steering and other mechanical functions, plus power

management devices for all circuitry, and it becomes

evident how much autonomous vehicle operation is

dependent on advanced electronic solutions.

When selecting electronics to fulfill various functions,

auto system designers must not only consider

performance and price, but also how well the

components fit into a scalable system. As the phased

introduction of vehicle automation indicates, the more

completely autonomous systems that appear later on

will be built using previously introduced automated

functions. For instance, surround-view cameras

that are used today for park assist will be integrated

eventually into the overall sensing and control system

for the final phase of fully autonomous driving. Along

the way, surround-view cameras will be joined by

radar and 3D scanning lidar (LIght Detection and

Ranging) sensors that provide complementary

information about what is around the vehicle. In

addition to offering complementary 3D information

to cameras, radar and lidar sensors are more robust

under severe weather conditions. Thus, over time,

the automated sensing and control system will grow

more complex, integrated and effective.

The increased use of complementary sensors (such

as camera, radar and lidar) shows one reason that

electronic systems should be scalable, because

scalability easily allows the addition of new sensors

to the system. Another reason is that the individual

sensors will improve over time, and will require more

communications and processing bandwidth.

Cameras provide an obvious example to address

the growing need for bandwidth. Cameras also play

a pivotal role in sensing systems because today they

are the most effective sensors for analyzing data

meant for human consumption. Cameras provide

an enormous data stream for communications and

processing, and future increases in image resolution

will magnify the load. To handle these load increases

effectively, the system must be designed from the

start to support rescaling.

Processing sensor inputs

In practical terms, system scaling depends on

where and how the various levels of processing are

performed. Figure 1 shows a functional view of the

data flow in a fully equipped sensing and control

system for an autonomous vehicle. At the left are

the input sensors, including global positioning

(GPS), inertial measurement unit (IMU), cameras,

lidar, radar and ultrasound. Each sensor has a

certain amount of dedicated sensor processing that

processes raw data in order to create an object

representation that can be used by the next stage in

a hierarchical fusion system.

The conceptual view shown in the figure

comprehends different types of sensor fusion

occurring at various levels. For instance, raw data

from a pair of cameras can be fused to extract

depth information, a process known as stereo

vision. Likewise, data from sensors of different

modalities, but with overlapping fields of view, can

be fused locally to improve the tasks of object

detection and classification.

Object representation provided by on-board

sensors, whether originated from a single sensor or

via fusion of two or more sensors, is combined with

additional information from nearby vehicles and the

infrastructure itself. This information comes from

dedicated short-range communication (DSRC), also

referred to as vehicle-to-vehicle (V2V) and vehicle-

Scalable electronics driving autonomous vehicle technologies 5 April 2014

to-infrastructure (V2I) communications. On-board

maps and associated cloud-based systems offer

additional inputs via cellular communications.

The outputs from all the sensor blocks are used to

produce a 3D map of the environment around the

vehicle. The map includes curbs and lane markers,

vehicles, pedestrians, street signs and traffic lights,

the car’s position in a larger map of the area and

other items that must be recognized for safe driving.

This information is used by an “action engine,” which

serves as the decision maker for the entire system.

The action engine determines what the car needs

to do and sends activation signals to the lock-step,

dual-core MCUs controlling the car’s mechanical

functions and messages to the driver. Other inputs

come from sensors within the car that monitor the

state of the driver, in case there is a need for an

emergency override of the rest of the system.

Finally, it is important to inform the driver

visually about what the car “understands” of its

environment. Displays that help the driver visualize

the car and its environment can warn about road

conditions and play a role in gaining acceptance of

new technology. For instance, when drivers can see

a 3D map that the vehicle uses for its operations,

they will become more confident about the vehicle’s

automated control, and begin to rely on it.

Algorithms and system scaling

With its heavy reliance on cameras, radar, lidar and

other sensors, autonomous vehicle control requires

a great deal of high-performance processing, which

by nature is heterogeneous. Low-level sensor

processing, which handles massive amounts of

input data, tends to use relatively simple repetitive

algorithms that operate in parallel. High-level

Cameras

Radars

Sensor Processing

Sensor Processing

Sensor Fusion

3D Scanning Lidars

Ultrasoundsensors

Sensor Processing

Sensor Processing

Action Engine

Vehicle Controls- Brake/acc- Steering- etc.

Visualization/DisplaySub-system

Raw data Object parameters- Time stamp- Dimensions- Position/velocity

3D Map Actions- Do nothing- Warn- Complement- Control

Compressed data

V2V / V2Icomm.

Sense Understand Act

GPSIMS

“Maps”a priori info

Driver state

Autonomous vehicle platform: a functional diagram

Figure 1. A functional view of the data flow in an autonomous car’s sensing and control system.

Scalable electronics driving autonomous vehicle technologies 6 April 2014

fusion processing has comparatively little data

but complicated algorithms. Various algorithms

are optimally implemented by different processing

architectures, including SIMD (single-instruction,

multiple data), VLIW (very long instruction word),

and RISC (reduced instruction set computing)

processor types. These architectures may also

be aided in performing specific functions by hard-

coded hardware accelerators.

The multi-level, heterogeneous processing demanded

by autonomous vehicle operation is well supported

by hierarchical distributed processing, in which sensor

processing is performed separately from the final-stage

fusion and action engine. Figure 2 shows an example

of a car with the sensor processing fully distributed near

the sensors at the vehicle periphery. All of these feed

outputs to the final-stage fusion and action processor

that decides what the car should do and issues

commands to the vehicle’s mechanical parts.

The alternative to the distributed processing scheme

shown is a fully centralized architecture, where all the

processing takes place in a single multicore unit.

A system based on fully centralized processing

depends on the high-speed transfer of vast amounts

of data from all the sensors. Communications and

processing not only have to be adequate for the

current year’s model, but must build in a great deal of

headroom in the initial design to accommodate sensor

additions and improvements in later years. By contrast,

in a distributed architecture, the sensor resolution can

increase without significantly affecting the bandwidth

requirements to communicate the resulting object

representation.

Sensor fusion—processing that forms a composite

understanding from two or more complementary

sensors—can take place in either a distributed or

centralized system. Usually the sensors being fused

are close together, perhaps housed in the same

unit, so that there is minimal need for high-speed

communications over a wide area. The addition of

radar and lidar can actually diminish the requirements

for camera processing because radar and lidar provide

richer, more accurate 3D information that facilitates the

detection and classification of objects.

Embedded sensor processing / interface tradeoffs

Figure 2. In the future autonomous car, sensor processing will occur throughout the car’s periphery to send signals and commands to the car’s mechanical parts.

Scalable electronics driving autonomous vehicle technologies 7 April 2014

TI research and development for autonomous vehicles

TI has a decades-long relationship with the

automotive industry, giving the company valuable

expertise to create the technology needed by

auto manufacturers and unique foresight into the

challenges ahead. Recognizing that ADAS and self-

driving features are evolving rapidly, the company

devotes a significant amount of effort to identify

and develop solutions to enable autonomous

vehicles. TI development includes all aspects of

sensing subsystems and control integration, with

methodologies to analyze data in real-time.

A great deal of TI research is directed at extending

the technology that will be used in autonomous

vehicles. The importance of lidar as a vehicle sensing

technology is undisputed, but to date lidar has been

too large and costly to consider for wide adoption—

a problem TI proposes to address. Other areas of

research that will prove beneficial to car makers

include ultrasound, high-speed data communications

for sensor fusion applications, and enhancements

within the vehicle sensing and user interfaces.

TI development also continues across the full

spectrum of analog and embedded processing

ICs required to support automotive electronics,

including autonomous vehicle operation. An extensive

portfolio of power management ICs, sensor signal

conditioning, interfaces and transceivers support the

signal chain and power supply. Besides supplying

sensor technology for the outside of the car, TI offers

ultrasound sensors that can be used in the cabin,

and inductive and capacitive sensors in the steering

wheel, to provide information about the state of the

driver. TI’s DLP® wide field of view heads-up display

technology is well suited to provide display capabilities

that can help drivers see what the car “knows” about

the road and keep them engaged while monitoring

vehicle operation, if needed.

In addition, advanced, heterogeneous processing,

along with the necessary foundation software,

promote rapid algorithm and application

development. For example, TI’s TDA2x system-

on-chip (SoC) technology, spans this range with a

programmable Vision AccelerationPac containing

Embedded Vision Engines (EVEs) that are

specialized for handling the massive data of video

systems, ISP (image signal processor) for camera

preprocessing, DSP (digital signal processor)

for more general signal processing, and RISC

options that include several ARM® processors. In

addition to its own software frameworks, TI is a

key contributor to the Khronos OpenVX standard

for computer vision acceleration that addresses the

need for low-power, high-performance processing

on embedded heterogeneous processors. The

system solution is scalable throughout TI’s SoC

portfolio, including MCUs, multicore DSPs and

heterogeneous vision processors.

Today’s investment in future transportation

The stakes are high for auto makers worldwide,

as they race to see who can introduce the

most successful implementations of ADAS and

autonomous vehicle operation. The phased

introduction of automated features, and ongoing

improvements in sensors, suggest that the

electronic systems will have to be designed for

scalability. Other factors influencing systems

design are the fusion of complementary sensor

data to provide a better 3D map of the car and

its environment, the availability of low-cost lidar

sensors, and other improvements in enabling

hardware and software.

Summary

TI recognizes the challenges that auto manufacturers face as they strive to make self-driving cars a reality. The company invests a large amount of research in learning what is involved in autonomous vehicle operation in order to design the right solutions for auto makers’ needs. This research complements TI’s extensive portfolio of leading solutions, along with the company’s strengths in worldwide manufacturing and support. TI technology will continue to play an important role in helping automotive manufacturers implement ADAS safety features that are being phased in today, and helping them to enable the self-driving cars of tomorrow.

For more information

•Visitourwebsite:www.ti.com/corp-ino-auto-wp-lp

•Readourrelatedwhitepaperonautomotivesafety:www.ti.com/corp-ino-auto-wp-mc

•WatchourADASwhiteboardvideo:www.ti.com/corp-ino-auto-wp-v

ImportantNotice: The products and services of Texas Instruments Incorporated and its subsidiaries described herein are sold subject to TI’s standard terms and conditions of sale. Customers are advised to obtain the most current and complete information about TI products and services before placing orders. TI assumes no liability for applications assistance, customer’s applications or product designs, software performance, or infringement of patents. The publication of information regarding any other company’s products or services does not constitute TI’s approval, warranty or endorsement thereof.

© 2014 Texas Instruments Incorporated SSZY010

The platform bar is a trademark of Texas Instruments.All other trademarks are the property of their respective owners.

IMPORTANT NOTICETexas Instruments Incorporated and its subsidiaries (TI) reserve the right to make corrections, enhancements, improvements and otherchanges to its semiconductor products and services per JESD46, latest issue, and to discontinue any product or service per JESD48, latestissue. Buyers should obtain the latest relevant information before placing orders and should verify that such information is current andcomplete. All semiconductor products (also referred to herein as “components”) are sold subject to TI’s terms and conditions of salesupplied at the time of order acknowledgment.TI warrants performance of its components to the specifications applicable at the time of sale, in accordance with the warranty in TI’s termsand conditions of sale of semiconductor products. Testing and other quality control techniques are used to the extent TI deems necessaryto support this warranty. Except where mandated by applicable law, testing of all parameters of each component is not necessarilyperformed.TI assumes no liability for applications assistance or the design of Buyers’ products. Buyers are responsible for their products andapplications using TI components. To minimize the risks associated with Buyers’ products and applications, Buyers should provideadequate design and operating safeguards.TI does not warrant or represent that any license, either express or implied, is granted under any patent right, copyright, mask work right, orother intellectual property right relating to any combination, machine, or process in which TI components or services are used. Informationpublished by TI regarding third-party products or services does not constitute a license to use such products or services or a warranty orendorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property of thethird party, or a license from TI under the patents or other intellectual property of TI.Reproduction of significant portions of TI information in TI data books or data sheets is permissible only if reproduction is without alterationand is accompanied by all associated warranties, conditions, limitations, and notices. TI is not responsible or liable for such altereddocumentation. Information of third parties may be subject to additional restrictions.Resale of TI components or services with statements different from or beyond the parameters stated by TI for that component or servicevoids all express and any implied warranties for the associated TI component or service and is an unfair and deceptive business practice.TI is not responsible or liable for any such statements.Buyer acknowledges and agrees that it is solely responsible for compliance with all legal, regulatory and safety-related requirementsconcerning its products, and any use of TI components in its applications, notwithstanding any applications-related information or supportthat may be provided by TI. Buyer represents and agrees that it has all the necessary expertise to create and implement safeguards whichanticipate dangerous consequences of failures, monitor failures and their consequences, lessen the likelihood of failures that might causeharm and take appropriate remedial actions. Buyer will fully indemnify TI and its representatives against any damages arising out of the useof any TI components in safety-critical applications.In some cases, TI components may be promoted specifically to facilitate safety-related applications. With such components, TI’s goal is tohelp enable customers to design and create their own end-product solutions that meet applicable functional safety standards andrequirements. Nonetheless, such components are subject to these terms.No TI components are authorized for use in FDA Class III (or similar life-critical medical equipment) unless authorized officers of the partieshave executed a special agreement specifically governing such use.Only those TI components which TI has specifically designated as military grade or “enhanced plastic” are designed and intended for use inmilitary/aerospace applications or environments. Buyer acknowledges and agrees that any military or aerospace use of TI componentswhich have not been so designated is solely at the Buyer's risk, and that Buyer is solely responsible for compliance with all legal andregulatory requirements in connection with such use.TI has specifically designated certain components as meeting ISO/TS16949 requirements, mainly for automotive use. In any case of use ofnon-designated products, TI will not be responsible for any failure to meet ISO/TS16949.Products ApplicationsAudio www.ti.com/audio Automotive and Transportation www.ti.com/automotiveAmplifiers amplifier.ti.com Communications and Telecom www.ti.com/communicationsData Converters dataconverter.ti.com Computers and Peripherals www.ti.com/computersDLP® Products www.dlp.com Consumer Electronics www.ti.com/consumer-appsDSP dsp.ti.com Energy and Lighting www.ti.com/energyClocks and Timers www.ti.com/clocks Industrial www.ti.com/industrialInterface interface.ti.com Medical www.ti.com/medicalLogic logic.ti.com Security www.ti.com/securityPower Mgmt power.ti.com Space, Avionics and Defense www.ti.com/space-avionics-defenseMicrocontrollers microcontroller.ti.com Video and Imaging www.ti.com/videoRFID www.ti-rfid.comOMAP Applications Processors www.ti.com/omap TI E2E Community e2e.ti.comWireless Connectivity www.ti.com/wirelessconnectivity

Mailing Address: Texas Instruments, Post Office Box 655303, Dallas, Texas 75265Copyright © 2014, Texas Instruments Incorporated