8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 1/15
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 2/15
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 3/15
3.1. Information flow
Intersection related applications comprise a special
type of V2X applications which are perfectly convenient
for the introduction phase of V2X communication systems.
That is the case because of the simplicity of the informa-
tion flow. If DSRC is used, the information flow consists
of a unidirectional broadcast of single-hop transmissions
from the intersection’s IRSs to vehicle’s IVSs (see Fig. 1).
If cellular mobile communication is used, IVSs build up cel-
lular mobile communication connections and download
the required information about near intersections from
an ICS (see Fig. 2). However, this does not change the infor-
mation flow within vehicles substantially.
The relevant information to realize intersection related
V2X applications comprise the intersection topology per
intersection (for example see Fig. 12) and the signal state
timing per each lane of each intersection (for example see
Fig. 11) [18–20]. Both are represented by messages which
are introduced in Section 2.3. The intersection topologies
are modeled by an intersection ID, vectorized driving lanes,
driving lane numbers, suitable connection points, stop
lines, and more. The signal state timing data are modeled
by an intersection ID, lane numbers, minimum time to
change, maximum time to change, likely time to change,
and more. Both message types are broadcasted periodically
by intersection’s IRSs or could be downloaded from ICSs.
Although the vehicle’s Electronic Control Units (ECUs)
provide relevant information to realize intersection related
V2X applications which is currently already exchanged by
vehicle internal bus systems like the CAN-Bus. Typical rel-
evant vehicle internal information are among others: vehi-
cle speed, single wheel speed, engine speed, yaw rate,
direction indicator status, or fog light status. So, the access
of those data could easily be realized by vehicle manufac-
tures and, thus, be used by V2X implementations.
Finally, an IVS collects all those information from its
vehicle and its environment and uses it to realize the appli-
cation’s functionalities. In addition, the applications could
use additional facilities of vehicles (e.g. HMI or signaling
tones) to interact with the drivers. Examples of those
applications are described in the following Section 3.2.
3.2. Examples of applications
According to [3,9,21], typical intersection related appli-
cations are among other:
Green-light Optimal Speed Advisory (GLOSA) ‘‘Drivers
receive a recommendation in order to hit the next traf-
fic lights in green phase and to avoid waste accelera-
tion.’’ [9]
Red-light Violation Warning (RVW) ‘‘Warns drivers
when they are going to violate a red traffic light signal.’’
[9]
Typically, the warnings occur in a double-staged
approach. E.g., the first stage realizes a visual or audible
signaling; whereas, the second stage realizes a short
jerky braking maneuver if the driver shows no reaction
on the first signaling.
Traffic Light Assistant (TLA) If the vehicle has to stop
because of the traffic light signaling, the vehicle shows
the remaining waiting time of the corresponding signal-
ing group to the driver [21].
4. Test needs
Testing software intensive systems is a complex and
permanent task throughout the software life cycle and
should therefore be done very efficiently and effectively.
Moreover, the test process’ quality has significant influence
on the resulting products’ quality. Since no V2X applica-
tions have already been brought into market so far, no suf-
ficient experiences in testing of those novel systems are
available. Therefore, the test needs of intersection related
V2X applications are analyzed from the literature’s point
of view. Finally, based on these analysis requirements on
the test system are derived.
4.1. Test levels and test types
According to [22], test levels ordinary comprise compo-
nent testing (unit testing), integration testing, system test-
ing, and acceptance testing; whereas, the transitions are
smooth (see Fig. 3). As component testing and low-level
integration testing targets low-level functional testing
and low-level non-functional testing close to software
implementations, test methodologies for these parts of test
levels are already available (see [22–25]). So, testing
activities closely related to software implementations of V2X applications should be organized analog to common
IRS
IVS ECU
<<flow>> <<flow>>IVS ECUIRS 1..*110..*
vehicle internDSRC
Fig. 1. information flow of DSRC based intersection related V2Xapplications.
IRS
IVS ECU
<<flow>> <<flow>>IVS ECUIRS 1..*10..*0..*ICS
<<flow>> 11..*
vehicle interncellular mobilebackbone
ICS
Fig. 2. Information flow of cellular mobile based intersection related V2X
applications.
3156 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 4/15
software tests. To avoid contradictions, the test levels and
test types are defined as follows:
component testing ‘‘Testing of individual hardware
or software components or
groups of related components.’’
[26];
integration testing ‘‘Testing performed to expose
defects in the interfaces and in
the interactions between inte-
grated components or systems.
See also component integration
testing, system integration test-
ing.’’ [27];
system testing ‘‘The process of testing an inte-
grated system to verify that it
meets specified requirements.’’
[28];
acceptance testing ‘‘Formal testing with respect to
user needs, requirements, and
business processes conducted
to determine whether or not a
system satisfies the acceptance
criteria and to enable the user,
customers or other authorized
entity to determine whether or
not to accept the system.’’ [27]
after [26];
functional testing ‘‘Testing based on an analysis of
the specification of the function-
ality of a component or system.’’
[27];
non-functional testing ‘‘Testing the attributes of a com-
ponent or system that do not
relate to functionality, e.g. reli-
ability, efficiency, usability,
maintainability and portability.’’
[27].
In contrast, testing activities for high-level integration
testing and system testing, namely, high-level functional
testing and non-functional testing could not be handled
adequately with available test technologies and methodol-
ogies. Except for the hardware equipment required to
emulate the V2X communication partners and vehicle’s
ECUs, a test system has to be able to handle diverse
requirements in order to address the specific needs of
V2X applications. Section 4.3 establishes a proposal for
those requirements.
Additionally, acceptance testing is another wide area of
testing software intensive systems; especially, for cooper-
ative V2X applications. For intersection related V2X appli-
cations, acceptance testing has to target among others: the
driver acceptance, influences on a road network’s traffic
throughput, or the traffic flow smoothness. So, this area
of testing should be considered separately.
This also applies for testing safety, security, and privacy
aspects of ITS systems in general and, thus, also for inter-
section related V2X applications. Since ITS systems include
various communication interfaces, there might be much
potential for security violation attacks. Moreover, vehicle
take part in road traffic and, thus, security violation attacks
might induce serious road accidents. Therefore, also for
safety, security, and privacy, separate test methodologies
should be explored and applied.
In summary, testing high-level functional and non-
functional aspects of intersection related V2X applications
opens a differentiated and unsolved area. So, this paper fo-
cuses on these two aspects of testing.
4.2. Testing of system characteristics
As described in Section 2.2, the software architecture of
V2X systems will most probably consist of a stack architec-
ture. Here, multiple applications and underlaying run-time
platform components (e.g. middleware or the operating
system) will run on one system simultaneously. So, the
workload of those systems varies over time and thus the
execution time of algorithms and the response-time to
interrupts is non-deterministic. The relevant effects which
lead to non-deterministic systems (e.g., scheduling algo-
rithms, concurrency) are described in [29]. Moreover, the
whole system might be distributed over multiple ECUs
which are connected over communication networks with
non-deterministic timing properties (e.g. CAN-Bus). So,
the timing characteristic of intersection related V2X sys-
tems should be targeted as non-deterministic. For that rea-
son, functional and non-functional testing which targets
the timing of the systems should be repeated several times
and evaluation statistically.
Moreover, different environmental and vehicle internal
information sources and information flows of IVSs might
suffer from communication channel disturbances, mea-
surement and intermediate calculation inaccuracies, or
other negative effects. Since it is impossible to resolve
the effects of input signal variations to the application’s
outputs, statistic test methodologies have to be applied.
By using randomized algorithms as described in [30] for
test input signal modifications and statistical evaluations
of resulting application’s reactions and outputs, the former
problem could be addressed.
Finally, for testing activities targeted to non-functional
aspects of the system, at least performance testing, load
testing, stress testing, and reliability testing should there-fore be taken into account [22].
component testing
integration testing
system testing
acceptance testingcheck system behavior
against consumer’s
requirements
functional and
non-functional testing
low-level
high-level
Fig. 3. Test levels with test types.
S. Röglinger/ Computer Networks 55 (2011) 3154–3168 3157
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 5/15
4.3. Requirements on the test systems
In this subsection, a proposal for requirements on test
strategies and test architectures for intersection related
V2X systems are presented. These requirements are de-
rived from the intersection related V2X’s architecture (Sec-
tion 2) and the test needs (Section 4).
Requirement 1 Emulation of the specified behavior of
cooperation partners who are con-
nected via DSRC.
Requirement 2 Emulation of influences of wireless
communication on the DSRC communi-
cation. Suitable influences are among
others radio interferences, package col-
lisions, communication channel conges-
tion, fading, or shading [31,32]. Those
influences show according to [33,34]
effects on package loss rates and statis-
tical distributions of package transmis-
sion delays and might be caused by
communication of other vehicles or by
environmental objects like buildings or
trees.
Requirement 3 Emulation of the specified behavior of
cooperation partners who are connected
via cellular mobile communication.
Requirement 4 Emulation of influences of cellular
mobile networks on the communication
between ICSs and IVSs. Suitable influ-
ences are among other: network cover-
age, and available local field strength.
Those influences show effects on: prob-
ability of connections loss, round trip
time, channel throughput, and others.
Requirement 5 Emulation of the specified behaviors of
other ECUs which are also located
inside the vehicle.
Requirement 6 Emulation of the vehicle’s self localiza-
tion facilities. In most cases, GPS and
odometer based techniques are used.
Requirement 7 Test runs have to be reproducible
exactly which addresses synchroniza-
tion problems. Those problems may
appear by reapplying test runs on dis-
tributed systems.
Requirement 8 It must be possible to modify the test
input data in order to adjust their qual-
ity (e.g., add noise or reject messages).
Requirement 9 It must be possible to arbitrary combine
various test input data to stress differ-
ent test cases.
Requirement 10 It must be possible to build test verdicts
based on statistic evaluation of test
runs.
To avoid contradictions, the term emulation is defined
by ‘‘technique of one machine obtaining the same results
as another’’ [35] whereas simulation is defined by ‘‘the
technique of representing the real world by a computerprogram; a simulation should imitate the internal pro-
cesses and not merely the results of the thing being simu-
lated’’ [35].
Summing up, test strategies and test architectures for
intersection related V2X applications have to fulfill all or
at least a subset of the above established requirements.
In case that just a subset is fulfilled, not all test needs could
be satisfied. E.g., the test approach, presented in Section 5,
uses a two-layered strategy to use real-world traces for
test input stimulation, whereas, the first layer realizes
not all requirements.
5. Approach
In this section, an approach on how to test intersection
related V2X applications is presented. The approach
emphasizes the points identified in Section 4:
test levels: high-level integration testing and system
testing,
test types: high-level functional and non-functional
testing, system characteristics: test input modification and sta-
tistical test output evaluation.
Moreover, the approach’s design fulfill the require-
ments established in Section 4.3 by adapting a channel
and stream concept.
5.1. Identification of system under test
For testing functional and non-functional aspects of dis-
tributed systems, like intersection related V2X applica-
tions, all involved components have to be considered.However, as described in Section 3, most of the complexity
of intersection related V2X applications is located in IVSs.
Hence, IRSs are also completely new products; those types
of stations should also be tested. However, the complexity
of the IRS’s application implementations does not reach
the difficulty of the IVS’s application implementations. Fur-
thermore, ECUs are already well tested and not modified
because they are only information sources. So, they do
not need extra testing in terms of testing intersection
related V2X applications. For all those reasons, most
testing activities have to take IVSs into account. Therefore,
IVSs are treated as System Under Test (SUT) in the test
methodology.
5.2. Test strategy
The test strategy comprises the following two points:
stimulations of SUTs to stress test cases, and evaluation
of test results. The realization of these is shown for inter-
section related V2X applications in the following subsec-
tions. Since both points are closely related to the channel
and stream concept, the next subsection describes how
this concept could be applied for intersection related V2X
applications. The following subsection treats stimulation
and evaluation techniques, and the last subsection handlesdefinitions of assessment strategies.
3158 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 6/15
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 7/15
Fig. 5 depicts how input streams and their modifica-
tions could be applied to SUTs. The additional timing mod-
ifications T cause positive or negative delays of the
modified input streams to create new test cases by using
the same input streams (requirement 9 of Section 4.3).
Thus, a set of T s could be seen as a configuration of differ-
ent test cases using the same input streams. However, this
construct is not in scope of this section but will be pickedup in Section 5.3.
Additionally, it could be seen in Fig. 5 and (3) that there
is a distinction between conventional and system-internal
channels. On the one hand, conventional channels are also
part of the final product. So, a is set to the final product’s
number of input channels and c is set to the number of re-
corded final product’s output channels. On the other hand,
system-internal channels are not part of the final product
but may be necessary to adjudicate a test verdict.
To construct test cases, the morphologic box whose first
column treats the test case’s test type is used. For functional
testing , only conventional input streams and sometimes
the flow of internal communication interfaces is relevant.So, b is set to 0 and d is set to the number of additionally,
probed internal communication channels. In contrast, for
non-functional testing , also flows of system-internal output
streams like:
CPU load,
memory usage,
network I/O, or
event count
are important. Additionally, to have influence on system-
internal parameters (e.g. additional CPU load) is relevant
to target the test goals: performance testing, load testing,stress testing, and reliability testing as already postulated
in Section 4.2. So, b is set to the number of modified sys-
tem-internal parameters (e.g. additional CPU load for stress
testing) and d is set to thenumber of relevant,recorded sys-
tem-internal output streams (e.g. resulting CPU load).
At second, it has to be decided if influences of modifica-
tions of input streams should be taken into account. If influ-
ences of input stream modifications are not targeted, all sets
M con,i = {} with i = 1, . . . , a are empty. Otherwise, the sets
M con,i contain potential modifications of the corresponding
input streams I con,i (see Fig. 5). Since influences of those in-
put stream modifications to the system’s outputs could not
be resolved analytically, randomized algorithms for deter-
mining the modifications may be applied [30].
Moreover, there are these two types of input streams
regarding their variation over time:
input streams which are static within one test run (e.g.
intersection topologies), and
input streams which are not static within one test run
(e.g. time to green/red of traffic light signal timings).
This effects the modification of input streams as stated
so far. For the static type of input streams, a single modifi-
cation of the input stream (e.g. shift) is chosen per test run.
For the non-static type of input streams, modifications of
input streams (e.g. noise, glitches, drift, message loss, or
additional messages) have to be defined as mapping over
a non-modified input stream I = P N , p 2 P , and time t 2 R
such that f : R P ! P with p 0 = f (t , p).
Before modifications of input streams are determined, it
has to be chosen, if the modifications are inside, on, or out of
the specified tolerances. In the first case, all modifications
are inside their ranges of tolerances. In the second case,
at least one modification is chosen exactly on its tolerance
boundaries. In the last case, at least one modification is out
of its specified tolerances.
Next, V2X applications have to deal with many non-
deterministic effects like communication channel distur-
bances, ego-positioning inaccuracies (e.g. limited GPS
accuracy), measurement inaccuracies, or non-determinis-
tic run-time platforms. Those effects might cause varia-
tions of output streams even when reapplying the same
test case. For gathering those output variations, it is sug-
gested to reapply a test case’s input streams several times
in the exactly same fashion. The index set Ind provides the
possibility to distinguish the different test runs.
Additionally, the evaluation of output streams could
either be done by humans reviewing the raw output
streams or by using metrics and assessment strategies. Ta-
bles 1 and 2 list not exhaustive lists of metrics and assess-
ment strategies for the evaluation of the correctness of the
stream’s values and the evaluation of the correctness of the
stream’s timing for single and multiple test runs. Since
streams combine the properties used in the tables, the
evaluation of streams requires a combination of the tables’
metrics and assessment strategies.
5.2.3. Definition of assessment strategies
By using the previous definitions of messages (1),
streams (2), and test cases (3), it is possible to define
assessment strategies formally. To show how this ap-
proach works, the approach is demonstrated with the fol-
lowing two examples:
(1) evaluation of distribution of points in time where
messages appear, and
(2) evaluation of influences of input data tolerances.
For both examples, the RVW application is used to
demonstrate the definition of assessment strategies. In
addition, these examples are picked up in Section 6 toshow their feasibility by a proof of concept.
IVS
System
(OS, HW, ...)
conventional
system internal
value timing
Mcon,1 Tcon,1
ITS Stack
Mcon,a Tcon,a
Icon,1
Icon,a
Ocon,1
Ocon,c
Osi,1
Osi,d
Isi,1
Isi,b
Fig. 5. Input and output of IVSs.
3160 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 8/15
5.2.3.1. First example. Scenario: A vehicle approaches anintersection with constant speed, whereas, the traffic light
will signalize red light for the vehicle’s driving lane at the
point in time when the vehicle will cross the stop line. The
RVW application (see Section 3.2) is active and should gen-
erate output events because a red light violation will occur.
The events’ content should indicate a red light violation.
Moreover, the IVS’ CPU could be stressed with various
additional CPU load conditions and, thus, the response
time of the RVW application might suffer.
Description: For the first example, the following path
through the morphologic box as depicted in Fig. 4 is used:
test type: non-functional testing ?modification of input
streams: not targeted? gather output variation: yes? evalu-
ation of output stream: metrics and assessment ? output
stream property: correctness of timing . Thus, the variables
are defined as follows: M con,1 = = M con,a = {}, b = 1, c = 1,
and d = 0. So, the test case could be notated as a subset of
the mapping:
TCI # I si;1 Ind ! Ocon;1; ð4Þ
whereas, the set I si,1 contains various additional CPU load
conditions. Furthermore, it is assumed that the RVW appli-
cation’s output stream Ocon,1 is of the type: event based con-
tinuous-time stream over the set {GONG,BRAKE }. So, multiple
test runs are needed and descriptive statistics (see Table 1)have to be applied for the output stream’s evaluation.
Evaluation: In order to not exceed the scope of theexample, only definitions of normalized histograms are gi-
ven exemplarily. Furthermore, it is expected that the RVW
application generates two messages in this example,
whereas, the first one has to have the value GONG and
the second one has to have the value BRAKE to follow
the double-staged warning approach. So, the histogram
H l, x with the k histogram bars H l; xk is defined for the load
condition l and the points in time where the x-th message
with x 2 {1,2} appeared as follows:
H l; xk : ¼ 1
jTCl
I j
tchl;ii 2 TClI jN hl;iicon;1 ¼ 2 ^ u
hl;iicon;1;1n
¼ GONG ^ uhl;iicon;1;2
¼ BRAKE ^ w k 6 t hl;iicon;1; x < w ðk þ 1Þ
o; ð5Þ
with
tchl;ii ¼ hl; ii ! ohl;iicon;1
D ED E 2 TClI ; ð6Þ
TCl¼ x
I ¼ ftchl;ii 2 TCI jl ¼ xg; ð7Þ
ohl;iicon;1 ¼ t
hl;iicon;1;1; u
hl;iicon;1;1
D E; . . . ; t
hl;ii
con;1;N hl;ii
con;1
; uhl;ii
con;1;N hl;ii
con;1
:
ð8Þ
Table 1
Metrics and assessment strategies for timing of streams.
Metrics Assessment strategies
Single run Multiple runs
Stream property: event based appearance
Appeared/not appeared – Ratio
Time stamp of
appearance
– Descriptive statistics
Stream property: sampling based (periodic appearance)
Jitter of sampling time Descriptive statistics for fluctuation of
sampling interval
Descriptive statistics for single-run results (e.g. distribution of mean
values of single runs)
Lack of samples Ratio Descriptive statistics for single-run results (e.g. distribution of ratio
of single runs)
Start/stop time stamp of
sampling
Start and stop time stamp
Duration of sampling
Descriptive statistics for start/stop time stamps
Descriptive statistics for durations
Table 2
Metrics and assessment strategies for messages of streams.
Metrics Assessment strategies
Single run Multiple runs
Stream property: messages in a discrete non-ordered set
Message’s value in subset of correct values Test result Ratio of in set and not in set of correct messages
Jumping message’s values – Set of appeared messages
Count of appearances per message (e.g. stored
in a set like: ðU NÞN )
Stream property: messages in a continuous or discrete totally ordered set
Raw values Descriptive statistics Descriptive statistics
Derivatives of values Descriptive statistics Descriptive statistics
Properties of signals modulated onto the samples (e.g. ramps,
or frequencies) (see [39])
Signal properties (e.g. max.
slope)
Local signal properties
(within window)
Descriptive statistics for signal properties
S. Röglinger/ Computer Networks 55 (2011) 3154–3168 3161
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 9/15
The symbols in (5)–(8) are defined as follows:
H l, x histogram for load condition l and the x-
th appeared message
H l; xk
height of k-th histogram bar of H l, x
N hl;iicon;1
number of messages in conventional
output stream 1 for load condition l and
the i-th test run
l 2 I si,1 additional CPU load
i 2 Ind index of test run
k 2 N [ f0g index of histogram bar
TClI
set of test cases for input streams I and
load condition l
tchl,ii test cases of i-th test run for load
condition l
ohl;iicon;1
conventional output stream 1 for load
condition l and the i-th test run, and
w 2 R>0 width of histogram bar in seconds
The histogram bars H l; xk as defined in (5) have all the
same width. This is not mandatory for all histograms and
thus the histogram bars might also be defined with varying
widths. Please note that Figs. 13 and 14 in Section 6.2.1
show histograms exemplary.
5.2.3.2. Second example. Scenario: A vehicle approaches an
intersection with constant speed, whereas, the traffic light
will signalize red light for the vehicle’s driving lane at the
point in time when the vehicle will cross the stop line. The
RVW application (see Section 3.2) is active and should gen-
erate output events because a red light violation will occur.
The events’ contents should indicate a red light violation.
Moreover, impacts of intersection topology center point
shifts and, thus, shifts of the whole intersection topology,
on the points in time where the RVW implementation de-
tects the red light violation should be analyzed.
Description: For the second example, the following path
through the morphologic box is used as depicted in Fig. 4:
test type: functional testing ? modification of input streams:
static + inside boundaries? gather output variation:
no? evaluation of output stream: metrics and assess-
ment ? output stream property: correctness of values. Thus,
the variables are defined as follows:
M con;1 ¼slatitudenorth ; slatitude
south ½R
slongitudeeast ; slongitude
west ½R
,
M con,2 = = M con,a = {},
b = 0, c = 1, and d = 0
with the following meanings for the variables:
slatitudenorth
max. deviation in northern direction
slatitudesouth
max. deviation in southern direction
slongitudeeast
max. deviation in eastern direction, and
slongitudewest
max. deviation in western direction
respectively for the intersection topology’s center point.
These variables have to be defined for the final test case.
Finally, the test case could be notated as a subset of themapping:
TCI #M con;1 Ind ! Ocon;1; ð9Þ
whereas, the set M con,1 contains the modifications of the
input stream I con,1 which ought to represent the input
stream of intersection topologies. Hence, the intersection
topology is broadcasted periodically, every send-event re-
quires modifications. In addition, it is assumed that the
RVW application’s output stream Ocon,1 is from the same
type as in the first example.For the assessment, test runs for different combinations
of modified input streams are needed and methodologies
from the area of descriptive statistics (see Table 1) have to
be applied. Since the impacts from intersection topology
center point shifts on the points in time where the RVW
implementation detects a red light violation could not be
handled analytically, it was decided to shift the intersec-
tion topology center point inside its tolerance boundaries
stochastically with uniform distribution in longitude and
latitude direction.
Evaluation: In order to not exceed the scope of the
example, only definitions of the minimum ðT xminÞ, maxi-
mum ðT xmaxÞ, and quantile ðT xqquantileÞ time stamp are given:
T xmin :¼ minðseqcon;1; xÞ; ð10Þ
T xmax :¼ maxðseqcon;1; xÞ ð11Þ
according to [40]:
T xqquantile :¼ sortðseqcon;1; xÞ:dq:jseqcon;1; xje; ð12Þ
with
seqcon;1; x :¼ ft con;1; xjN con;1 ¼ 2 ^ ucon;1;1
¼ GONG ^ ucon;1;2 ¼ BRAKE g; ð13Þ
which contains all time stamps t con,1, x where a GONG
( x = 1) or a BRAKE ( x = 2) message appeared. Please notice
that in (12) the sort-function returns the ascending sorted
input sequence, the first dot denotes the access of an ele-
ment of the sequence (seq.n returns the n-th element of
seq), and the second dot denotes a multiplication.
5.3. Usage of real world traces
Since the previous sections deal with test case configu-
rations, description of stream modifications, and assess-
ment strategies for evaluating output streams, it is still
open which streams could be used as input streams. So,this section deals with the question:
How to use real-world traces exhaustively to test inter-
section related V2X applications?
Therefore, a two-layered approach with and without
modifying real-world traces is presented in the next two
subsections.
First, information flows of intersection related V2X
applications (Section 3.1) show two independent informa-
tion flows from IRSs or ICSs, and ECUs to IVSs. Additionally,
the complete information flow shows no feedback loops
and, thus, the content of the flows is not affected by reac-tions or outputs of IVSs. Therefore, it is possible to record
3162 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 10/15
traces from the information flows. Second, the traces could
be modified as described in Section 5.2 before reapplying
them to IVSs. This limits the amount of required real-world
traces.
Vehicle driving simulators can also generate ECU traces
and, thus, can substitute real-world traces. This may pro-
vide more flexibility by taking traces in required intersec-
tion topologies and traffic situations. Moreover, traces
taken this way are not even affected by disturbances like
real-world traces.
5.3.1. Reapply raw traces time shifted
The simplest way to use information flow traces from
IRSs or ICSs, and ECUs is to leave them unchanged and
reapply them time shifted. Herewith, test cases regarding
the timing between equipped intersections and equipped
vehicles could be stressed and reapplied. For example, it
is possible to stress the following situations by reusing
the same traces for IRS to IVS and ECU traces:
the vehicle crosses the stop line at red light,
the vehicle crosses the stop line at green light,
the vehicle crosses the stop line at the point in time
when the traffic light switches from red/yellow to
green, and so on.
So, the requirements 1, 3, 5, 6, 7, and 9 which are pos-
tulated in Section 4.3 could be fulfilled. Moreover, test runs
could be executed automatized and, thus, the expensive
and time consuming test drives could be minimized. Also,
regression test strategies could be applied even in develop-
ment phases.
To be able to use this approach, two independent sets of
traces are required:
set of intersection IRS traces, and
set of vehicle ECU traces.
The set of intersection IRS traces covers DSRC and cellular
mobile communication and has to include traces covering
all relevant intersection behaviors of all relevant intersec-
tion topologies. The intersection’s behavior targets the sig-
nal state timing as described in Section 3.1. Analogously,
the set of vehicle ECU traces has to include traces covering
all relevant approaching and leaving behaviors of vehicles
on all intersections which are covered by the set of inter-
section traces. Among others this includes modifications
of: speed/acceleration, lane changing, stop & go, and driv-
ing direction. In addition, the GPS stream has to be part
of an ECU trace.
5.3.2. Reapply modified traces time shifted
In this subsection, a refined approach of the previous
subsection’s approach is described to target the remaining
requirements 2 and 4 of Section 4.3. The refinement targets
modifications of raw streams as foreseen in Section 5.2.
This approach uses the same traces as the approach in
the previous subsection. However, those traces combine
several streams and, thus, it is not possible to modify them
directly. To achieve a modification of the streams, the fol-lowing steps have to be applied (see Fig. 6):
(1) split up traces into streams
(2) modify messages per stream
(3) modify timing behavior per stream
(4) combine streams into new traces.
5.4. Test system architecture
A test system for testing intersection related V2X appli-cations needs, on the one hand, the possibility to stimulate
IVSs with test input data and, on the other hand, the pos-
sibility to record the system’s outputs including automatic
verdict building. Fig. 7 depicts a test hull which encloses
IVSs as SUTs and handles test runs. To target the test needs
for intersection related V2X applications, the test hull con-
tains subcomponents for emulating the behavior of IRSs,
ICSs, ECUs, and localization facilities and subcomponents
for emulating the influences of DSRC, cellular mobile com-
munication, vehicular internal communication busses, and
localization methodologies as postulated in Section 4.
In addition, a test system requires a test management
component to initiate and evaluate test runs as depictedin Fig. 8. These points are realized by a test controller sub-
component and a test evaluation subcomponent. The test
controller subcomponent initiates test runs which are
specified in test suit documents, whereupon, the test hull
applies the test input data to the IVSs. Thereupon, the test
evaluation subcomponent takes the recorded IVSs outputs
and evaluates test verdicts. For test cases which require a
replay of the test runs until a test end criterion is fulfilled,
the test controller subcomponent handles the iterations
original
trace
modified
trace
split merge
value timing
value timing
value timing
Mcon,1 Tcon,1
Icon,1
Mcon,2 Tcon,2
Icon,2
Icon,n
Mcon,n Tcon,n
Fig. 6. Trace modification.
<<flow>>
<<flow>> <<flow>>
<<flow>>
TestHull
CeMo
ViBus
DSRC
Rec.
IRS
ICS
ECUs
IVS
<<flow>>
<<flow>>
<<flow>>
Local. GPS<<flow>> <<flow>>
Fig. 7. Test hull.
S. Röglinger/ Computer Networks 55 (2011) 3154–3168 3163
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 11/15
and the test end criteria evaluation. Finally, all relevant
information is stored in test report files.
6. Proof of concept
To proof the concept described in the previous section,
the two examples presented in Section 5.2.3 were used.
As IVS, a CarPC which executes the RVW implementation
from the Audi travolution predevelopment project was
used. First, a short description of the Audi travolution sys-
tem architecture, the used test system architecture and the
used input streams is given. At second, the input stream
modifications and the results of the test runs are presented
for both examples. Furthermore, it has to be mentioned
that only DSRC communication and no cellular mobile
communication is used in this proof of concept.
6.1. Environment
6.1.1. Travolution system architecture
As depicted in Fig. 9, the Audi travolution system archi-
tecture consists of the following components:
CarPC The CarPC acts as IVS and executes the
Audi travolution software. Additionally,
the CarPC generates the visualization
which could be seen in the driver’s
instruments display and in the display
located in the center of the instrument
panel.
CarGate The CarGate connects the CarPC to the
relevant vehicle’s CAN-Busses and thus
to the other ECUs. Moreover, the CarGate
includes a GPS receiver which is used for
the positioning of the vehicle.
V2XGate The V2XGate receives the DSRC messages
from the traffic lights and forwards them
to the CarPC. Communication in the
opposite direction is possible but not
used.
UMTS modem The UMTS modem is used to establish
connections to servers which act as ICSs
and are available over internet. Those
servers are used by some applications as
alternatives to the DSRC communication.
As well as for the V2XGate, communica-
tion in the opposite direction is possible
but not used.
HMI The vehicles HMI displays the visualiza-
tions generated by the CarPC. For that
purpose, there is one display which is
part of the driver’s instruments and an
additional display in the center of the
instrument panel.
For this proof of concept, a commercially available per-
sonal computer based on a 2.2 GHz Intel Pentium 4 CPU
with 1.25 GB RAM and Microsoft Windows XP was used
as CarPC.
6.1.2. Test system architecture
Since only information flows originated from ECUs and
IRSs are relevant for the two examples, it is sufficient to
emulate the behavior of the corresponding components.
Hence, traces for testing the Audi travolution system could
be recorded by sniffing the ethernet communication be-
tween CarGate and CarPC, and V2XGate and CarPC. The test
hull used in this proof of concept has only to include a
component to stub a CarGate which includes the emula-
tion of ECUs and localization facilities (see Fig. 7) and a
component to stub a V2XGate which includes the emula-
tion of intersection’s IRSs. Beyond that, the test hull hasto include another component to record the CarPC’s out-
puts. Hence, the Audi travolution CarPC provides an en-
tirely rendered video stream to the vehicle’s HMI display
(see Fig. 9), the RVW application’s output events are addi-
tionally fed out via UDP datagrams and, thus, can be
recorded by the test hull’s recording component.
TestManagement
TestController
TestEvaluation
TestReport
TestSuit
IVS
TestHull
< < f l o w > >
<<flow>>
<<flow>>
<<read>>
< < w r i t e > > <<flow>>
Fig. 8. Test management.
CarPCV2XGate
CarGate
UMTSmodem
ECUs <<flow>>
CAN
HMI
<<flow>>Ethernet
<<flow>>
Ethernet
<<flow>>
<<flow>>
USB
DVI
Fig. 9. Travolution system architecture.
10 20 30 40
time
in sec
25
30
35
40
45
50
in kph vehicle speed
10 20 30 40
time
in sec
-80
-60
-40
-20
20
40
in degree steering wheel angle
Fig. 10. ECU input trace plot.
3164 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 12/15
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 13/15
6.2. Examples
This sub section picks up both examples introduced in
Section 5.2.3, whereas, Section 6.2.1 uses the first and
example and Section 6.2.2 uses the second one.
6.2.1. Example 1
As foreseen in Example 1 in Section 5.2.3, the input
streams were applied in exactly the same fashion several
times. In order to not exceed the scope of this proof of con-
cept, only the following two additional CPU load conditions
were used:
no additional CPU load, and
two threads calculating the 35th fibonacci number in
endless loops in parallel.
For both additional CPU load conditions, the input
streams were applied N = 300 times. The resulting histo-
gram plots for the points in time where the BRAKE signal
appeared are depicted in Fig. 13. In the context of this sub-
section, the difference in time between the 0.02-quantile
and the 0.98-quantile is used as definition of the term
t variation.
It can be seen that the Audi travolution system reacts
onto the applied input streams in normal load conditions
within a variation of t variation = 271 ms, whereas, the BRAKE
signal arises as foreseen approximately 2 s before the
vehicle crosses the stopline. In nontypical heavy loadcondi-
tions, the Audi travolution system reacts reliably within a
variationof t variation = 812 ms. Finally,it has to be mentioned
that in all test runs the BRAKE signal appeared and, thus, no
histograms for faulty test results can be generated.
However, this system behavior was not considered as
sufficient. So, after inspecting the involved software com-
ponents, a non-optimal decision in the software design.
After a redesign and modification of this software compo-
nent, the measurements were repeated. As it could easily
be seen in Fig. 14, the Audi travolution RVW implementa-
tion acts now more deterministic. If the Audi travolution
system runs now in normal load conditions, the variation
amounts now to t variation = 174 ms, whereas, in heavy load
conditions the variation amounts now to t variation = 642 ms.
6.2.2. Example 2
As foreseen in Example 2 in Section 5.2.3, modifications
of the intersection’s center point shifted within its bound-
aries were used. Since no exact boundary specifications
had been available, the following three ranges for
slatitudenorth ; slatitude
south ; slongitudeeast , and s longitude
west were used:
5 m,
3 m, and
1 m
for shifting the intersection’s center point in longitude
and latitude direction. So, for each test run, values for the
uniform distributed random variables for shifting the
intersection’s center point ware determined.
Theresulting – not requiredin thisexample– histogramsare depicted in Fig. 15 and the statistical valuations are pos-
tulated in Table 3. The terms zero (and non-zero) in Fig. 15
indicates that the histogram bars in the referred areas are
really zero (not zero) and not just small. The results show,
that specifications of the acceptable inaccuracy of the posi-
tion of intersection center points may have considerable
influence on the behaviorof RVWapplications if theapplica-
tion is not capable to deal with it.For example,T x¼2min :
valuates
to 14.95 s for a variation range of ±3 m, whereas, it valuates
to 21.00 s for a varation range of ±1 m. Thus, RVW imple-
mentations should indeed be tested regarding this issue.
7. Related work
Hence, testing of V2X communication systems is cur-rently a topic of major research interest; many parties
21.0 21.5 22.0 22.5 23.0 23.5 24.0
time in
sec0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Prob. no additional load
21.0 21.5 22.0 22.5 23.0 23.5 24.0
time in
sec0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Prob. two threads calc. fibonacci(35) in endless loops
Fig. 13. Histograms H l=no_add_load, x = 2 and H l=2_fibonacci_35, x=2 for example 1
with not improved software; bar width w = 0.05 s, number of test runs
N = 300 in each case.
21.0 21.5 22.0 22.5 23.0 23.5 24.0
time in
sec0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Prob. no additional load
21.0 21.5 22.0 22.5 23.0 23.5 24.0
time in
sec0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Prob. two threads calc. fibonacci(35) in endless loops
Fig. 14. histograms H l=no_add_load, x=2 and H l=2_fibonacci_35, x=2 for example 1
with improved software; bar width w = 0.05 s, number of test runs
N = 300 in each case.
3166 S. Röglinger / Computer Networks 55 (2011) 3154–3168
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 14/15
8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf
http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 15/15