CADDYCADDY SeCoGIS 2008 1
Managing Sensor Data of Urban Traffic
M. Joliveau1, F.De Vuyst1, G. Jomier2,
C.M. Bauzer Medeiros3
ACI Masses de Données CADDY (2003-2007)
(1) MAS, Ecole Centrale de Paris
(2) LAMSADE, Université Paris-Dauphine
(3) IC, UNICAMP
CADDYCADDY SeCoGIS 2008 2
Goals Urban road traffic
analysis congestions Query the past
behavior Foresee the future
behavior Show understandable
résults
(Google Maps)
CADDYCADDY SeCoGIS 2008 3
Outline
Received Data Exploratory studies Deeper Analysis Work to do Concluding remarks
(Google Maps)
CADDYCADDY SeCoGIS 2008 4
Data about the system to be studied
- Graph with hundreds of sensors
- Flow rate, occupancy rate, 3’
- States: fluid (0) / congestion (1)
- Annotations
From INRETS
CADDYCADDY SeCoGIS 2008 5
Mass of Data
Sensor number (I)
Day number (J)
Number of measures in a day (K)
High rate of missing dataBad quality of dataSize order of the volume O(109) as I, J, K : O(103)
CADDYCADDY SeCoGIS 2008 6
Exploratory study
Temporal view
Space-time view : dynamic vizualization of the sensor
state map
Flow rate Occupancy Rate
Hours 0->24h
CADDYCADDY SeCoGIS 2008 7
Traffic States: fluid/congestion
It appears : 2 states are not enough to characterize the dynamic behavior of the system Urban Traffic
Spatio-temporal patterns
CADDYCADDY SeCoGIS 2008 8
Space-Time Vizualization flow rate
x
Time
y
CADDYCADDY SeCoGIS 2008 9
Analysis of temporal series
Extract of one week for a sensor among 400 Regularity of the human activity generating traffic
CADDYCADDY SeCoGIS 2008 10
Schema of Data Base for Analysis
sensor-id
Sensors
day-id
Days
hour-id
Hours
annotation-id
Annotation
weather-id
Weather
sensor-idday-idhour-idannotation-idweather-id
Traffic
flow rateoccupancy ratetraffic state
CADDYCADDY SeCoGIS 2008 11
Symbolic representation of sets of temporal series
Symbol = label associated to a class reduction of size and intelligibility Class identification of typical behavior, detection of atypical behaviors
Episod partitionSymbol AlphabetSymbolic Representation
CADDYCADDY SeCoGIS 2008 12
Plan
Received Data Exploratory studies Deeper Analysis
STPCA Continuous Traffic
State Variable Concluding remarks
CADDYCADDY SeCoGIS 2008 13
STPCA Spatio-Temporal Principal Component Analysis
Goal : data representation in a reduced number of spatial dimensions => sensors temporal dimensions => daily instants Result :Data projection simultaneously on the first spatial and temporal
eigenmodes
1st experiment : Flow rate (Monday to Friday) for a family of reliable sensors
CADDYCADDY SeCoGIS 2008 14
Spatial Reduction
Xd (complete) matrix of daily realizations
element xi,t ,i sensor , t instant , d day
T number of instants by dayN number of daysI number of sensors
Y assembles horizontally N matrices Xd : Y = col (X1, X2,...... ,XN)
CADDYCADDY SeCoGIS 2008 15
Sensors Number
Number of Measure Instants
Daily Data
Matrix Y for spatial reduction
CADDYCADDY SeCoGIS 2008 16
Spatial Reduction
Y assembles horizontally N matrices Xd : Y = col (X1, X2,... ,XN)Each line is a temporal serie for 1 sensor
Singular value decomposition of Y Spatial correlation matrix: MS = YYT
Eigenvalues l1 >= l2 >= ... lKM
Eigenvectors (Fk) for k = 1…KM
CADDYCADDY SeCoGIS 2008 17
Spatial Reduction
Spatial correlation matrix Ms = YYT
Eigenvalues: λ1 ≥ λ2 ≥... λKM
Eigenvectors: Ψk for k = 1…KM
P matrix of the K first eigenvectors Ψk
P = col (Ψ1, Ψ2, ... ΨK) for K<< KM
CADDYCADDY SeCoGIS 2008 18
Spatial Reduction
Estimate X’d of each realization Xd :X’d = P PT Xd K reduced spatial orderReduced order matrix : Xr = PT Xcontains latent (hidden) variables of Xsize : K * T (T instants)
If T is large, the dimension of the reduced order representation is too large
CADDYCADDY SeCoGIS 2008 19
Temporal Reduction
Z assembles vertically N day realizations Xd :
Z = row (X1, X2,... ,XN)
one colon corresponds to one instant t
one line corresponds to one sensor i for one day d
the data of one day d are grouped
I* N lines
CADDYCADDY SeCoGIS 2008 20
Sensors Number
Number of Measure Instants
Daily Data
Matrix Z for temporal reduction
CADDYCADDY SeCoGIS 2008 21
Temporal Reduction
Z assembles vertically N day realizations Xd :
Z = row (X1, X2,... ,XN)
Singular value decomposition of Z Temporal correlation matrix Mt = ZTZ
Eigenvalues μ1 ≥ μ2 ≥ ... μLM
Eigenvectors (Φl) for l = 1, 2…LM
Q matrix of the L first eigenvectors Φl
Q = col (Φ1, Φ2, ...ΦL) for L << LM
CADDYCADDY SeCoGIS 2008 22
Temporal Reduction
Estimate X’ for each realization X:X’ = X Q QT
Reduced order matrix: Xr = XQcontains the latent variables of Xsize : I *L
If I (space : number of sensors) is large the dimension of the reduced order representation is too high
CADDYCADDY SeCoGIS 2008 23
Results of temporal component analysis
temps temps
temps temps
tempstemps
Mode 1
Mode 3
Mode 5
Mode 2
Mode 4
Mode 6
temps temps
temps temps
tempstemps
Mode 1
Mode 3
Mode 5
Mode 2
Mode 4
Mode 6
The 6 first temporal modes(ACP-t)-order :colon- definethe matrix Q (Jr=6)
CADDYCADDY SeCoGIS 2008 24
Reduction: projection on the first temporal mode
Flow rates, 1 work day, 6 sensors - Observed flow rate- Projection on the1rst temporal mode
tempstemps
tempstemps
temps temps
Cap
teur
1C
apte
ur 3
Cap
teur
5
Cap
teur
2C
apte
ur 4
Cap
teur
6
Time Time
Time Time
Time Time
Sens
or 1
Sens
or 2
Sens
or 3
Sens
or 4
Sens
or 5
Sens
or 6
CADDYCADDY SeCoGIS 2008 25
Spatio-Temporal Reduction
Combines spatial and temporal analysis
new estimate of each realization X
X’ = PPTXQQT
Reduced order matrix:
Xr =PTXQ
contains the latent variables of X
size K*L
CADDYCADDY SeCoGIS 2008 26
Cumulative Energy
Spatial correlation matrix
Eigenvalue Index
Eigenvalue Index
Cum
ulat
ive
Ene
rgy
Cum
ulat
ive
Ene
rgy
Temporal correlation matrix
CADDYCADDY SeCoGIS 2008 27
Sensor 1 Sensor 2 Sensor 3 Sensor 4
Sensor 5 Sensor 6 Sensor 7 Sensor 8
Sensor 9 Sensor 10 Sensor 11 Sensor 12
Sensor 13 Sensor 14 Sensor 15 Sensor 16
Work days K=3, L=3
CADDYCADDY SeCoGIS 2008 28
Mean Direct Error
Standard Deviation
Reduced-order Matrix
Size
Mean Direct Error
Standard Deviation
Reduced-order Matrix
Size
Mean Direct Error
Standard Deviation
Reduced-order Matrix
Size
CADDYCADDY SeCoGIS 2008 29
g
Sensor 1 Sensor 2 Sensor 3 Sensor 4
Sensor 5
Sensor 9
Sensor 13
Sensor 6
Sensor 10
Sensor 14
Sensor 7
Sensor 11
Sensor 15
Sensor 8
Sensor 12
Sensor 16
Chrismas Day K=3, L=3
CADDYCADDY SeCoGIS 2008 30
Error Distribution FunctionSensors
Num
ber
of S
enso
rs
CADDYCADDY SeCoGIS 2008 31
Error Distribution Function Days
Nu
mb
er
of
da
ys
CADDYCADDY SeCoGIS 2008 32
Plan
Received Data Exploratory studies Deeper Analysis
STPCA Continuous Traffic State
Variable Concluding remarks
CADDYCADDY SeCoGIS 2008 33
Generation of 7 new traffic states using analysis in phase space
Saturé
Fluide
Grandecirculation
Occ
upan
cy R
ate
Flow Rate
CADDYCADDY SeCoGIS 2008 34
Continuous traffic state variable
Occupancy rate
Throughput
CADDYCADDY SeCoGIS 2008 35
Sensor 1
Time (hour)
Time (hour)
Time (hour)
Flo
w R
ate
(n
b v
eh
icle
s)O
ccu
pa
ncy
Ra
te (
%)
Circ
ula
tion
Sta
te (
%)
CADDYCADDY SeCoGIS 2008 36
State Name Symb. State Num. Symbol E value at t Deriv. Sign in t
Calm
Negative
Very high level circ.
Saturation level 1
High level circul.
Saturation level 2
Saturation level 3
Positive
Back to Calm
New circulation states
CADDYCADDY SeCoGIS 2008 37
Dynamic Visualization of the Traffic State
Fluid
Congestion
Animation :spatio-temporal patterns appear
CADDYCADDY SeCoGIS 2008 38
Other results
Missing Data STPCA for state variables Spatio-temporal patterns
See Marc Joliveau ‘s PhD Thesis
CADDYCADDY SeCoGIS 2008 39
Work to be done
Enrich the datawarehouse with summaries, GIS, results of STPCA…
Symbolic spatio-temporal analysis Adaptation to evolution Visualization, user interaction Refinement on types of days, episodes Datawarehouse : queries
CADDYCADDY SeCoGIS 2008 40
Concluding Remarks Reduction : from data masses to intelligible and manipulable
elements
Generic Approach For spatio-temporal analysis of flow systems, described by data coming from a network of static
georeferenced sensors with diffuse sources and wells
CADDYCADDY SeCoGIS 2008 41
Future Prospects
Data coming from embarked sensors
Go farther in spatio-temporal reduction