Development of a traffic lights control system based on image

processing for pedestrians safety

Andre Pereira Gomesandre.gomes@tecnico.ulisboa.pt

Instituto Superior Tecnico, Universidade de Lisboa, Lisboa, Portugal

November 2017

Abstract

According to statistics, about 16% of traffic accidents in Portugal in 2015 and 2016 were relatedto runovers. Not rarely, runovers take place in crosswalks with traffic lights, when pedestrians werecrossing the road with the light still on the red state The timing of the traffic lights is often optimizedby prioritizing the flow of traffic, leading to very long waiting times for pedestrians, leading to attemptson crossing ignoring safety conditions. On the other hand, vehicles with exceeding speeds near crosswalkscan cause quite violent vehicle-man collisions, increasing the possibility of injuries to the pedestrians,or even their death.

In this sense, considering the advances in methods of Artificial Vision, it was developed a method-ology for a system that, using cameras, is able to estimate the speed of vehicles along the traffic lanesand identify pedestrians and analyze their behavior in the sidewalk, in order to realize their intentionto cross the road. That way, traffic lights actuation rules can be rethink in order to better create safetyconditions for pedestrians, specifically when they are about to cross the road, while also reducing thespeed of vehicles that are approaching the crosswalk.

In this work, the software Matlab/Simulink was used, implementing the concepts of Optical Flow,applied in the detection and monitoring of vehicles and pedestrians next to a crosswalk For this purpose,Lucas-Kanade and Horn-Schunck methods were implemented. Finally, through the experimental results,the results are compared and the advantages and disadvantages of each of the two methods are evaluated.

As for pedestrians monitoring, it was always possible to predict that a pedestrian is trying to crossthe road, while evaluating possible behaviors in a region near a crosswalk.Keywords: Speed estimation, Monitoring of pedestrians, Optical flow, Road safety, Control of trafficlights

1. Introduction

During the year of 2015, 5477 pedestrians were runover by vehicles in Portugal. 142 of these accidentsresulted in death. During the year of 2016, thisnumber has increased to 5624 runovers, but thenumber of death victims was lower, although stillreaching 119 victims [1]. Based on the statisticsissued by the authorities, between 2010 and 2013,the total number of victims was 22534, distributedin 20119 minor injuries, 1718 serious injuries and697 deaths. By analyzing the statistical numbersof run overs and by grouping them into districts,it is verified that big cities are those that registera greater number of run overs. From these cities,Lisbon stands out with 6752 run overs.

From the total number of accidents with pedes-trians, 2750 took place in Lisbon County where, forexample, in the year of 2013, the total number ofvictims was of 670 [2].

A more in-depth assessment of the same cases in

Lisbon in 2014 led to the conclusion that the totalnumber of victims due to crossing in disrespect forpedestrian lights (red light) was of 42, including 1death. All this shows that in the period between2010 and 2014 there was no significant change in thenumber of accidents of this type, period on whichthere were 245 victims.

In order to outline the typical scenario of theseaccidents, it is considered a pedestrian that has athis disposal a cross-walk with traffic light. Althoughthere are campaigns to raise awareness of the dangerof crossing the road with the red signal still active,it is not guaranteed that the pedestrian will respectthem. In most cases, the traffic signals timing isdetermined in order to optimize traffic flow as themain objective, leading to long periods of waitingtime for pedestrians until the road crossing is safe,represented by a green light.

This situation leads to impatience, especiallywhen pedestrians are in a hurry. They try to cross

1

the road even when the signal is still red and thetraffic flow is not closed yet. If, due to poor visibil-ity or lack of attention, the pedestrian is not able tosee a vehicle or if the crossing time is not correctlyestimated considering the speed of the approachingvehicle, there is a danger of being hit by the samevehicle.

There are also several studies with focus on traf-fic monitoring, using computer vision. For instance,Choudekar et al. implemented image processingtechniques, such as edge detection using Prewittfilters for that purpose. A reference image of theempty road is captured and saved, after edge de-tection. Next, new images of moving traffic are ac-quired edge detection is once again performed andresults are compared to the original image, usingimage matching techniques. Traffic lights can thenbe controlled, based on the percentage of matching[3].

Also, Ramzan et al. proposed pedestrians de-tection by combining Optical Flow and HOG (His-togram of Oriented Gradients) [4]. However, thesescientific works do not combine traffic and pedestri-ans monitoring as a whole system. Moreover, someevaluation regarding the effect of adverse conditionson the images (such as rain, sun glare, etc.) shouldalso be taken into account when evaluating a prac-tical application of the studied methods.

2. BackgroundOptical Flow or Optic Flow (OF) is vastly appliedin several computer vision problems. Not only inobject movement detection and tracking, but alsoin video compression and stabilization that can befound in most digital cameras and smartphones.OF basically consists on estimating the motion ofan object by evaluating several sequential images,or frames of a video. Virtually, an object has a tree-dimensional motion vector that can be projectedonto the image plane (area scan camera). Theseprojections are what is called ”image flow field”.OF can be mathematically defined with equation 1:

∇IT · ~V = −It (1)

in which ∇IT = (Ix, Iy) represents the spatial gra-

dient, ~V = (u, v) = (∆x∆t ,

∆y∆t ) is the movement vec-

tor and It is the temporal gradient.By looking into equation 1, it can be stated that

it has two variables and due to that, it can notbe solved as it is. This is considered the ApertureProblem.

OF field can be computed by using methodsbased on local and global differentiation techniques,correlation, feature based methods and hierarchicalapproaches. These methods apply additional con-straints in order to solve equation 1. On this disser-tation, two differentiation techniques were studied:

Horn-Schunck (HS), and Lukas-Kanade (LK) beingglobal and local methods, respectively.

2.1. Lucas-Kanade MethodThe Lucas-Kanade (LK) method is a two-frame dif-ferential method that estimates the OF from a seriesof images using the least squares principle. It wasdeveloped by Bruce D. Lucas and Takeo Kanade[5]. This method establishes a neighborhood of aspecific pixel and then assumes that the flow of thesame neighborhood remains constant during the im-ages sequence. It then solves the OF equation 1, byapplying a weighed least squares criterion to thepixel and its established neighborhood. If a win-dow at center p is considered, it can be stated thatthe local velocity (or flow) vector must satisfy thefollowing equation:

Ix(qn)Vx + Iy(qn)Vy = −It(qn) (2)

where qn represents pixels inside the referred win-dow and the subscripts in the image letter I repre-sent partial derivatives. This equation can also bewritten in the matrix form

After applying a weighed version of the leastsquares technique, results are given by:∑

x∈Ω

W 2[Ixu+ Iy + It]2 (3)

The LK method is known to be less sensitive toimage noise, when compared, for instance, to point-wise methods. It is also considered easier, with veryfast calculations and accurate time derivatives [6].

2.2. Horn-Schunck MethodThe Horn-Schunck (HS) method assumes that theflow of the whole image is smooth. This meansthat the method tends to minimize distortions inthe flow. The flow is formulated as a global energyfunction (objective function) which is then soughtto be minimized. This global energy function bal-ances the error from a brightness constancy con-straint equation given by ec with a smoothness con-strain error es:

ec =

∫∫(Ixu+ Iyv + It)

2∂x∂y (4)

es =

∫∫(u2

x + u2y) + (v2

x + v2y)∂x∂y (5)

In equation 5, ux for instance, represents thederivative of the u velocity vector changing in thex direction. This equation penalizes changes in uand v over the image, which means that if pixels aremoving always with the same u and v, es would beequal to zero. After the establishment of the twoadditional error functions, the problem is rewrittenas finding u and v that minimize the global error(or energy) function:

2

e = es + αec (6)

α is a weighing factor, also known as smoothnessfactor, that must be chosen regarding the amountof noise it is present in the image and how smooththe motion is.

After some mathematical steps, it is possible toobtain the iterative solution of the HS method ap-plied to the OF problem:

uk+1 = uk − Ix(Ixuk + Iyv

k + It)

α2 + I2x + I2

y

(7)

vk+1 = vk − Iy(Ixuk + Iyv

k + It)

α2 + I2x + I2

y

(8)

This algorithm has the disadvantage to be moresensitive to noise than other methods, specially lo-cal methods as it is the LK method. As such, itmust be applied with care. On the other hand,since the inner parts of the homogeneous objects(or voxels) is filled with information from the mo-tion boundaries, this algorithm has the advantageof yielding a higher density of flow vectors, whencompared to other methods.

3. ImplementationTwo codes were developed in Simulink/Matlab.One for speed estimation of moving vehicles andanother one for pedestrians monitoring near a cross-walk. Both codes also made use of morphologicaloperators in order to fulfill blobs created by the OFblock.

Video footage of moving vehicles was acquiredwith a Rollei Actioncam 300 Plus. The actual speedof the vehicles was measured using a speed radargun for validation of the developed code.

3.1. Vehicle Speed EstimationFor the purpose of vehicle detection and speed esti-mation, the flowchart illustrated in figure 1 providesa general understanding of the steps of the code de-veloped.

Speed estimation is computed using virtual trig-gers in lines regarding the image points of interest.After evaluation of the image, marks on the roadwere chose, since it was easy to perform real-worldmeasurements. Virtual lines were then traced, foreach camera position that was studied, while con-firming that the virtual lines were passing throughthe measured road marks. A timer measures whatis the time interval that a centroid of the vehicletakes in order to travel from one virtual line to an-other. Since it is known the real distance of thevirtual lines, speed estimation is computed accord-ing equation 9:

Si =Di

tj+1 − tj(9)

Figure 1: Flowchart for vehicle speed estimation

where Si is the estimated speed in section i, Di

is the real distance between the lines that form thesection, j is the index of the line at each sectionand t is the time instant when the vehicle passesthrough line j. Only two sections were used for thiswork, defined between lines A, B and C. Figure 2is a computed generated image that illustrates theprinciple.

Figure 2: Virtual lines chosen regarding marks onthe road

3

3.2. Pedestrians Detection and Monitoring

The code that was implemented in order to detectpedestrians near a crosswalk (considering a safetyenvelope area) and to predict their intention oncrossing the road is pictured in figure 3.

Figure 3: Flowchart for pedestrians detection andmonitoring

The ideal camera position for the monitoring ofpedestrians is shown in figure 4

Figure 4: Position of the camera and safety enveloparea for the monitoring of pedestrians

Also in figure 4 the virtual lines that form thesafety envelope are shown (in red).

4. Results

For both vehicles and pedestrians detection, OFhad to be combined with morphological operations,such as erosion and dilation [7]. Combining theseoperators, it was possible to close the scarce OFfield of the moving object in order to obtain a solidblob for a robust blob analysis.

The effect of the morphological operators over theOF results can be seen in figures 5 and 6.

(a) OF result (b) After morphology

Figure 5: Blob creation for vehicles

(a) OF result (b) After morphology

Figure 6: Blob creation for pedestrians

As a result, both vehicles and pedestrians weresuccessfully detected, as shown in figure 7.

(a) Vehicle detection (b) Detection of two pedestri-ans

Figure 7: Detection of moving bodies with the de-veloped code

A detailed study was conducted in order to un-derstand how would the LK and HS methods pa-rameters change the OF field.

4.1. Lucas-KanadeIn this section the main results when applying theLK method to the OF problem for both vehiclesand pedestrians are shown.

4.1.1 Vehicles

Main results for vehicle monitoring using the LKmethod can be found in table 1, for a section of thewhole RoI. The simulation code was stopped alwaysat the same instant of time. Here it is visible theeffect of the threshold parameter for noise reduc-tion (τ) on the direct result of the OF method fora chosen vehicle. Moreover, it is also presented the

4

result when applying the same morphological op-erators on the direct OF results. For this case, itwas used dilation and erosion, both with a circularstructuring element with 20 pixels radius. At last,for each evaluation of the τ value, it is also presentedif the vehicle was detected by the system or if afterblob analysis, it was not possible to acknowledge itspresence.

Table 1: Effect of the noise reduction threshold (τ)in LK method for vehicle monitoring

τ = 0.0001 τ = 0.003 τ = 0.005

τ = 0.02 τ = 0.03 τ = 0.05

4.1.2 Pedestrians

Main results of the LK method for pedestrians de-tection are presented in table 2.

It is noticeable by analyzing table 2 how the noisereduction threshold parameter (τ) influences di-rectly the information of the moving object (in thiscase, the pedestrian). Reducing τ leads to an appar-ently robust detection. Contrarily, the increasing ofτ penalizes the optical flow in such a way that mostof the information is lost. In this case (τ = 0.02 intable 2), morphological operators would not be ableto recreate the scattered and scarce optical flow.

Next, the most important results are shown forboth speed estimation of vehicles and pedestriansdetection and monitoring.

4.2. Horn-Schunck

In this section the main results when applying theHS method to the OF problem for both vehicles andpedestrians are shown.

In order to better compare the results from HSto LK, the video footage was stopped always at thesame instant of time, while evaluating the effect ofthe variation of α, i.e. the smoothness factor.

Table 2: Effect of the noise reduction threshold (τ)in LK method for pedestrians monitoring

τ = 0.0005 τ = 0.001 τ = 0.003

τ = 0.004 τ = 0.005 τ = 0.02

4.2.1 Vehicles

While applying the HS method for OF estimation,the results presented on table 3 were obtained.

Table 3: Effect of the noise smoothness factor (α)in HS method for vehicle monitoring

α = 0.0001 α = 0.001 α = 0.01

α = 0.1 α = 1 α = 10

It is visible how the OF field deteriorates whenthe smoothness factor increases. However, whencomparing to the results from the LK method, itis obvious that this happens much more subtly. Infact, while using the HS method, it was always pos-

5

sible to compensate the lost information of the mov-ing vehicle OF field with the usage of morphologicaloperators resulting in a correct detection even whenthe smoothness factor was fixed at 20, as shown ontable 3

4.2.2 Pedestrians

Direct results for pedestrians detection can be foundin table 4.

Table 4: Effect of the smoothness factor (α) in HSmethod for pedestrians monitoring

α = 0.001 α = 0.01

α = 10 α = 100

The HS method resulted in a more consistentOF field computation. As seen on table 4, the αamplitude that was studied was from 0.001 up to100, giving very similar results. In fact, it was alsounnecessary to change the structuring elements formorphological operators in order to perform a cor-rect detection and tracking of the pedestrian.

4.3. Speed Estimation of Vehicles

Some of the estimated speeds for all the 42 vehiclesfrom which speed is know, measured with the speedgun radar, are shown in table 5.

Table 5: Speed estimation for some vehiclesMeasured Speed

[km/h]Estimated Speed

[km/h]64 63.6879 82.3066 68.5871 70.5471 70.5443 43.4454 55.11109 111.2552 54.48

Table 5 shows how close the estimated speed wasto the real vehicle speed. In fact, speed was esti-mated with a maximum relative error of 4.8% (lastline of table 5).

4.4. Pedestrians DetectionWhile acquiring video footage of pedestrians, sev-eral locations and positions of the camera were stud-ied. Initially, the camera was placed on the side-walk, away from the crosswalk. After further ex-amination, it was found that this camera positionwas not ideal. One of the reasons is that it wouldrequire additional infrastructures in order to mountthe camera, instead of using an already existent one(normally, the traffic light pole). Secondly, it wasmounted rather low, at about 2m. Due to this, sev-eral issued arose. Namely, due to the perspective,pedestrians could overlap, as shown in figure 8.

(a) OF field (b) After morpho-logical operators

(c) Final detec-tion

Figure 8: Two pedestrians overlapping inside theFoV

Afterwards, camera was mounted on a trafficlight pole, pointing down at pedestrians.

Regarding possible pedestrian behaviors, severalsituations were addressed, while evaluating the out-put of the system. First of all, if a pedestrian ismoving parallel to the road (and, therefore, per-pendicular to the crosswalk axis), the system is ableto acknowledge his presence, but does not considerthat the pedestrian is trying to cross the road. Vi-sually, this is indicated by the ”black” state of thelines that form the safety envelope, as shown in fig-ure 9.

Figure 9: Pedestrians outside the safety envelopearea

When a pedestrian is moving towards the cross-walk and enters the safety envelope zone, the system

6

changes its state, acknowledging that a pedestrianis about to cross the road. Visually, this can bechecked by the color of the virtual lines changingfrom black to white, as demonstrated in figure 10.

(a) Pedestrian still out-side the envelope (blacklines)

(b) Pedestrian alreadyinside the envelope(white lines)

Figure 10: A pedestrian entering the safety enve-lope area

All pedestrians were detected by the system.Moreover, it was always possible to change the stateof the system whenever someone entered the safetyenvelope area. Some more examples of pedestriansentering the safety envelope are presented in figure11.

Figure 11: Pedestrians entering the safety envelope

5. ConclusionsResults support the conclusion that both LK andHS methods are adequate for motion detection ap-plied to vehicles and pedestrians. When coupledwith simple morphological operators such as erosionand dilation, they deliver quality OF fields that canbe easily segmented in a typical blob analysis.

Due to the fact that in real implementation, codemust be performed fast enough in order to processlive video, code must be optimized. For that pur-pose, dividing the video into segments for each roadlane for a multi-thread architecture was crucial in

order to obtain processing times adequate for a sys-tem that requires real-time decision. Additionally,for optimal OF computation speed, LK method ispreferable to the HS, since it performs faster, withless computer demanding tasks.

Although LK and even HS are valid methodsfor the objective of the developed system, properresults are only achieved when video is acquiredin adequate positions. Namely, camera positionsthat lead to overlapping of vehicles or pedestriansmust be avoided. This is particularly important if apedestrian or a vehicle is totally covered by another,since it would not be possible to detect all vehi-cles and pedestrians and, consequently, to correctlyevaluate the conditions for a safe road crossing.

Regarding speed estimation, it is quite accept-able that the biggest relative error was of 4.8%,when comparing the estimated speed with the ac-tual speed. In this case, it represented an absoluteerror of about 2km/h.

Regarding the detection and monitoring of pedes-trians, 100% accuracy was achieved when pedestri-ans were approaching the safety envelope area inorder to cross the road.

References[1] Autoridade Nacional de Seguranca

Rodoviaria (ANSR). Vıtimas a 30 dias -ano 2016. Relatorio, Camara Municipal deLisboa, August 2017.

[2] Equipa do Plano de Acessibilidade Pedonal. At-ropelamentos em Lisboa Relatorio 2010 - 2013.Relatrio, Camara Municipal de Lisboa, January2015.

[3] Pallavi Choudekar, Indian Journal, and Com-puter Science. Real Time Traffic Light ControlUsing Image. Indian Journal of Computer Sci-ence and Engineering, 2(1):6–10, 2011.

[4] Sepehr Aslani and Homayoun Mahdavi-nasab.Optical Flow Based Moving Object Detectionand Tracking for Traffic Surveillance. 7(9):963–967, 2013.

[5] S. Baker and I. Matthews. Lucas-Kanade 20Years On: A Unifying Framework, volume 56.2004.

[6] Dhara Patel and Saurabh Upadhyay. Opticalflow measurement using lucas kanade method.International Journal of Computer Applica-tions, 61(10):133–138, 2013.

[7] J. Ponce and D. Forsyth. Computer vision: amodern approach.

7