evaluating quality of uav´s products: study case uruguay · traditional photogrammetric product,...

Evaluating Quality of UAV´s Products: Study Case Uruguay. (7644)

Rosario Casanova, Melissa Pérez and Veronica Pampinella (Uruguay)

FIG Working Week 2015

From the Wisdom of the Ages to the Challenges of the Modern World

Sofia, Bulgaria, 17-21 May 2015

1/16

Evaluating Quality of UAV´s Products: Study Case Uruguay

Rosario CASANOVA, Melissa PÈREZ RODRIGUEZ and Verónica PAMPINELLA

EGAÑA, Uruguay

Keywords: Photogrammetry, Unmanned Aerial Vehicle (UAV), Non-metric camera,

Positional Accuracy.

SUMMARY

As the increase of applications using non-traditional photogrammetric flights has been

growing up vertiginously, the need of knowing the real positional accuracy obtained on these

products takes a significant place, specially to be able to certify their certain possibilities of

use. For this reason, in this research, it has been made a positional quality control of a non-

traditional photogrammetric product, based on an UAV flight done in Montevideo, Uruguay.

The main goal of the project was to analyze products obtained from a digital photogrammetric

process based on data captured with a UAV´s flight and topographic measure points, applying

a known quality control method. The digital images were taken with a non-metric camera

(Sony NEX-7) mounted on an unmanned aerial vehicle (UAV) Microdrone MD4-1000 and

processed with the ERDAS IMAGINE 2013 software. As the average flight height was close

to 100m and the covered area was about 1 ha 4000 m2, it was required the use of four images

on one flight line to obtain tridimensional coordinates.

During the process, 65 tie points were defined to generate the stereoscopic model, 13 manual

tie-points according Von Grüber’s distribution and 52 automatics tie-points, and five control

points to the whole area. Those were measured by classic topographic methods, using total

station and automatic level to enhance their accuracy.

The quality control´s method has done based on the Positional Accuracy Handbook Using the

National Standard for Spatial Data Accuracy to measure and report geographic data quality

(1999). In order to apply this method it was necessary to obtain 29 check-points, which were

topographically and photogrammetrically measured. Those points were signalized before the

flight, to ensure they were visible on the aerial images.

Therefore, the research includes the analysis of residual vectors of point´s coordinates and an

evaluation of the restitution procedure. Finally, some suggestions are made in order to

improve the results obtained on a UAV´s flight.






2/16

Evaluating Quality of UAV´s Products: Study Case Uruguay

Rosario CASANOVA, Melissa PÈREZ RODRIGUEZ and Verónica PAMPINELLA

EGAÑA, Uruguay

1. INTRODUCTION

The research has been developed as the final project of the required curricula to obtain the

degree of Surveyor Engineer, of the Faculty of Engineering of the University of the Republic

of Uruguay. It took place since January to September of 2014, when Melissa Pérez and

Verónica Pampinella presented their research and ended their academic studies. It was

directed by the Prof. MSc. Eng. Rosario Casanova.

The main goal of the project was to analyze (studying positional accuracy) products obtained

from a digital photogrammetric process based on data captured by a UAV´s flight and

topographical measured points, applying a known quality control method.

2. PREPARING THE DIGITAL RESTITUTION

2.1 Introduction

To be able to test the quality of the products, the main idea of the research was to obtain all

data required to generate the photogrammetric products with the maximum possible accuracy.

This was done in order to reduce its incidence to the final product; therefore it was done a

detailed plan to minimize errors.

This section includes the selection of the flight zone, the calibration proceeds, the aerial flight

plan, the method used to measure the control and check points.

2.2 Flight Zone

To avoid the influence of the topographic surface in the final product, it was considered flying

on a flat area. Anyway, the product tested was produced in real conditions, for example, the

coverage of the images was 65%.

Most of the area 1 is concrete esplanade, but also includes natural grassland, a rough road and

part of a route that is the National Route No. 101.

It is located on the rural area of the department (similar to a state) of Canelones, Uruguay. In

Figure 1, the study zone can be seen, that is delimited by a yellow polyline.

1 It took into account if accuracy depends on the type of surface.






3/16

Figure 1. Aerial image of area where the flight was done.2

2.3 Camera calibration

Camera calibration is a process in which the geometric characteristics of an individual

mapping camera are measured. In the case of the study, the camera calibration process was

hired by the company owner of the camera; Agrimensura Birriel & González.

The parameters measured were the calibrated focal length, the image size, the principal point

coordinates, the number and coordinates of the fiducial marks, the symmetrical lens

distortion's polynomial coefficients: K0, K1, K2, K3 and the decentering lens coefficients: p1,

p2, p3. The company did not report the date of the calibration certificate.

2.4 Flight plan

The design of the flight plan provides an opportunity for cost optimization (directly to the

amount of fuel needed to flight) due to the determination of the optimal route, altitudes and

speeds.

In this case, the flight plan was performed by UAV-Agrimensura Birriel & González

Company, using the mdCockpit software.

To define the vehicle route, is necessary to configure the project requirements in that

application, for our project it was set:

Camera: Sony NEX-7.

Focal length: 16 mm.

Picture size: 23.50 x 15,60.

Average flight size: 100 m.

Speed: Horizontal: 10 m/s, vertical: 2 m/s.

Overlap: Along: 65 %, cross: 25.

Detention time: 3 s.

2 Source:Google Earth (2014).






4/16

The software´s output includes a list with the number and coordinates of the sampling points,

the number of flight lines and the flight time.

The design of the distribution of the ground control points and testing points was made over a

fight plan that finally was changed, unexpectedly, at the moment of the real UAV´s flight.

In in the left side of Figure 2 it is shown the flight plan used to define the points distribution,

and at the right side it can be seen the effective flight.

Figure 2. Left: Initial flight´s plan, Right: Real flight.3

2.5 Control and test points

The test´s points were defined based on the NSSDA Standard. For this reason 30 points4 were

included, randomly distributed, according to standard requirements. See Figure 3.

Figure 3. Check point distribution recommended by the NSSDA Standard. 5

Complementary, it was necessary to have control points therefore six control points were

defined and signalized in the ground, at the borders of the planned stereoscopic model.

3 Source:Google Earth (2014).

4 The standard recommends that at least 20 points be used to defining the positional accuracy, but it assumes normality and

randomness of the residual (dX, dY, dZ).

5 Source: Minnesota Planning (1999) Positional Accuracy Handbook, Using National Standard for Spatial Data Accuracy to

measure and report geographic data quality; Land Management Information Center.






5/16

Control and testing points were signalized on the ground using a cross pattern and white

spray. The size of the marks was defined taking into account the literature6 recommendations

and the analysis performed to ground-based images taken with the flight

2.6 Topographic survey

As the objective of the research was to test non-traditional photogrammetric products, it was

extremely important to measure the ground control and test points as well as possible.

Moreover, “The independent data set must be acquired separately from the data set being

tested. It should be of the highest accuracy available. In general, the independent data set

should be three times more accurate than the expected accuracy of the test data set.”

Minnesota Planning Land Management Information Center (1999). 7

To apply the NSSDA standard, it was necessary to make this measurement based on a specific

plan of survey to get the highest possible accuracy.

The equipment used was:

Leica 407 Total Station.

Pentax automatic Level.

In planimetry, it was taken special attention to:

Set the first station on an arbitrary point to measure an auxiliary point network and the control

and test points.

Measure every point at least twice, recording properly its coordinates.

Change the station's location, when it was measured at least four points in the net.

In elevation, it was considered important to:

Make double geometric leveling, meaning that each slope was measured at least twice.

Consider closed paths (six points each approximately) to calculate the vector error.

Finally, taking into account all these precautions it was made all the required measurement,

therefore once in the office the definitive topographic coordinates were calculated. It was

obtained root mean square (RMS) of the survey that is shown in them Table 1.

6 Ariza López, F.J. & James Atkinson, A.D. (2006) Metodología de Control Posicional: Visión General y Análisis Crítico,

Universidad de Jaén.

7 Provider’s values of accuracy: Ground pixel size: 2.5 cm, altimetric standard deviation: 5.5 cm, flying at 100 m altitude.






6/16

RMS (cm)

Planimetry 2.3

Altimetry 0.3

Table 1. Topographic measurement accuracy report.

Topographical survey results are graphically shown in Figure 4.

Figure 4. Contour graph of the flight zone.

As it can be seen, the area is gently undulating and there are not abrupt break lines.

2.7 Flight

The flight took place on June 1st,

2014, at 10.30 am, on a sunny day with wind that oscillated

between 10 and 15 km/h.

The photogrammetric flight was performed by UAV, which owner is Agrimensura Birriel &

González Company. It was used an Md 4-1000 Microdrone and the digital camara was the

Sony NEX-7.

Some characteristics of the flight are:

Average flight height: 106.64 m.

Maximum flight height: 11.30 m.

Minimum flight height: 101.51 m.

Flight lines: 2.

Shooting points: 12.

Flight Duration: 4 minutes.

Finally, to make the restitution process for this project there were included only four images

in one flight line.






7/16

3. DIGITAL RESTITUTTION

3.1 Introduction

To obtain the possible products from a UAV´s flight it was necessary to use a digital

photogrammetric software. Therefore, as the University has the Erdas Imagine 2013 with the

LPS Photogrammetric software, it was used for this research.

As the Surveying Institute has recently acquired the software there was not enough experience

on its use, causing a delay on the times of schedule planned for this stage.

3.2 Interior orientation

To rebuild the geometry of the camera, it was configured as “Frame Camera”. It was possible

because the calibration certificate contains data about the coordinates of the vertex of the

image, the fiducial length, etc. This procedure was recommended by Eng. William Kim

(photogrammetry expert) of the Leica Company.

3.3 Exterior orientation

3.3.1 Tie points

In restitution procedure, the initial coordinates of perspective centers determined by the

inertial navigation system (INS) were not used. Instead, 12 tie points were manually

introduced, to reduce the search area of the automatic tie point generation. Those were

defined based on Von Grüber's distribution.

For the automatic tie point generation, it was used default distribution. It was configured to

search 25 points on each image, but software found 53 tie points (success rate: 56%). In

Figure 5, all tie points can be seen.

Figure 5. Tie point distribution.8

8 Image captured from LPS, Erdas Imagine 2013.






8/16

3.3.2 Control points

As the flight done was quite different from the one in which the distribution control points

where defined, it caused that some of them could not been used. Therefore, the totally control

points defined in the section Control and test points, could not be used. In some cases, the

defined control points were seen in a single image and in others cases, the resolution of the

images near to the control point was not good. For that reason it was decided to use some test

points as control points. Finally, five points were used; which were referred to a local

reference system. As the final result of this stage it was obtained as Total Image Unit-Weight

RMS an RMSw=1 of 1,3 cm.

4. CONTROLLING THE POSITIONAL QUALITY

4.1 Introduction

In this section, the application of the NSSDA is used to obtain the absolute error and the

relative control is also made. Finally, the analysis of the results are presented.

4.2 Absolute positional quality

4.2.1 Prerequisites

Before applying the statistical NSSDA for a 95% confidence, it was necessary to check that

the data satisfy three conditions, which are:

1. At least 20 points distributed in the territory are needed.

2. Each component of the residual vectors (dX, dY, dZ) must be a random sample.

3. Random samples mentioned in item 2 must be normally distributed.

As it is shown in Table 2, the first condition was not satisfied in the third trigonometric

quadrant.

Area Number of

points

Percentage (%)

All 31 100

1st quadrant 7 23

2nd

quadrant 11 35

3rd

quadrant 6 19

4th

quadrant 7 23

Table 2. Ground control point distribution.






9/16

The non-homogeneous distribution was a consequence of the change in the flight plan.

Remember that the flight plan used to design the distribution of the points and the flight plan

carried out are considerable different.

As the difference on the percentage was quite little, it was assumed that the first condition was

satisfied, so it was accepted to continue with the analysis.

Regarding the 2nd and 3rd condition, requirements were verified applying the Runs test and

Kolmogrov Smirnoff test respectively.

In Table 3, Runs test is outlined, H0 is the null hypothesis, Ha is the alternative, R is the

number of runs, ZR is the test statistic and Z(1-α)/2 is the critical value for a 99 % of

significance.

dX dY dZ

Ho DX is a random

sample

Dy is a random

sample

DZ is a random

sample Ha DX is not a

random sample

DY is not a

random sample

DZ is not a

random sample

R 13 17 16

ZR -0,97 0,59 0,38

Z(1-α)/2 2,57 2,57 2,57

Table 3. Runs test applied to residual vector. 9

Therefore dX, dY and dZ are random samples.

About the Regarding Kolmogrov Smirnoff test, the parameters are shown in Table 4.10

dX dY dZ

Ho DX is normally

disdistributed

DY is normally

distributed

DZ is normally

distributed

Ha DX is not

normally

distributed

DY is not

normally

distributed

DZ is not

normally

distributed dN 0,07 0,06 0,06

Dα 0,18 0,18 0,18

Table 4. Kolmogrov Smirnoff test applied to residual vector.

dN is the test parameter and dα is the critical value for a 99 % of significance, therefore the

residual vector components are normally distributed.

9 Where R is the observed number of runs, µ is the expected number of runs and σ is the standard deviation of

the number of runs.

10 The parameters were calculated based on the text written by Rodriguez Avi (2008).






10/16

Finally, it is important to mention that 2 points were discarded, which were in the second

trigonometric quadrant.

4.2.2 Applying the Standard Test

The calculation procedure is simple and it included the determination of the value of the

NSSDAxy and NSSDAz statistic for a 95 % of significance.

In planimetry, NSSDAxy (95 %) = 1,7308 * RMSxy

In altimetry, NSSDAz (95 %) = 1,9600 * RMSz

From the data obtained, it can be said that:

Positional accuracy tested: 0,23 m horizontal and 1,12 m vertical at 95 % confidence level.

4.3 Relative positional quality

As absolute positional quality control, for the relative control it was compared

photogrammetric data with topographic data. The procedure applied is described in the

Standard 19138.

Instead of consider the coordinates of the checkpoints, in this method it is compared the

relative distances between each pair of checkpoints.

About planimetry:

Number of points: 29

Number of distances: 406

Initial standard deviation in the X component (σX): 0.10 m

Initial standard deviation in the Y component (σY): 0.11 m

In altimetry:

Number of points: 28

Number of distances: 378

Initial standard deviation in the Z component (σZ): 0.81

From the data obtained, it can be said that:

Relative accuracy tested: 0,25 m horizontal and 1,33 m vertical at 90 % confidence level.






11/16

5. ANALYSIS OF RESULTS

5.1 Residuals magnitude

The planimetric residual component can be classified as is shown in Table 5.

Number of points Reference color

dZ < 10 cm 14 Green

10 cm < dZ < 30

cm

13 Yellow

dZ < 30 cm 2 Red

Table 5. Planimetric residuals.

Analyzing these results can be said that only the 7% of the control points have residuals dXY

greater than 30 cm, whereas 48% of the points have residuals less than 10 cm. Figure 6 (b).

Furthermore, the quintile analysis shows that only 20% of the control points have residuals

greater than 16 cm and the 20% of the control points have residuals lower than 6 cm. Figure 6

(a).

Figure 6. Residuals magnitude analysis: planimetry.

On the other hand, the altimetric analysis was based on the classification of the control points

according to dZ. It shows that the three intervals defined have about the same amount of

points, which is associated with a wider sample range. Table 6 and Figure 7.

Number of

points

Reference color

dZ < 20 cm 9 Green

20 cm < dZ < 50 cm 9 Yellow

dZ < 50 cm 10 Red

Table 6. Altimetry residuals.

In general, the vertical residual magnitude is greater than the horizontal component of

residuals dXY.






12/16

This is also clearly seen on from the quintile analysis, showing that 60% of the points have

residuals greater than 30 cm.

Figure 7. Residuals magnitude analysis – altimetry.

5.2 Residuals spatial distribution

In the Figure 8, the residuals magnitudes were classified according to the values for each

point and the location of each point.

The dots symbolize the check points and the triangles represent the control points. Also the

polyline defined by the ground control points is represented in blue.

Figure 8. Spatial distribution of check points residuals. The left side is planimetric

classification result, the right is altimetric result.

It can be seen that most of the large-scale residuals are found outside the ground control point

polyline. This is completely consistent with photogrammetric theory as it is known that it can

be obtained better results applying interpolation instead of extrapolating.






13/16

To show this observation, the study area was subdivided into inside and outside the polyline,

and this classification is shown in the Figure 9.

Figure 9. Residuals magnitude in/out analysis: planimetry.

Figure 10. Residuals magnitude in/out analysis: altimetry.






14/16

6. CONCLUSIONS

As general conclusion it can be said that there some suggestions that can be applied to

improve the accuracy of the products obtained from a UAV´s flight.

The following list shows some topics that directly affect to the quality of the results obtained:

the distribution of the control points:

should cover all the area of interest, locating in the borders of it.

should be properly signalized in the ground before the flight.

when you designed a plan for the flight that is used to design the position of the

control points, this flight should be the one that is effectively flown.

should maximize the number of control points.

the relevance of obtaining a real calibration of the camara, which correspond with the

date of the flight.

Considering the results obtained in this research, these types of flights are useful for

applications in thematic like: cadastre, measurement of irregular settlements, rural mapping,

and agronomics activities, to calculate volumes and to supervise mining process.






15/16

REFERENCES

Ariza López, F.J. & James Atkinson, A.D. (2006). Metodologías de Control Posicional:

Visión General y Análisis Crítico. Universidad de Jaén.

CEFOCCA-UNSJ, Fotogrametría. Documento de Cátedra, Centro de Fotogrametría,

Cartografía y Catastro. Facultad de Ingeniería, Universidad Nacional de San Juan, Argentina.

IGN. Grupo A.2, Fotogrametría y Teledetección, Instituto Geográfico Nacional, Centro de

Información Geográfica, España.

Leica Geosystems Geospatial Imaging, LLC. (2006). Leica Photogrammetry Suite Project

Manager.

Lerma García, J.L. (1999). Aerotriangulación: Cálculo y Compensación de un Bloque

Fotogramétrico. Departamento de Ingeniería Cartográfica, Geodesia y Fotogrametría.

Universidad Politécnica de Valencia, España.

Lopez, M. & Thoss, M.. Aerial Mapping: Photogrammetry with the help of microdrones md4-

1000. Company Microdrones GmbH.

Minnesota Planning, Land Management Information Center, (1999). Positional Accuracy

Handbook. Using the National Standard for Spatial Data Accuracy to measure and report

geographic data quality

Muñoz, P. (2004). Apoyo aéreo cinemático y aerotriangulación digital frente a los sistemas de

navegación inercial: análisis de precisiones. Tesis Doctoral, Universidad Politécnica de

Madrid, Escuela Técnica Superior de Ingenieros Agrónomos.

Pérez Álvarez, J. A. (2001). Apuntes de Fotogrametría II. Universidad de Extremadura,

Centro Universitario de Mérida.

Pérez Álvarez, J. A. (2001). Apuntes de Fotogrametría III. Universidad de Extremadura,

Centro Universitario de Mérida.

Rodríguez Avi, J. (2009 – 2010). Título de experto universitario en evaluación de la calidad

de la información geográfica. España.

Sarría, F.A. Sistema de Información Geográfica. SIGMUR: SIG y Teledetección en la

Universidad de Murcia.

Silva Gayoso, R. (2008). Plan Cartográfico de Aragón. “Vuelo Fotogramétrico, Ortoimagen y

Cartografía Topográfica escala 1/5000”. Zaragoza.






16/16

BIOGRAPHICAL NOTES

Rosario Casanova, Director of the Surveying Institute of Engineering University of Uruguay.

She is Professor on the Geomatic’s Department of that Institute. Has a degree on Land

Surveyor Engineer and a Master in Land Planning and Local Development, also is preparing

her thesis for her PH Degree. She has published many papers in local and international events.

Melissa Pérez and Verónica Pampinella, have recently obtained their Land Surveyor

Degree, and are working for public institution and private company related with these topics.

CONTACTS

MSc. Ing.Agrim. Casanova, Rosario

Institution: Surveying Institute, Faculty of Engineering, University of Republic

Av. Herrera y Reissig 565

Montevideo

URUGUAY

Tel. +598.27110395

Fax + 598.271060161

Email: [email protected]

evaluating quality of uav´s products: study case uruguay · traditional photogrammetric product,...

Documents