Investigation of indoor and outdoor performance of two portable

mobile mapping systems

Erica Nocerinoa*, Fabio Mennaa, Fabio Remondinoa, Isabella Toschia,

Pablo Rodríguez-Gonzálvezb,c

a 3D Optical Metrology unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy

Email: <nocerino><fmenna><remondino><toschi>@fbk.eu, Web: http://3dom.fbk.eu

b TIDOP Research Group, Higher Polytechnic School of Ávila, University of Salamanca, Hornos

Caleros, 50, 05003, Ávila, Spain. Email: pablorgsf@usal.es

c Dept. of Mining Technology, Topography and Structures, University of León, Avda. Astorga, s/n,

24401, Ponferrada, León, Spain. Email: p.rodriguez@unileon.es

ABSTRACT

The paper investigates the performances of two portable mobile mapping systems (MMSs), the handheld GeoSLAM

ZEB-REVO and Leica Pegasus:Backpack, in two typical user-case scenarios: an indoor two-floors building and an

outdoor open city square. The indoor experiment is characterized by smooth and homogenous surfaces and reference

measurements are acquired with a time-of-flight (ToF) phase-shift laser scanner. The noise of the two MMSs is

estimated through the fitting of geometric primitives on simple constructive elements, such as horizontal and vertical

planes and cylindrical columns. Length measurement errors on different distances measured on the acquired point clouds

are also reported. The outdoor tests are compared against a MMSs mounted on a car and a robust statistical analysis,

entailing the estimation of both standard Gaussian and non-parametric estimators, is presented to assess the accuracy

potential of both portable systems.

Keywords: Mobile mapping system, backpack, handheld, accuracy, robust statistical analysis, length measurement error,

noise

1. INTRODUCTION

The ability of acquiring and recording precise, dense and geo-referenced 3D information is a constant request for a vast

variety of applications, ranging from civil engineering and construction to cultural heritage, from environment to

industry, etc. The most effective way to satisfy this need is represented by mobile mapping systems (MMSs), instruments

able to acquire 3D information on-the-move, using a moving platform. A MMS is a ‘compound’ system consisting of

three main components1: mapping sensors for the acquisition of 3D/2D data (point coordinates and/or images), a

positioning and navigation unit for spatial referencing, and a time referencing unit operating as central system for data

synchronization and integration.

Since their early development in the late 1980s, MMSs have been progressively improved in order to provide

increasingly more precise and denser data, acquired in shorter time. Besides the progresses in optical sensors, one of the

key advances in MMS is related to spatial referencing technology. While the very early applications were restricted to

environments where the sensor positions were computed using ground control2, thanks to advantages in satellite and

inertial technology, today spatial referencing is commonly possible in previously unknown and undiscovered places.

Also, the miniaturization and cost reduction of components have played a fundamental role in the spread of MMS,

allowing for more and more flexible, portable and low-cost systems.

* nocerino@fbk.eu; phone +39 0461 314507; http://3dom.fbk.eu

a)

b)

c)

d)

Figure 1. The devices under investigation: the GeoSLAM ZEB-REVO (a and b) and Leica Pegasus:Backpack (c and d).

The majority of the MMSs rely on light detection and ranging (LiDAR) sensors as 3D mapping unit, but are usually

equipped also with cameras for retrieving color information of the scene. Traditionally, MMSs are fitted into vehicles,

like vans, cars, planes, boats, etc., which represent the optimal choice for recording vast areas, but hardily permit the

acquisition of narrow passages and are ineffective for indoor scenarios. A more efficient alternative for 3D mapping in

complex and indoor scenarios, characterized by rough terrain, small obstacles, stairs, entails robotic platforms, which are

however still limited to the research domain3,4,5. More widespread are MMSs designed and adapted to be easily carried or

worn by persons walking through and, consequently, mapping the environment of interest. Nowadays, popular solutions

available on the market and among research laboratories are:

1. man-portable MMS backpacks: these systems usually feature a conventional positioning and navigation unit

integrating GNSS (global navigation satellite system) and IMU (inertial measurement unit) sensors (Akhka R26;

Leica Backpack:Pagasus7; ROBIN8); however, solutions with only IMU or even without positioning and

navigation components exist9 (HERON10). Some systems are also equipped with cameras providing panoramic

spherical video or still imagery (UltraCam Panther11);

2. handheld MMSs: the main difference from backpack solutions is that the user holds the scanning device in

hand. These systems are usually equipped with IMU based navigation unit (no GNSS), 2D laser profilometer

(iMS2D12, ZEB113, ZEB-REVO14), and may also feature a very wide angle camera (PX-8015, ZEB-CAM16);

3. trolley MMSs: in this configuration, mainly designed for indoor mapping applications the different sensors

(lidar, IMU, cameras, etc.) are fitted in a cart (iMS 3D17, Timms18).

Previous works presented accuracy evaluation of vehicle-based MMSs: (i) using an established urban test field focusing

on planimetric and elevation errors19, (ii) adopting reference values from an existing 3D city model20, (iii) in terms of

target representation21, (iv) through robust statistical assessment with respect to photogrammetry and terrestrial laser

scanner (TLS) data23,24. Backpack MMSs were also tested: (i) in outdoor scenarios, using as reference data 3D points

collected with unmanned aerial vehicle (UAV) equipped with a laser scanner24, reporting planimetric and elevation

errors with respect to a TLS25; (ii) in indoor environment, employing a high-precision laser-based positioning and

tracking system26.

1.1 Paper overview

The aim of the article is to investigate the performance of two portable MMS: the GeoSLAM ZEB-REVO (Figure 1a and

1b) and the Leica Pegasus:Backpack (Figure 1c and 1d). Both systems, described in section 2, are evaluated indoor

(section 3) and outdoor (section 4). In the first test, two floors of a building are surveyed with the two portable MMS and

independent check measurements are acquired using a ToF (time-of-flight) phase-shift TLS (Leica HDS700027). The

indoor scene, characterized by smooth and homogenous surfaces, as well as constructive elements like columns, is also

used to derive meaningful information about noise on horizontal and vertical planes, along with fitting of geometric

primitives for the two systems. The outdoor experiment is carried out in an 80 m by 70 m historical city square, where

the two portable MMS are compared against a classical van-based MMS (RIEGL VMX-45028) whose accuracy potential

is discussed by Toschi et al22. A robust statistical analysis is performed to evaluate the portable MMSs geometric

performance in an outdoor environment.

2. THE TWO PORTABLE MMS

2.1 The GeoSLAM ZEB-REVO device

The GeoSLAM ZEB-REVO is a lightweight portable MMS, commercialized by the GeoSLAM company. The

instrument is the evolution of the ZEB1 device, commercial version of the ZEBedee handheld 3D range sensor

developed by the Autonomous Systems Laboratory, CSIRO ICT Centre in Brisbane (Australia)29. The main difference

between the two devices is that the ZEB1 is equipped with a spring that allows the scan head to nod, or oscillate both in a

‘front-back’ and ‘side-by-side’ direction; in the updated version of the system, the ZEB-REVO, the spring is no longer

present and it has been replaced by an automatic rotating head.

The instrument features a 2D infrared laser scanner profilometer (UTM-30LX) coupled with an IMU sensor, without

GNSS receiver. The 2D laser profilometer is a compact laser scanning range finder (LRSF), which consume less power,

is more compact and light-weight than a classic 3D laser scanner system30. The UTM-30LX emits pulsed light beams in

the near infrared, with a wave length of 905 nm; the travelling time of the pulse from the sensor to the object and back

provides the range measurement, according to the well-known time-of-flight measurement principle. The laser source

scans a 270° semicircular field, so that the coordinates of the recorded points are calculated using the measured distance

and laser pulse step angle31. The industrial-grade microelectromechanical (MEMS) IMU is mounted beneath the scanner

and consists of triaxial gyros and accelerometers, providing measurements of angular velocities and linear accelerations

that, combined with the laser data, allow to estimate the sensor trajectory. The IMU also contains a three-axis

magnetometer to reduce environmental magnetic interference32.

The scanning head, consisting of the laser and IMU sensors, is connected to a backpack containing the battery and a

data-logger unit, where the acquired data are stored in real-time.

The 2D laser profiles are aligned through a 3D SLAM (simultaneous localization and mapping) approach that estimates

the six degree of freedom (6DoF) of the sensor head motion, and generates the 3D point cloud of the scene (i.e. the map).

Details of the implemented SLAM approach are provided in29,33.

Recently, the ZEB-REVO has been equipped with a GoPro camera. The authors haven’t had the opportunity to test the

new version of the device yet; however, according to the specifications from the manufacturer16, the camera is intended

to provide imagery co-registered with the scanning data, but not to enrich the 3D laser point cloud with RGB

information. The authors developed an in-house method to map color information to the 3D data acquired with the

GeoSALM devices based on two GoPro cameras attached to the scan head (Figure 2a). The images from the cameras are

processed through a photogrammetric workflow and the obtained RGB information is mapped onto the ZEB point cloud

(Figure 2b and 2c).

a)

b)

c)

Figure 2. An in-house method is developed to map color information based on two GoPro cameras fixed on the GeoSLAM ZEB1

device (a). The original (b) and colored (c) 3D point clouds.

The proper acquisition approach with both GeoSLAM instruments requires to leave them stationary on a horizontal,

planar surface for the so-called initialization procedure, which allows to define a local coordinate reference system with

the vertical axis perpendicular to this planar surface. Then the user preforms the acquisition walking through the scene

and, at the end, must go back to the initial position to close the acquisition loop.

The collected data are post-processed through proprietary software applications, either running on local machine or via

cloud processing. The data presented in this contribution were processed via the desktop software application (version

v1), where the processing was totally automatic and no user intervention or alignment refinement was possible.

Table 1. Technical specifications of the tested portable MMSs. (*) An optional GoPro camera is now available.

GeoSLAM ZEB-REVO Leica Pegasus:Backpack

Max

range

indoor 30 m 50 m

outdoor 15-20 m

# Laser scanners 1 2

Measuring principle ToF ToF

Scanner data acquisition rate 43,200 points/sec 300,000 points/sec per scanner

Scanner

Resolution

horizontal 0.625° 0.1°-0.4°

vertical 1.8° 2.0°

Scanner angular FOV 270° x 360° 360° x 30° per scanner

Channels 1 16 per scanner

Laser wavelength 905 nm 903 nm

Scanner line speed 100 Hz 5-20 Hz

Laser head rotation speed 0.5 Hz -

Scanner weight 1.0 kg 0.83 kg per scanner

Scanner dimensions 86 x 113 x 287 mm 72 x 72 x 72 mm per scanner

# Cameras 0 (*) 5

CCD / pixel size - 2046 x 2046 / 5.5 um x 5.5 um per camera

Focal length - 6.0 mm

Cameras total angular FOV - 360° x 200°

Cameras max frame rate - 8 Hz

IMU type MEMS FOG

GNSS - Triple band, single and dual antenna support

Relative accuracy 2-3 cm 3-5 cm

Absolute

position

accuracy

indoor 3-30 cm

(10 mins scanning, 1 loop)

5-50 cm

(10 mins scanning, minimum 3 loop closures

or double passes conditions)

outdoor 5 cm

Tot system weight 4.1 kg 11.9 kg

Backpack dimensions 220 mm x 180 mm x 470 mm 310 mm x 270 mm x 730 mm

Operating time 4 hours 3 hours, up to 6 hours with optional batteries

2.2 The Leica Pegasus:Backpack

The Leica Pegasus:Backpack (summer 2016 version) combines two laser profilometers (Velodyne PUCKTM VLP-16)

synchronized with five cameras, a triple band GNSS receiver (NovAtel ProPak6™), and a fiber optic gyroscope (FOG)

IMU.

Each VLP-16 features 16 laser/detector pairs, mounted in a rotating housing providing 360° field of view34. The device is

also equipped with five high-dynamic-range cameras, placed to acquire 360°x200° field of view. The cameras provide

images co-registered with the laser data and RGB color information projected to the point cloud.

The full inertial navigation system (INS) is composed of GNSS and IMU sensors; the absolute and outdoor positioning is

delivered by GNSS, while indoor positioning and in GNSS-denied environments is based on the IMU and a SLAM

algorithm. The integrated IMU is a FOG type, which is usually more costly and of slightly higher performance than

MEMS systems for standalone INS performance35.

The acquired data, i.e. laser profiles and spherical images, are automatically processed via the proprietary software

application; however, the user may refine the alignment by either importing ground control points (GCPs) or identifying

3D tie points on the point clouds. A plug-in working in ArcMAP, component of the geospatial processing toolkit ArcGIS

by Esri, is available to visualize and analyze the data.

To assure a fair investigation of the two MMSs, the Leica Pegasus:Backpack data presented in this study are processed

following an automatic approach with minimal user intervention. Evaluating the system calibration and adjustment

procedures of the collected raw data is out of the scope of the tests here described.

3. INDOOR PERFORMANCE EVALUATION

The method adopted to test the indoor performances of the two portable MMSs is based on two measures for quality

assessment, i.e. the root mean square (RMS) of the residuals from fitting of geometry primitives (section 3.1) and length

comparison (section 3.2).

The scanning operations in the building (Figure 3) are carried out with the two MMSs preforming the initialization

procedure outside the building and following a close-loop acquisition path.

Figure 3. The two-floor building, test area for the indoor performance evaluation. Left: a picture of the interior. Right: cross section of

the ZEB-REVO point cloud.

3.1 Reference data and alignment of point clouds

To evaluate the indoor accuracy potential, a single point cloud, consisting of a great portion of the second floor and part

of the first floor, is acquired with a phase-shift ToF terrestrial laser scanner (TLS), the Leica HDS7000, whose technical

specifications are reported in Table 2. Only one point cloud is used as refence to avoid any additional uncertainty that

might be caused, for example, by uncontrolled residual errors in the registration process of several point clouds.

Figure 4. Up: top view of second floor walls; down: perspective view of a section of the two floors. In white the HDS7000 point

cloud, in cyan the ZEB-REVO, in yellow the Pegasus.

The point clouds acquired with the three different devices are aligned in the same coordinate reference system, defined

as follows: the z axis coincides with the gravity vector, i.e. the vertical direction provided by leveling the HSD7000 with

the digital bubble; the x axis runs along the width of the floors; the y lays along the length. Some views of the aligned

point clouds are shown in Figure 4.

Table 2. Technical specifications of the Leica HDS7000 TLS.

Measuring principle phase-shift ToF

Max range 187 m (ambiguity interval)

Data acquisition rate Up to 1,016,727 points/sec

Scanner resolution (horizontal/vertical) 0.3°-0.004°

Scanner angular FOV 360° x 320°

Laser wavelength 1500 nm

Range noise (on a black 14% target, i.e. worst condition) 0.5 / 2.7 mm rms @ 10 / 50 m

Linearity error ≤ 1 mm

Angular accuracy (horizontal/vertical) 127 urad

3.2 Noise estimation

The device’s noise is evaluated through the fitting of geometric primitives (planes and cylinders) to the acquired point

clouds: vertical planes are fitted on the walls, horizontal planes on the floors and ceilings, cylinders on the columns.

Table 3 reports the RMS results. As expected, the HDS7000 greatly outperforms the two MMSs, which anyway show

RMS values significantly within the levels declared by the vendors (see the ‘relative accuracy’ row in Table 1).

Table 3. Noise estimation: RMS of fitting procedures with outliers removal.

HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack

Vertical planes 0.1 cm 1.1 cm 1.9 cm

Horizontal planes 0.2 cm 0.9 cm 1.2 cm

Cylindrical columns 0.1 cm 0.9 cm 1.8 cm

3.3 Length measurements

The positional or coordinate standard error (SXYZ), theoretical length measurement error (TLME) and relative theoretical

length measurement accuracy (RTLMA) are reported in Table 4.

The SXYZ for the two MMSs are derived from the datasheet, i.e. the ‘absolute position accuracy’ in Table 1, whilst for the

HDS7000 is computed taking into account the angular accuracy, linearity error and range noise for two distances (10 m

and 50 m) in the worst condition (laser reflected from a 14% black surface, Table 2).

The TMLE and RTLMA are, respectively, calculated according to (1)36 and to (2), for two distance values D equal to 3

m and 50 m, considering the best and worst positional standard errors.

XYZs 23TLME (1)

TLME:1RTLMA

DROUND (2)

Table 4. Positional or coordinate standard error (SXYZ), theoretical length measurement error (TLME) and relative theoretical length

measurement accuracy (RTLMA). The value marked with () are taken from the instrument datasheet (Table 1 - ‘absolute position

accuracy indoor’).

HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack Min Max Min Max Min Max

XYZs 0.1 cm @ 10 m 0.5 cm @ 50 m 3 cm 30 cm 5 cm () 50 cm ()

TLME 0.6 cm 2.3 cm 12.7 cm 127.3 cm 21.2 cm 212.1 cm

RTLMA D = 3 m 1:500 1:150 1:25 1:2 1:15 1:1

D = 50 m 1:9000 1:2200 1:500 1:50 1:250 1:25

The theoretical length measurement accuracy of the HDS7000 is from one to two orders of magnitude better than the two

portable MMS, and increases, as expected, with the distance.

Figure 5: Distances measured for the indoor performance evaluation.

Figure 5 depicts the distance measurements extracted from the point clouds of the two-floor building. The length, L, two

width values, B1 and B2, and one height, H1, are measured on the second floor; the height H2 is measured from the first

floor to the ceiling of the second.

Table 5 reports the analysis performed on the measured distances. Each value is the mean of eight distance

measurements between two planes fitted on the opposite walls (L, B1 and B2), and between floor and ceiling (H1 and

H2). The relative length measurement error (RLME) is computed according to (3) as the relative difference between the

measured distance Dm for the GeoSLAM ZEB-REVO and Leica Pegasus:Backpack, and the distance from the HSD7000,

assumed as reference length Dr. The relative length measurement accuracy (RLMA) is defined as the rounded absolute

reciprocal value of the RLME times 100 (4).

100RLME

r

rm

D

DD (3)

rm

r

DD

DROUNDROUND :1

RLME

100:1RLMA (4)

The RLMA results always better than the computed theoretical accuracies (Table 4); in particular, the GeoSLAM ZEB-

REVO provides distance measurements that are closer to the HDS7000 than the Leica Pegasus:Backpack, which also

features higher standard deviation values (σ).

Table 5. Indoor performance evaluation: measured distances with standard deviations (σ), relative length measurement errors

(RMLE) and accuracies (RMLA).

HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack

Dr [m] σ [cm] Dm [m] σ [cm] RLME RLMA Dm [m] σ [cm] RLME RLMA

L 45.112 0.4 cm 45.135 0.7 0.051% 1:2000 45.133 0.9 0.047% 1:2000

B1 14.888 0.5 cm 14.906 0.6 0.115% 1:900 15.227 8.3 2.274% 1:50

B2 14.836 0.3 cm 14.851 2.0 0.103% 1:1000 14.925 0.8 0.600% 1:150

H1 3.021 0.2 cm 3.021 0.3 -0.001% 1:7000 3.094 7.3 2.416% 1:50

H2 7.666 0.5 cm 7.659 0.6 -0.087% 1:1000 7.814 1.7 1.941% 1:50

4. OUTDOOR PERFORMANCE EVALUATION

The outdoor tests are performed in the cathedral square in Trento (Figure 6a). The site represents a challenging scenario

for SLAM based systems due to moving objects, people, cars, trucks, buses, and the façades geometry. The city is

surrounded by high mountains that might constitute an unfavorable environment also for GNSS based systems, although

the square itself is quite big (80 m x 70 m). It is worth noting the evident blunders in the GeoSLAM ZEB-REVO point

cloud (Figure 6c, highlighted in yellow): the fountain and tree are duplicated, clearly due to huge error in the alignment

of the laser profiles.

The point clouds acquired with the two portable MMSs (Figure 6c and 6d) are compared through a robust statistical

analysis with data derived from the RIEGL VMX-450 MMS mounted on a van.

a) b)

c) d)

Figure 6. The cathedral square in Trento (Italy), test area for the outdoor performance evaluation (a). RIEGL VMX-450 MMS colored

point cloud of the square (b); GeoSLAM ZEB-REVO point cloud (c); Leica Pegasus:Backpack colored point cloud (d).

4.1 Reference data

The RIEGL VMX-450 MMS platform (Table 6) integrates two synchronously operated VQ-450 laser scanners, a

portable control unit (VMX-450-CU) and IMU/GNSS navigation hardware. The system measures up to 1.1 million

points and 4 profiles per second, with an online waveform processing that allows to penetrate obstructions, such as

fences and vegetation. The platform is also equipped with the modular VMX-450-CS6 camera system, where up to six

industrial digital color cameras can be integrated. The cameras complement the acquisition of geometric data from the

laser scanners with time-stamped images. A detailed performance investigation of the RIEGL VMX-450 MMS is

presented in Toschi et al22.

Table 6. Technical specifications of the RIEGL VMX-450 system.

# Laser scanners 2

Measuring principle phase-shift ToF

Max range 800 m

Data acquisition rate 550,000 points/sec per scanner

Scanner angular FOV up to 360°

Laser wavelength near infrared

# Cameras up to 6

CCD / pixel size 2452 x 2056 / 3.45 um x 3.45 um per camera

Focal length 5 mm

Cameras angular FOV 80° x 65° per camera

Relative position accuracy 1 cm

Absolute position accuracy 2-5 cm

Tot system weight > 100 kg

4.2 Alignment and signed distance computation

Figure 7. Color-coded map of the signed distances computed between the GeoSLAM ZEB-REVO (left) and Leica Pegasus:Backpack

(right) point clouds and the RIEGL VMX-450 data. The differences are in m.

To evaluate the accuracy potential of the two portable MMSs against the reference data, the three point clouds are first

cleaned from noisy elements, such as pedestrians, vegetation, cars, etc. Due to the blunder in the ZEB-REVO point cloud

(Figure 6c), the fountain is also removed; consequently, only the building façades overlooking the square are used in the

performance evaluation.

The blunder-free point clouds from GeoSLAM ZEB-REVO and Leica Pegasus:Backpack are aligned in a local

coordinate reference systems to the cleaned RIEGL VMX-450 data by means of the iterative closest point (ICP)

registration method implemented in the open source software application CloudCompare v2.8

(http://www.cloudcompare.org/). The signed distances between the portable MMSs and RIEGL VMX-450 point clouds

are then computed, using the CloudCompare M3C2 plugin, which implements the Multiscale Model to Model Cloud

Comparison method37. It allows a direct comparison of 3D points, without the need of a preliminary meshing or gridding

phase. If the data do not contain normal vectors, they are estimated on the basis of the local surface roughness. Then, for

each 3D point, the local distance between the two clouds is computed. The color-coded maps of the signed distances for

both the portable MMMs are shown in Figure 7.

4.3 Robust statistical analysis

ZE

B-R

EV

O

VS

RIE

GL

VM

X-4

50

Peg

asu

s:B

ack

pack

VS

RIE

GL

VM

X-4

50

Figure 8. Histograms of the signed differences with the superimposed curve for the normal distribution (left). Q-Q plots of the

distribution of the signed differences (right).

Several studies38,39,40 have demonstrated that in the accuracy assessment of data provided by laser scanner systems, as

well as photogrammetry, the hypothesis that errors follow a Gaussian distribution is hardly verified. This might be due to

the presence of residual system errors, but also unwanted objects not correctly filtered out from the data. In the following

analyses, the hypotheses (i) that many outliers will exist in comparing data provided by different instruments and (ii) that

the normality assumption of distribution of the differences is not valid, are first verified. Then, suitable accuracy

measures are computed and reported in Table 7.

Two visual diagnostic tests are reported to test the normality assumption (Figure 8), i.e. the histogram of the signed

differences with the superimposed curve for the normal distribution, and the quantile-quantile (Q-Q) plot of the

distribution of the signed differences38. The Q-Q plot depicts the quantiles of the empirical distribution plotted against

the theoretical quantiles of the normal distribution. If the actual distribution is normal, the Q-Q plot should provide a

straight line. Big deviation from the straight line indicates that the distribution of the errors is not normal.

Both graphical evaluations show that the differences do not follow a Gaussian distribution: the Q-Q plot relative to the

GeoSLAM ZEB-REVO comparison shows a shape significantly far away from the theoretical normal hypothesis, while

the histogram for the Leica Pegasus:Backpack a two-peaked distribution with a quite high dispersion.

The previous conclusion is also supported by the computed kurtosis (5) and skewness (6) values, which represent,

respectively, a measure of whether the data are peaked or flat with respect to a normal distribution, and an indication of

departure from symmetry in a distribution (asymmetry around the mean value):

3

1 Kurtosis

41

4

n

xn

ii

(5)

31

3

1 Skewness

n

xn

ii

(6)

Where n, µ and σ are the sample size (i.e. number of data points x), mean and standard deviation. When the sample

distribution follows the normal hypothesis, both values should be equal to zero. The distance distribution for the

GeoSLAM ZEB-REVO reveals a highly peaked shape, while the distribution for the Leica Pegasus:Backpack results

slightly more asymmetric.

In both cases, the distribution of the differences discloses kurtosis, skewness, hence a significant amount of outliers;

consequently the classical µ and σ parameters are not adequate to provide the accuracy measures for the two portable

MMSs. When this is the case, other non-parametric estimators are to be adopted, such as the median m, normalized

median absolute deviation – NMAD (7) and the square root of the biweight midvariance – BWMV (8):

MAD4826.1 NMAD (7)

2

1

22

1

422

511

1BWMV

n

iiii

n

iiii

UUa

Umxan (8)

1,0

1,1

i

ii

Uif

Uifa (9)

MAD9

mxU i (10)

being the median absolute deviation – MAD (9), i.e. the median (m) of the absolute deviations from the data’s median

(mx):

xi mxm MAD (11)

The computed accuracy measures are summarized in Table 7. The value of the median is closer to the mean for the

GeoSLAM ZEB-REVO, while is smaller for the Leica Pegasus:Backpack. The values are inside the expected a-priori

error of the point clouds analysis (Table 1); they are likely to represent the residual of the registration process. In both the

cases, the values of the median are smaller than the absolute position accuracy quoted in the technical sheets (Table 1).

Concerning the values of the standard deviations, they are higher than the NMAD and BWMV, and lower than the

specifications from the vendors. In accordance to the visual analysis of the histograms (Figure 9), the values for the

Leica Pegasus:Backpack indicate a wider dispersion than the GeoSLAM ZEB-REVO.

Since the data sample does not follow a Gaussian distribution, the standard deviation values are affected, so it does not

represent correctly the error dispersion. It appears clearly in the case of GeoSLAM ZEB-REVO when is compared with

the robust measures of the dispersion (NMAD and square root of BWMV), being undervalued (more than two times).

Please note also, the asymmetry of the error distribution, being not possible provide a plus-minus range, but an absolute

interpercentile range. In the case of Table 7, it is computed according to the real error distribution. They (dispersion

measurement and error distribution function) are related, showing for the Leica Pegasus:Backpack a wider dispersion

than the GeoSLAM ZEB-REVO, in according to the visual analysis of the histograms (Figure 9). Moreover, in the

reported interpercentile range at 95 % of confidence level, both portable MMS differs from their relative declared

position precision. As a remark, the 50 % of the observed points (interpercentile range at 50% of confidence level) are in

a 3.0 cm size interval for the GeoSLAM ZEB-REVO, and 13.5 cm for Leica Pegasus:Backpack, while the 95% of data

points (interpercentile range at 95% of confidence level) are within 30 cm and 51 cm for the GeoSLAM ZEB-REVO and

Leica Pegasus:Backpack (respectively).

Table 7. Statistical analysis of the signed distance computation.

ZEB-REVO

VS

RIEGL VMX-450

Pegasus:Backpack

VS

RIEGL VMX-450

Sample size n 4,061,074 1,0351,184

Gaussian

assessment

Kurtosis 19.331 1.157

Skewness 0.079 0.093

Sample mean µ 2.0 cm 1.9 cm

Standard deviation σ 7.1 cm 13.3 cm

Robust

assessment

Median 2.0 cm 1.0 cm

NMAD 2.2 cm 9.6 cm

Sqrt(BWMV) 2.9 cm 12.5 cm

Interpercentile range 50% 3.0 cm 13.5 cm

Percentile 0.025 -9.7 cm -21.8 cm

Percentile 0.975 14.3 cm 29.1 cm

Interpercentile range 95% 23.9 cm 50.9 cm

5. CONCLUSIONS

The presented investigation aimed at evaluating the performance of two portable MMSs, the handheld GeoSLAM ZEB-

REVO and Leica Pegasus:Backpack, in indoor and outdoor scenarios. The tests were designed to specifically address

relevant issues related to mapping environments, such as building interiors or complex city parts, which represent typical

applications for such systems. Consequently, the analyses were performed in order to estimate the magnitude of errors in

measuring distances and acquiring 3D data. To this end, the data provided by the two MMSs were compared against

those acquired by two different reference devices, a TLS for the indoor and a van-based MMS for the outdoor.

While indoor the performance assessment was carried out reporting noise evaluation and length measurement errors, a

robust statistical analysis was performed on data acquired outdoor. In both scenarios, the two portable MMSs performed

within the accuracy specifications provided by the vendors, with the GeoSLAM ZEB-REVO generally outperforming

the declared values despite the evident gross error in the point cloud acquired outdoor. Worth to note is that errors in

Leica Pegasus:Backpack data might be further mitigated by including control points and constraints within the post-

processing adjustment.

The results and statistical analyses presented in this paper were achieved using instruments tested during the summer

2016. Other performances could be expected using new releases of the instruments.

ACKNOWLEDGMENTS

Authors are thankful to Dr. Nadia Guardini (ME.S.A. srl), Mr. Marco Formentini and Mr Simone Oppici (Leica

Geosystem Italy) for giving the possibility to use and test the two instruments investigated in this article.

REFERENCES

[1] Puente, I., González-Jorge, H., Martínez-Sánchez, J. and Arias, P., "Review of mobile mapping and surveying

technologies," Measurement 46(7), pp. 2127-2145 (2013).

[2] Schwarz, K.P. and El-Sheimy, N., "Mobile mapping systems–state of the art and future trends, "International

Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 35(Part B), p.10 (2004).

[3] Surmann, H., Nchter, A., Lingemann, K. and Hertzberg, J., "Kurt3D-an autonomous mobile robot for modelling

the world in 3D," ERCIM NEWS, 55, pp.24-25 (2003).

[4] Nüchter, A., Elseberg, J. and Borrmann, D., "Irma3D—An Intelligent Robot for Mapping Applications," IFAC

Proceedings Volumes, 46(29), pp.119-124 (2013).

[5] Ziparo, V.A., Zaratti, M., Grisetti, G., Bonanni, T.M., Serafin, J., Di Cicco, M., Proesmans, M., Van Gool, L.,

Vysotska, O., Bogoslavskyi, I. and Stachniss, C., "Exploration and mapping of catacombs with mobile robots,"

In Safety, Security, and Rescue Robotics (SSRR), 2013 IEEE International Symposium on, pp. 1-2 (2013).

[6] Centre of Excellence in Laser Scanning Research, http://laserscanning.fi/laser-scanner-in-a-backpack/, (April,

2017).

[7] Leica, http://leica-geosystems.com/products/mobile-sensor-platforms/capture-platforms/leica-pegasus-

backpack, (April, 2017).

[8] ROBIN, Three-in-one LiDAR mapping system, 3D Laser Mapping, < http://www.3dlasermapping.com/robin/>

(April, 2017).

[9] Nüchter, A., Borrmann, D., Koch, P., Kühn, M. and May, S., "A man-portable, IMU-free mobile mapping

system," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences II-3 W, 5,

pp.17-23 (2015).

[10] Gexcel, http://www.gexcel.it/it/soluzioni/heron, (April, 2017).

[11] Vexcel Imaging, http://www.vexcel-imaging.com/products/ultracam-panther/, (April, 2017).

[12] VIAMETRIS, http://www.viametris.com/wcms/file/323-VIAMETRIS_-_iMS2D_v1b_Leaflet_EN_-_Light.pdf,

(April, 2017).

[13] Geo-SLAM, http://geoslam.com/hardware-products/zeb1/, (April, 2017).

[14] Geo-SLAM, http://geoslam.com/hardware-products/zeb-revo/, (April, 2017).

[15] Paracosm, http://labs.paracosm.io/hubfs/Paracosm-PX-80-Specs.pdf, (April, 2017).

[16] GeoSLAM, http://geoslam.com/wp-content/uploads/2016/12/ZEB-CAM-Brochure-v1.pdf, (April, 2017).

[17] VIAMETRIS, viametris.info/iMS3D/EN/, (April, 2017).

[18] Applanix, https://www.applanix.com/products/timms-indoor-mapping.htm, (April, 2017).

[19] Kaartinen, H., Hyyppä, J., Kukko, A., Jaakkola, A. and Hyyppä, H., "Benchmarking the performance of mobile

laser scanning systems using a permanent test field," Sensors, 12(9), pp.12814-12835 (2012).

[20] Haala, N., Peter, M., Kremer, J. and Hunter, G., "Mobile LiDAR mapping for 3D point cloud collection in

urban areas—A performance test," The International Archives of Photogrammetry, Remote Sensing and Spatial

Information Sciences, 37, pp.1119-1127 (2008).

[21] Lin, Y., Hyyppä, J., Kaartinen, H. and Kukko, A., "Performance analysis of mobile laser scanning systems in

target representation," Remote Sensing, 5(7), pp.3140-3155, (2013).

[22] Toschi, I., Rodríguez-Gonzálvez, P., Remondino, F., Minto, S., Orlandini, S. and Fuller, A., "Accuracy

evaluation of a mobile mapping system with advanced statistical methods," The International Archives of

Photogrammetry, Remote Sensing and Spatial Information Sciences, 40(5), p.245 (2015).

[23] Rodríguez-Gonzálvez, P., Jiménez Fernández-Palacios, B., Muñoz-Nieto, Á.L., Arias-Sanchez, P. and

Gonzalez-Aguilera, D., "Mobile LiDAR System: New Possibilities for the Documentation and Dissemination of

Large Cultural Heritage Sites," Remote Sensing, 9(3), p.189 (2017).

[24] Rönnholm, P., Liang, X., Kukko, A., Jaakkola, A. and Hyyppä, J., "Quality analysis and correction of mobile

backpack laser scanning data," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information

Sciences, pp.41-47 (2016).

[25] Glennie, C., Brooks, B., Ericksen, T., Hauser, D., Hudnut, K., Foster, J. and Avery, J., "Compact multipurpose

mobile laser scanning system—Initial tests and results," Remote Sensing, 5(2), pp.521-538 (2013).

[26] Lauterbach, H.A., Borrmann, D., Heß, R., Eck, D., Schilling, K. and Nüchter, A., "Evaluation of a backpack-

mounted 3D mobile scanning system," Remote Sensing, 7(10), pp.13753-13781 (2015).

[27] Leica Geosystems, http://hds.leica-geosystems.com/downloads123/hds/general/HDS7000/manuals/

HDS7000_UserManual_en.pdf, (April 2017).

[28] RIEGL, http://www.riegl.com/uploads/tx_pxpriegldownloads/DataSheet_VMX-450_2015-03-19.pdf, (April

2017).

[29] Bosse, M., Zlot, R. and Flick, P., "ZEBedee: Design of a spring-mounted 3-d range sensor with application to

mobile mapping," IEEE Transactions on Robotics, 28(5), pp.1104-1119 (2012).

[30] Mader D, Westfeld P and Maas HG, "An integrated flexible self-calibration approach for 2D laser scanning

range finders applied to the Hokuyo UTM-30LX-EW," The International Archives of Photogrammetry, Remote

Sensing and Spatial Information Sciences, 385 (2014).

[31] Hokuyo, http://www.hokuyo-aut.jp/02sensor/07scanner/download/pdf/UTM-30LX_spec_en.pdf, (April 2017)

[32] Zlot, R. and Bosse, M., "Three-dimensional mobile mapping of caves," Journal of Cave and Karst Studies,

76(3), p.191 (2014a).

[33] Zlot, R. and Bosse, M., "Efficient Large-scale Three-dimensional Mobile Mapping for Underground Mines,"

Journal of Field Robotics 31(5), pp.758-779 (2014b).

[34] Velodyne Lidar, http://velodynelidar.com/docs/manuals/63-9243%20Rev%20B%20User%20Manual%20and

%20Programming%20Guide,VLP-16.pdf (April 2017)

[35] Goodall, C., Carmichael, S. and Scannell, B., "The battle between MEMS and FOGs for precision guidance."

Analog Devices Technical Article MS-2432 (2013).

[36] Luhmann, T., Robson, S., Kyle, S. and Boehm, J., [Close-range photogrammetry and 3D imaging]. Walter de

Gruyter (2014).

[37] Lague, D., Brodu, N. and Leroux, J., "Accurate 3D comparison of complex topography with terrestrial laser

scanner: Application to the Rangitikei canyon (NZ)," ISPRS Journal of Photogrammetry and Remote Sensing,

82, pp.10-26 (2013).

[38] Höhle, J. and Höhle, M., "Accuracy assessment of digital elevation models by means of robust statistical

method," ISPRS Journal of Photogrammetry and Remote Sensing, 64(4), pp.398-406 (2009).

[39] Hasan, A., Pilesjö, P. and Persson, A., "The use of LIDAR as a data source for digital elevation models–a study

of the relationship between the accuracy of digital elevation models and topographical attributes in northern

peatlands," Hydrology and Earth System Sciences Discussions, 8(3), pp.5497-5522 (2011).

[40] Rodríguez-Gonzálvez, P., Garcia-Gago, J., Gomez-Lahoz, J. and González-Aguilera, D., "Confronting passive

and active sensors with non-Gaussian statistics," Sensors, 14(8), pp.13759-13777 (2014).