a unifying view of geometry, semantics, and data...

A Unifying View of Geometry, Semantics, and Data Association in SLAM∗

Nikolay Atanasov1, Sean L. Bowman2, Kostas Daniilidis2 and George J. Pappas21 Electrical and Computer Engineering, University of California, San Diego, CA 92093, USA

2 GRASP Laboratory, University of Pennsylvania, PA 19104, [email protected], [email protected], [email protected], [email protected]

Abstract

Traditional approaches for simultaneous localiza-tion and mapping (SLAM) rely on geometric fea-tures such as points, lines, and planes to infer theenvironment structure. They make hard decisionsabout the (data) association between observed fea-tures and mapped landmarks to update the envi-ronment model. This paper makes two contribu-tions to the state of the art in SLAM. First, it gen-eralizes the purely geometric model by introduc-ing semantically meaningful objects, represented asstructured models of mid-level part features. Sec-ond, instead of making hard, potentially wrong as-sociations between semantic features and objects, itshows that SLAM inference can be performed effi-ciently with probabilistic data association. The ap-proach not only allows building meaningful maps(containing doors, chairs, cars, etc.) but also offerssignificant advantages in ambiguous environments.

1 IntroductionThis paper bridges the gap between the advances in SLAM,relying on purely geometric information, and those in vi-sual recognition, relying on purely semantic information. Wepresent an abridged version of [Bowman et al., 2017] butalso state key results leading to a unified view of geome-try, semantics, and data association in SLAM. SLAM is theproblem of estimating the motion of a sensor system, whilesimultaneously building a map of the environment, using adense occupancy representation or a sparse set of landmarks(e.g., corner, edge, or plane features). A comprehensivesurvey can be found in [Cadena et al., 2016]. The prob-lem is closely related to structure from motion (SfM) [Maet al., 2012] with a moot distinction that SfM is typicallyan offline process relying on camera measurements, whileSLAM is an online operation using heterogeneous sensorsfound on mobile robots, including inertial measurement unit(IMU), camera, and LIDAR. The problem is also related tovisual-inertial odometry (VIO) [Mourikis and Roumeliotis,2006], which uses geometric features to infer the sensor’s

∗Supported by ARL DCIST CRA W911NF-17-2-0181.

Figure 1: This paper proposes a SLAM approach that uses inertial,geometric, and semantic measurements (top left) to reconstruct thesensor trajectory (blue), detect objects of interest (cars), and estimatethe positions of their parts (green points, e.g., doors, wheels).

motion but does not construct a global map. SLAM ap-proaches went through several transformations, starting withparticle and Kalman filtering [Thrun et al., 2005] and con-verging to iterative nonlinear optimization over the wholesensor trajectory [Strasdat et al., 2010]. Current approachesmodel the problem as a (factor) graph of sensor and land-mark states connected via edges capturing measurement con-straints [Kummerle et al., 2011; Kaess et al., 2012]. Theycan track visual-inertial systems over long trajectories inreal time [Bloesch et al., 2015; Forster et al., 2016; Qin etal., 2017], while recovering the sparse [Engel et al., 2014;Mur-Artal and Tardos, 2017] or dense [Whelan et al., 2016;Hornung et al., 2013] environment structure. Surprisingly,most methods rely only on geometry and occurred in isola-tion from the impressive results in object recognition based onstructured models [Felzenszwalb et al., 2010] and deep neu-ral networks [Krizhevsky et al., 2012; Girshick et al., 2014;Simonyan and Zisserman, 2014; He et al., 2016]. Our goalis to provide a meaningful and efficient environment repre-sentation in SLAM by taking advantage of object recognitionmethods. A recent survey on semantic mapping can be foundin [Kostavelis and Gasteratos, 2015]. The works closest toours [Bao and Savarese, 2011; Salas-Moreno et al., 2013;Galvez-Lopez et al., 2016; Murthy et al., 2017] overlook thedata association problem (matching observations to correctlandmarks in the map) which is critical for reconstructinga global object map with accurate loop closures (recogniz-ing already visited locations). Data association is challenging

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18)

5204

Figure 2: Geometric (ORB) fea-tures (green points) and seman-tic (car) features (red points).

Figure 3: Structured object model with prior shapeconstraints (purple lines) and online camera con-straints (red lines).

Figure 4: Factor graph (right) of two sensor states xt,xt+1 constrained by IMU measurement it+1 and se-mantic observations St, St+1 (green) of landmarks li.

when many areas look alike (perceptual aliasing) and a singlewrong choice can drastically affect the estimation accuracy.We make the following contributions.• We generalize the geometric environment model of

SLAM by introducing structured object models of mid-level semantic landmarks (Fig. 1).• Instead of making hard, potentially wrong associa-

tions among observations and landmarks, we show thatSLAM inference can be done efficiently using a proba-bility distribution over all possible associations, leadingto significant advantages in ambiguous environments.

2 Metric-Semantic SLAM2.1 RepresentationConsider a sensor system consisting of an IMU and monoc-ular camera that are time synchronized. Choose a subset ofthe camera images as keyframes (e.g., by selecting every nthframe) and let xt be the system state (6-D pose, velocity, andIMU biases) corresponding to the tth keyframe. Let X :={xt}Tt=0 be the sensor state trajectory and let the environmentbe represented as a collection of landmarks L := {lm}Mm=1with positions lm ∈ R3. We distinguish between geomet-ric landmarks λj estimated from geometric features Yt (e.g.,Harris corners, SIFT, ORB, etc.) and semantic landmarkslm estimated from semantic features St (parts of detected ob-jects) extracted from each keyframe image as shown in Fig. 2.Geometric Structure. The IMU provides a set It+1 oflinear acceleration and rotational velocity measurements be-tween keyframes t and t + 1. These measurements are inte-grated using the approach of [Forster et al., 2015] to obtaina single constraint that relates xt and xt+1 via an IMU mea-surement model f with noise covariance U :

it+1 = f(xt+1, xt) + ut, ut ∼ N (0, U). (1)The model in (1) relates two system states and specifies aGaussian density function p(it+1|xt, xt+1) for the IMU mea-surement it+1. Similarly, each geometric feature ytk ∈ Yt fork = 1, . . . ,Kt extracted at time t relates the position λj ∈ R3

of a corresponding geometric landmark to the system state xtvia a camera projection model g [Ma et al., 2012, Ch.3]:

ytk = g(xt, λj) + vt, vt ∼ N (0, V ) (2)

To recover the landmark positions {λj}Jj=1, it is necessaryto know the data association πt(k) = j of each feature ytk

to the correct landmark λj . Estimating πt : {1, . . . ,Kt} →{1, . . . , J} can done by tracking the features ytk in consec-utive keyframes. For example, we extract ORB features andtrack them by minimizing the ORB descriptor distance be-tween consecutive frames [Rublee et al., 2011]. As discussedin Sec. 2.2, this visual odometry allows us to infer the sensortrajectory X locally but is not sufficient for loop closure andestimation of the global landmark positions {λj}.Semantic Structure. Instead of using geometric landmarksto model the environment, our first innovation is to extractsemantic features (object parts) and represent the environ-ment as a set of objects, with inferred positions, orientations,and classes (Fig. 1). We assume a known list of object classes(with prior training data) and detect bounding boxes in eachkeyframe via a real-time detector such as DPM [Dubout andFleuret, 2013], RCNN [Ren et al., 2015], or YOLO [Redmonand Farhadi, 2016]. Within each bounding box, we extractsemantic features specific to the object class via the stackedhourglass convolutional network of [Pavlakos et al., 2017].Let St be the set of all |St| = Nt semantic features extractedfrom the tth keyframe. Similar to the geometric case, each se-mantic feature stn ∈ St is related to the position lm ∈ R3 ofa semantic landmark via the camera perspective projection:

stn = g(xt, lm) + wt, wt ∼ N (0,W ) (3)

and an unknown function dt : {1, . . . , Nt} → {1, . . . ,M}stipulating the data associations among St and L. Unlikein the geometric case, we cannot simply track the semanticfeatures because we are interested in recovering their globalpositions L, which requires loop closure (recognizing previ-ously observed landmarks instead of instantiating new ones).We focus on mapping semantic landmarks because they arefewer (leading to efficient performance) and yet more robustto viewpoint changes than geometric ones. This allows usto perform accurate visual-inertial odometry relying on bothgeometric and semantic features locally and efficient globalloop closure over the map of semantic landmarks (Fig. 1).

To obtain the poses of the objects associated with the se-mantic landmarks, we use a representation similar to the de-formable parts model [Felzenszwalb et al., 2010]. We in-troduce structure constraints among an object’s landmarks,given by a root location and displacements of the object partswith respect to it (Fig. 3). For example, the front-left wheel ofa car should not deviate significantly from an expected posi-


5205

tion relative to its front-left door. The structure constraint cijbetween the positions li, lj of two semantic parts belonging tothe same object class are captured by a function h, while thepart displacement costs are captured by a Gaussian model:

cij = h(li, lj) + η, η ∼ N (0, E). (4)Given its semantic part positions, an object’s orientation canbe recovered via the Kabsch algorithm [Kabsch, 1978].

Problem. Determine the sensor trajectory X and seman-tic landmark positions L that maximize the likelihood ofthe inertial, geometric, and semantic measurements Z :={(It, Yt, St)}Tt=0under the nonlinear Gaussian models (1)-(4)

X , L = arg maxX ,L

log p(Z | X ,L) (5)

2.2 InferenceA factor graph is a probabilistic graphical model [Kollerand Friedman, 2009] that allows representing an optimizationproblem such as (5) as a graph of random variables (nodes)with constraints among them (factors). In our case, the nodesare Gaussian variables corresponding to X and L. The fac-tors are constraints imposed by the inertial It, geometric Yt,and semantic St measurements via the models (1)-(4). Fig. 4shows an example graph of two sensor states related by in-ertial and semantic measurements. Inference over the graphcan be formulated as a maximization (5) of the likelihood ofall measurements with respect to the latent variables X , L:

X , L = arg maxX ,L

logT∏t=0

p(It, Yt, St | X ,L)

= arg minX ,L

T∑t=1

‖it − f(xt, xt−1)‖2U

+T∑t=0

Kt∑k=1

J∑j=1

1{πt(k)=j}‖ytk − g(xt, λj)‖2V

+T∑t=0

Nt∑n=1

M∑m=1

1{dt(n)=m}‖stn − g(xt, lm)‖2W

(6)

where we have skipped prior and structure constraints (4) tosave and have used ‖e‖2Σ := eTΣ−1e. The second equalityutilizes that the models (1)-(4) are Gaussian; e.g., p(it+1 |xt+1, xt) ∝ exp

{− 1

2‖it+1 − f(xt+1, xt)‖2U}

. We also em-phasized the dependence on the unknown data associationsπt, dt via the indicator functions. Assuming for now that πt,dt are known, (6) is a nonlinear least-squares problem, whichcan be solved efficiently via incremental linearization (e.g.,Gauss-Newton algorithm). Given initial estimates X 0, L0,e.g., obtained from IMU integration and feature triangulation,the functions f and g are linearized to obtain the JacobiansF x, F y and Gx, Gy with respect to each input, leading to:

minδX ,δL

T∑t=1

‖it − f(x0t , x

0t−1)− F xt,t−1δxt − F

yt,t−1δxt−1‖2U+

T∑t=0

Kt∑k=1

J∑j=1

1{πt(k)=j}‖ytk − g(x0t , λ

0j )−Gxt,jδxt −G

yt,jδλj‖

2V

T∑t=0

Nt∑n=1

M∑m=1

1{dt(n)=m}‖stn − g(x0t , l

0m)−Gxt,mδxt −G

yt,mδlm‖2W

To avoid estimating the geometric landmark positions {λj},a key idea used in structureless VIO approaches [Mourikisand Roumeliotis, 2006] is to multiply the second term aboveby a unitary matrix whose columns form the the basis ofthe left nullspace of Gyt,j , making the term Gyt,jδλj disap-pear. Collecting the Jacobians into one large but sparse ma-trix A and the error vectors it−f(x0

t , x0t−1), ytk−g(x0

t , λ0j ),

and stn − g(x0t , l

0m) into one vector b, leads to a standard

least-squares problem minθ ‖Aθ− b‖2 where θ := (δX , δL).Even though we are estimating the whole history X of sen-sor states, since the models (1)-(4) depend only on pairs ofsensor-landmark states, A remains sparse and the problemcan be solved efficiently via QR factorization. The optimiza-tion can be performed incrementally as new measurementsarrive by updating (rather than recomputing) the QR factor-ization [Kaess et al., 2008; 2012].

2.3 Data AssociationWe return to the data association problem. Since we do notaim to recover {λj}, it is sufficient to obtain the geometricassociations πt via feature tracking as mentioned in Sec. 2.1.It is, however, necessary to estimate the semantic associationsD := {dt}Tt=0, if we are to perform loop closure and recon-struct the semantic landmark positions L. We revisit the maxlikelihood estimation in (5), emphasizing that the semanticassociations D are unknown and should also be inferred:

X , L, D = arg maxX ,L,D

log p(Z | X ,L,D) (7)

Existing work decomposes this joint SLAM and data associ-ation problem into two subproblems. First, given prior esti-mates X 0,L0, the max likelihood estimate D of the associ-ations is computed, e.g., via JCBB [Neira and Tardos, 2001]or the Hungarian algorithm [Munkres, 1957]. Then, givenD, the most likely landmark and sensor states are estimated.This is a special case of coordinate descent:

Di+1 = arg maxD

p(D | X i,Li,Z)

X i+1,Li+1 = arg maxX ,L

log p(Z | X ,L,Di+1)(8)

which allows revisiting association decisions once state es-timates improve but does little to resolve the problem withambiguous measurements since a hard decision on data asso-ciations is required and may have a highly detrimental effecton the estimation performance. Rather than simply selectingD as the mode of p(D | X ,L,Z), our second innovationis to consider the entire distribution of D when estimatingX and L. Since D is a latent variable whose realization weare not interested in inferring but is needed to estimate Xand L, probability theory suggests that we should marginal-izeD. Our main result is that expectation maximization (EM)allows us to maximize the expected (w.r.t. D) measurementlikelihood, utilizing the whole distribution of D:


∑D∈D

p(D | X i,Li,Z) log p(Z | X ,L,D) (9)

To compare this with coordinate descent (8), we drop the in-ertial and geometric terms in (6) since their data associations


5206

Figure 5: Estimated sensor trajectory andposes of swivel and four-legged chairs infirst office experiment.

0 20 40 60 80 1001201401601802002202402602803000

5

10

15

20

25

t [s]

Absolutetranslationerror[m

] OursVISO2OrbSLAM MonoOrbSLAM Stereo

Figure 6: Norm of the position error be-tween the estimated and ground truth ve-hicle trajectories on KITTI sequence 05.

KITTI Sequence 05Method Trans. err [%] Rot. err [deg/m]

Ours 1.31 0.0038VISO2 4.08 0.0050

ORBSLAM2 Mono 5.39 0.0019ORBSLAM2 Stereo 0.63 0.0017

KITTI Sequence 06Method Trans. err [%] Rot. err [deg/m]

Ours 0.77 0.0037VISO2 1.81 0.0036

ORBSLAM2 Mono 6.71 0.0015ORBSLAM2 Stereo 0.29 0.0013

Figure 7: Mean translational and rotational errorover subpath lengths (100, 200, . . . , 800) meterson the KITTI odometry dataset.

are known or estimated via feature tracking and re-write (9):

maxX ,L

T∑t=0

Nt∑n=1

M+Nt∑m=1

∑dt∈Dt(n,m)

p(dt | X i,Li, St)︸︷︷︸=:wt,i

n,m

log p(stn | xt, lm) (10)

Above Dt(n,m) := {dt | dt(n) = m} is the set of possibledata associations at time t that assign semantic feature stn tolandmark lm and wt,in,m quantifies the probability that stn wasproduced by landmark lm via (3). Note that (6) and (8) havea similar form to (10), except that for each n there is exactlyone m such that wt,in,m = 1 and wt,in,m′ = 0 for all m′ 6= m.The EM formulation has the advantage that no hard decisionsare required since it “averages” over all possible data associa-tions. The following proposition shows that, besides general-izing coordinate descent, EM can be performed efficiently viaa connection between the data association distribution wt,in,mand the permanent1 of a suitable matrix.

Proposition. Optimization (10) can be performed by iterating:• E step: given X i,Li compute the associations distribution:

ωt,in,m ∝ Qtnmper Qt−nm

whereQt is a matrix of data association probabilities2 withelements Qtnm and Qt−nm is the matrix with the nth rowand mth column removed.

• M step: given the data association distribution ωt,in,m, up-date the sensor and landmark state estimates:


T∑t=0

Nt∑n=1

M+Nt∑m=1

wt,in,m log p(stn | xt, lm)

This result allows us to take advantage of matrix perma-nent approximation algorithms [Jerrum et al., 2004] to effi-ciently summarize the combinatorial data association space

1The permanent of an n×mmatrixA = [A(i, j)] with n ≤ m isdefined as per(A) :=

∑d

∏ni=1A(i, d(i)), where the sum is over

all one-to-one functions d : {1, . . . , n} → {1, . . . ,m}.2Qt := [Gt Ht], whereGt ∈ RNt×M is a matrix of scaled like-

lihoodsGtnm = ptmλ(1−ptn)

p(stn | xt, lm), ptm is the probability thatlm is detected from xt according to a Poisson process with meanλ, and Ht = diag ([pnew(st1) · · · pnew(stNt)]) ∈ RNt×Nt isa matrix of probabilities that new landmarks should be created.See [Atanasov et al., 2015] for details.

in polynomial time. Instead of max likelihood data associa-tion as in (8), we estimate a data association distribution viathe weights wt,inm (“E” step) and then maximize the expectedmeasurement log likelihood over it (“M” step).

3 EvaluationOur algorithm uses GTSAM and its iSAM2 implementa-tion [Kaess et al., 2012] to perform the M step. Our front-end selects every 15th camera frame as a keyframe and tracksgeometric ORB features. Outlier tracks that do not fit the es-sential matrix constraint between consecutive views are elim-inated via RANSAC. The essential matrix is estimated usingrelative orientation from gyro measurements and two-pointcorrespondences. Objects are detected via the deformableparts model [Dubout and Fleuret, 2013] and semantic fea-tures are extracted via a stacked hourglass convolutional net-work [Pavlakos et al., 2017]. Experiments were performedindoors, building a map of doors, swivel chairs and four-legged chairs, and outdoors, building a map of cars on theKITTI odometry dataset [Geiger et al., 2012]. The indoorexperiments included loops around a room equipped withmotion tracking, a medium trajectory (175 m) spanning onefloor, and a long trajectory (625 m) spanning two floors. Theestimated sensor trajectory and chair poses in the medium se-quence are shown in Fig. 5. The performance is comparedqualitatively to ROVIO [Bloesch et al., 2015] and ORB-SLAM2 [Mur-Artal and Tardos, 2017]. Since ORB-SLAM2uses only geometric features and no inertial information, itloses tracking for long durations in feature-deprived areasbut is able to recover when entering previously mapped re-gions. In the long indoor sequence due to the repetitive na-ture of the hallways, the bag-of-words loop-closure approachof ORB-SLAM2 also makes incorrect associations leading tounsuccessful tracking. The outdoor experiments used visualodometry from VISO2 [Geiger et al., 2011] instead of iner-tial odometry. The translation and rotation errors of our algo-rithm, VISO2, and ORB-SLAM2 with respect to the groundtruth vehicle trajectory are compared in Fig. 6 and Fig. 7.

References[Atanasov et al., 2015] N. Atanasov, M. Zhu, K. Daniilidis, and

G. Pappas. Localization from semantic observations via the ma-trix permanent. IJRR, 35:73–99, 2015.


5207

[Bao and Savarese, 2011] S. Bao and S. Savarese. Semantic Struc-ture from Motion. In IEEE CVPR, pages 2025–2032, 2011.

[Bloesch et al., 2015] M. Bloesch, S. Omari, M. Hutter, andR. Siegwart. Robust visual inertial odometry using a direct EKF-based approach. In IEEE/RSJ IROS, pages 298–304, 2015.

[Bowman et al., 2017] S. Bowman, N. Atanasov, K. Daniilidis, andG. Pappas. Probabilistic Data Association for Semantic SLAM.In IEEE ICRA, 2017. (Best Conference Paper Award).

[Cadena et al., 2016] C. Cadena, L. Carlone, H. Carrillo, Y. Latif,D. Scaramuzza, J. Neira, I. Reid, and J. Leonard. Past, Present,and Future of Simultaneous Localization and Mapping: Towardthe Robust-Perception Age. IEEE T-RO, 32(6):1309–1332, 2016.

[Dubout and Fleuret, 2013] C. Dubout and F. Fleuret. Deformablepart models with individual part scaling. In BMVC, 2013.

[Engel et al., 2014] J. Engel, T. Schops, and D. Cremers. LSD-SLAM: Large-scale direct monocular SLAM. In ECCV, 2014.

[Felzenszwalb et al., 2010] P. Felzenszwalb, R. Girshick,D. McAllester, and D. Ramanan. Object Detection withDiscriminatively Trained Part-Based Models. IEEE Trans. onPAMI, 32(9):1627–1645, 2010.

[Forster et al., 2015] C. Forster, L. Carlone, F. Dellaert, andD. Scaramuzza. IMU Preintegration on Manifold for EfficientVisual-Inertial Maximum-a-Posteriori Estimation. In Robotics:Science and Systems, 2015.

[Forster et al., 2016] C. Forster, Z. Zhang, M. Gassner, M. Werl-berger, and D. Scaramuzza. SVO: Semi-Direct Visual Odometryfor Monocular and Multi-Camera Systems. IEEE T-RO, 2016.

[Galvez-Lopez et al., 2016] D. Galvez-Lopez, M. Salas, J. Tardos,and J. Montiel. Real-time monocular object SLAM. Roboticsand Autonomous Systems, 75:435–449, 2016.

[Geiger et al., 2011] A. Geiger, J. Ziegler, and C. Stiller. Stere-oScan: Dense 3d Reconstruction in Real-time. In Intelligent Ve-hicles Symposium (IV), pages 963–968, 2011.

[Geiger et al., 2012] A. Geiger, P. Lenz, and R. Urtasun. Are weready for Autonomous Driving? The KITTI Vision BenchmarkSuite. In IEEE CVPR, 2012.

[Girshick et al., 2014] R. Girshick, J. Donahue, T. Darrell, andJ. Malik. Rich feature hierarchies for accurate object detectionand semantic segmentation. In IEEE CVPR, 2014.

[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep resid-ual learning for image recognition. In IEEE CVPR, 2016.

[Hornung et al., 2013] A. Hornung, K. Wurm, M. Bennewitz,C. Stachniss, and W. Burgard. OctoMap: An efficient proba-bilistic 3D mapping framework based on octrees. AutonomousRobots, 34(3):189–206, 2013.

[Jerrum et al., 2004] M. Jerrum, A. Sinclair, and E. Vigoda. APolynomial-time Approximation Algorithm for the Permanentof a Matrix with Nonnegative Entries. Journal of the ACM,51(4):671–697, 2004.

[Kabsch, 1978] W. Kabsch. A discussion of the solution for thebest rotation to relate two sets of vectors. Acta CrystallographicaSection A, 34(5):827–828, 1978.

[Kaess et al., 2008] M. Kaess, A. Ranganathan, and F. Dellaert.iSAM: Incremental Smoothing and Mapping. IEEE T-RO,24(6):1365–1378, 2008.

[Kaess et al., 2012] M. Kaess, H. Johannsson, R. Roberts, V. Ila,J. Leonard, and F. Dellaert. iSAM2: Incremental Smoothing andMapping Using the Bayes Tree. IJRR, 31(2):216–235, 2012.

[Koller and Friedman, 2009] D. Koller and N. Friedman. Proba-bilistic Graphical Models. MIT press, 2009.

[Kostavelis and Gasteratos, 2015] Io. Kostavelis and A. Gasteratos.Semantic mapping for mobile robotics tasks: A survey. Roboticsand Autonomous Systems, 66:86–103, 2015.

[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and GeoffreyH. ImageNet Classification with Deep Convolutional Neural Net-works. In NIPS, pages 1097–1105. 2012.

[Kummerle et al., 2011] R. Kummerle, G. Grisetti, H. Strasdat,K. Konolige, and W. Burgard. g2o: A General Framework forGraph Optimization. In IEEE ICRA, pages 3607–3613, 2011.

[Ma et al., 2012] Y. Ma, S. Soatto, J. Kosecka, and S. Sastry. AnInvitation to 3-D Vision: from Images to Geometric Models, vol-ume 26. Springer Science & Business Media, 2012.

[Mourikis and Roumeliotis, 2006] A. Mourikis and S. Roumeliotis.A Multi-State Constraint Kalman Filter for Vision-aided InertialNavigation. Technical report, University of Minnesota, 2006.

[Munkres, 1957] J. Munkres. Algorithms for the Assignment andTransportation Problems. Journal of SIAM, 5(1):32–38, 1957.

[Mur-Artal and Tardos, 2017] R. Mur-Artal and J. Tardos. ORB-SLAM2: an open-source SLAM system for monocular, stereoand RGB-D cameras. IEEE T-RO, 33(5):1255–1262, 2017.

[Murthy et al., 2017] J. Murthy, S. Sharma, and K. Krishna. Shapepriors for real-time monocular object localization in dynamic en-vironments. In IEEE/RSJ IROS, pages 1768–1774, 2017.

[Neira and Tardos, 2001] J. Neira and J. Tardos. Data Associationin Stochastic Mapping Using the Joint Compatibility Test. IEEET-RO, 17(6):890–897, 2001.

[Pavlakos et al., 2017] G. Pavlakos, X. Zhou, A. Chan, K. G. Der-panis, and K. Daniilidis. 6-DoF object pose from semantic key-points. In IEEE ICRA, pages 2011–2018, 2017.

[Qin et al., 2017] T. Qin, P. Li, and S. Shen. VINS-Mono: A robustand versatile monocular visual-inertial state estimator. In arXivpreprint:1708.03852, 2017.

[Redmon and Farhadi, 2016] J. Redmon and A. Farhadi. Yolo9000:Better, faster, stronger. arXiv preprint:1612.08242, 2016.

[Ren et al., 2015] S. Ren, K. He, R. Girshick, and J. Sun. FasterR-CNN: Towards Real-Time Object Detection with Region Pro-posal Networks. In NIPS, pages 91–99. 2015.

[Rublee et al., 2011] E. Rublee, V. Rabaud, K. Konolige, andG. Bradski. ORB: An efficient alternative to SIFT or SURF. InICCV, pages 2564–2571, 2011.

[Salas-Moreno et al., 2013] R. Salas-Moreno, R. Newcombe,H. Strasdat, P. Kelly, and A. Davison. SLAM++: SimultaneousLocalisation and Mapping at the Level of Objects. In IEEECVPR, 2013.

[Simonyan and Zisserman, 2014] K. Simonyan and A. Zisserman.Very Deep Convolutional Networks for Large-Scale ImageRecognition. arXiv:1409.1556, 2014.

[Strasdat et al., 2010] H. Strasdat, J. Montiel, and A. Davison.Real-time monocular SLAM: Why filter? In IEEE ICRA, 2010.

[Thrun et al., 2005] S. Thrun, W. Burgard, and D. Fox. Probabilis-tic Robotics. MIT Press Cambridge, 2005.

[Whelan et al., 2016] T. Whelan, R. Salas-Moreno, B., A. Davison,and S. Leutenegger. ElasticFusion: Real-Time Dense SLAM andLight Source Estimation. IJRR, 35(14):1697–1716, 2016.


5208

a unifying view of geometry, semantics, and data...

Documents