augmented reality dimensional space - key issues in … · identifying the location of tetragons in...

Szybkobie�ne Pojazdy G�sienicowe (42) nr 4, 2016

Michał BUGAŁA – O�rodek Badawczo-Rozwojowy Urz�dze� Mechanicznych "OBRUM" sp z o.o., Gliwice

Michał BUGAŁA

IDENTIFYING THE LOCATION OF TETRAGONS IN THE THREE-DIMENSIONAL SPACE - KEY ISSUES IN THE FIELD OF

AUGMENTED REALITY

Abstract. The paper presents one of the methods of calculating the pose1 of tetragons in the three-dimensional space on the basis of coplanar two-dimensional points. In conclusion, reference is made to the effects obtained and to the possibilities of practical application of the algorithm.

Keywords: computer graphics, augmented reality, virtual reality, PnP problems, matrix decomposition

1. INTRODUCTION

The results of calculations of the position and orientation of a tetragon lying in a plane constitute a very valuable and sometimes indispensable information in a number of augmented reality issues. The presented method of calculation may be utilised in augmented reality applications as a basis for a system based on markers2 and in techniques of SLAM algorithms calibration [1] [2]. There are many methods of calculating the pose of tetragons in the three-dimensional space. However, the calculation method chosen here has proved to be sufficiently reliable and fast to enable its practical application. The wotk described in the paper was carried out in the course of implementing a project titled Procedural system for the repair and maintenance of the engine of the Jelcz 442.32 vehicle at the OBRUM's Simulator Department.

The calculation of the position and orientation of a tetragon based on the relationship between the vertices of the tetragon and its corresponding points in perspective is not an easy task. The PnP3 (Perspective-n-Points) problem refers in this case to the relationship between four coplanar points (P4P).

The problem to be resolved in this paper is to determine the matrix of rotation of the tetragon and of the translation vector based on four object points (P1, P2, P3, P4) and four reference points (P1', P2', P3', P4') as shown in Fig. 1.

1Pose – position and orientation.

2Markers in augmented reality systems – characteristic patterns applied onto flat surfaces, e.g. a square printed on a sheet of paper.

3PnP problem – is related to the calculation of the position and orientation of objects on the basis of the relationship between a set of 3D or 2D points and corresponding 2D projections of these points in an image [3].

110 111

Fig. 1. An example of a tetragon and its perspective view

2. CALCULATION METHOD

The most common mathematical tool used in the concept of a vector. Vectors can be represented in the form of matrices, which facilitates numerous algebraic operations.

In order to calculate the position and orientation of the tetragon in threespace, an algorithm described in the paper Points by Denis Oberkampf, Daniel F. Dementhon and Larry S. Davisa [4] was applied.

2.1. Matrix of model

Usually, in Cartesian coordinates, bound vectors are condefined by the coordinates of its terminal point, as its initial point is always the origin of the coordinate system O=(0,0,0). Thus the bound vector represented by a sample pointis a vector of unit length pointing fro

Four points of the tetragon can be described using three twobound at the origin of the coordinate system (Fig. 2).

Fig. 2. Tetragon described with three vectors


2. CALCULATION METHOD

The most common mathematical tool used in computer graphics is linear algebra and the concept of a vector. Vectors can be represented in the form of matrices, which facilitates numerous algebraic operations.

In order to calculate the position and orientation of the tetragon in threeace, an algorithm described in the paper Iterative Pose Estimation Using Coplanar Feature

by Denis Oberkampf, Daniel F. Dementhon and Larry S. Davisa [4] was applied.

Usually, in Cartesian coordinates, bound vectors are considered. A bound vector is defined by the coordinates of its terminal point, as its initial point is always the origin of the

. Thus the bound vector represented by a sample pointis a vector of unit length pointing from the origin in the positive direction of the x axis.

Four points of the tetragon can be described using three two-dimensional vectors bound at the origin of the coordinate system (Fig. 2).

Fig. 2. Tetragon described with three vectors

Michał BUGAŁA


computer graphics is linear algebra and the concept of a vector. Vectors can be represented in the form of matrices, which facilitates

In order to calculate the position and orientation of the tetragon in three-dimensional Iterative Pose Estimation Using Coplanar Feature

by Denis Oberkampf, Daniel F. Dementhon and Larry S. Davisa [4] was applied.

sidered. A bound vector is defined by the coordinates of its terminal point, as its initial point is always the origin of the

. Thus the bound vector represented by a sample point P=(1,0,0) m the origin in the positive direction of the x axis.

dimensional vectors

110 111Identifying the location of tetragons in the three-dimensional space...

Vectors can be expressed as a column of a matrix:

� � ��

If we take as an example the following points: P1 (-0.5, 0.5), P2 (0.5, 0.5), P3 ( 0.5, -0.5), P4 (-0.5, -0.5), we obtain the matrix:

� � ��

�

2.2. Singular value decomposition of a matrix

Singular value decomposition (SVD) of a matrix is a certain factorization of a real matrix into a product of three specific matrices expressed by the following formula:

��,

where U and V are orthogonal matrices, and � is a diagonal matrix [5].

An example of a singular value decomposition of matrix A having the form:

� � ��

�

is equal to:

��

��

��

��

��

� ��

��

��

��

,

� � ��

�!�

��

��

��

��

.

SVD is useful in solving many problems of statistical analysis, such as calculating a series, norms, condition number, pseudoinverse of a matrix [6].

112 113Michał BUGAŁA

There are many computation solutions for singular value decomposition of matrices, for example the cvSVD function in the OpenCV library. One can also use the decomposition algorithm presented in one of the series of books "Numerical Recipes"4 by William H. Press, Brian P. Flannery and others.

2.3. Generalised inverse matrix and SVD

The generalised inverse of a matrix, or otherwise its pseudoinverse, is the basic tool for calculating linear functions Ax = b, and provides an elegant geometric interpretation. The pseudoinverse is calculated using SVD:

"#��-��,

where �-1 is the inverse of matrix �, wherein all non-zero values are called singular values, while $ is the transpose of matrix [7].

For the following example of a matrix:

��

�

its pseudoinverse is:

"# � % �� &.

2.4. Coplanar Posit positioning algorithm

The flowchart of the Coplanar Posit algorithm, described in the article "Iterative Pose Estimation Using Coplanar Feature Points" and presented in Fig. 3, shows the subsequent steps that must be performed to determine the pose of the tetragon [4].

4The title of the book depends on the programming language: "Numerical Recipes in Pascal", "Numerical

Recipes in Fortran", "Numerical Recipes in C", "Numerical Recipes in Basic".

112 113Identifying the location of tetragons in the three

Fig. 3. Flowchart of the Coplanar Posit positioning algorithm

For the example object points P1 (their corresponding reference points P1' (rotation matrix �'( and the translation vector

'

)

Identifying the location of tetragons in the three-dimensional space...


the example object points P1 (-0.5, 0.5), P2 (0.5, 0.5), P3 ( 0.5, -0.5), P4their corresponding reference points P1' (-30, 20), P2' (20, 30), P3' (30, -10), P4' (

and the translation vector �)* are as follows:

�!+,-.��/�! ,+��/�!�.0��'(��!,011��!+�.,��/�!,+,��!��!,00+��!,,�-��!+ .,��

�)*��!� --!��!�0+ !��,!0,1�2


0.5), P4 (-0.5 -0.5) and 10), P4' (-10, -30), the

114 115

The calculation of these values, taking Fig, 4 as an aid, staequations:

where:

i, j, k - unit vectors of the coordinate system of camera,I, J

�34 - Correction factors between perspective and orthographic projection

to the following form:

Fig. 4. Perspective projection (m

For �3�� (1, 1, 1) in the first iteration and for the relationships where:

The calculation of these values, taking Fig, 4 as an aid, started with the rearrangement of

�'�'4�5�6�74��834�/�7�!�'�'4�5�9�:4��834�/�:�!

�'� - reference point, �'4 - object points,

unit vectors of the coordinate system of camera,I, J - vectors proportional to i and j,

ction factors between perspective and orthographic projection

;$4)4<4=;6>6?6<=@��74��834�/�7�!�;$4)4<4=;9>9?9<=@��:4��834�/�:�2

Fig. 4. Perspective projection (mi) and orthographic projection (pi) for object points Mand reference points Mo

(1, 1, 1) in the first iteration and for the relationships �6��7AB8!�9��:AB

�7A - vector with ith coordinate, �:A - vector with ith coordinate,

Michał BUGAŁA

rted with the rearrangement of

ction factors between perspective and orthographic projection

for object points Mi

�:AB8!

114 115Identifying the location of tetragons in the three-dimensional space...

we arrive at:

�� C � � �-�� C � � �-��-� C � � �-�� ,

�� C �/ �� C �/ �� C �/ ��

and:

DE � �F0!0001! ��0!0001! �, GE � �� ! ! F0!0001! �.

Another equation of the coplanar POSIT algorithm in the form:

H��E� � DE� � ,��E C GE,

in a polar coordinate system:

H� � ;��=,

where: ��GE� � DE�� 8 F�DE C GE�� I ,

�� !�"��,DE C GE �GE� � DE�I , if GE� C DE� J � and �� !�"��,DE C GE �GE� � DE�I #$, if GE� C DE� K ��

(for GE� C DE� � �� %�&"�DE C GEL��MNO �� P ,DE C GE P)

enables the calculation of R and �: GE� C DE� � �''�'''(,

��(��))�, ��'*�'.

A tetragon in an orthographic projection has two possible orientations. By solving the equations:

��E#�� +,QR, ��E#��,�"QR

and equations for the second option:

��E � S� +,QR, ��E � S�,�"QR,

where: -��[3],

we arrive at:

��* � )�� )�� *(), �� ! !F0!0001!�F!0�.�


and:

��* � )�� )� � �� *(), ��* � )� � *� �(�.

The vector product of normalised vectors I, J:

T � DU�GU, TV � DVW�GVW

enables the construction of the rotation matrix:

XY � �D� D� D�G� G� G�T� T� T�

�,

XY �� D�V D�V D�VG�V G�V G�VT�V T�V T�V

and calculation of the translation vector:

Z[ � �\! ]! ^,

\ ��_ � �̀!��] �

�_ � �̀!��^ � a _I

where:

_ � bcb#bdb�

,

�̀ � D� C �� 8 D� C �, �̀ � G� C �� 8 G� C �,

a � 0F��efghi�jeklm�en�oe�ipq�[4].

For the first iteration we obtain:

XY � ��!+,�1 ��! ,.. �!,��!,0 � �!+,�1 �!,..,��!,.., ��!,�� !+ F,

�,

XYV � ��!+,�1 ��! ,.. ��!,��!,0 � �!+,�1, ��!,..,�!,.., �!,�� !+ F,

�,

Z[ � ��!� ,+! �!�0-.! �,!0,1�,

Z[V � ��!� ,+! �!�0-.! �,!0,1�.


In order to calculate � 3with a smaller error was selected (the error can be calculated based on the avebetween the reference points and the points of the calculated location).

According to the formula:

where the correction factor �3

��3��

the value of �3 was:

3��!+11�1F++1 ,+1�,!��!++ .� -�F1�+F1 !��!��00 .-�1 1+102

The number of subsequent iterations depends on the tendency of the value of the error for the next iteration to increase or decrease and on the set limit. The rotation matrix enables straightforward calculation of the rotation about the X, Y and Z axes. Fig. 5 shofrom a camera with superimposed coordinate system.

Fig. 5. Pose of the tetragon (2x2 sector of a checkerboard) with superimposed coordinate

3. CONCLUSIONS

The method of calculating the position and orientation of tetragons in threedimensional space, implemented by the OBRUM's Simulator Department, enables constructing a system for positioning virtual objects based on markers (Fig. 6), the location of homography of templates from a number of methods for detecting and determining the features of characteristic points (methods without express markers), such as: SIFT [8], BRISK [9], FAST [10], ORB, A-KAZE, etc. It can also be used as a calibration step in SLAM algorithms [2]. Confronting the method with more recent P4P developments, lik


3 for the next iteration, a rotation matrix and translation vector with a smaller error was selected (the error can be calculated based on the avebetween the reference points and the points of the calculated location).

According to the formula:

�3��rs�'�'4�5�t�;F=!

for the subsequent iteration can be written in the form:

��5�u�rs�8��5�u,rs�8��5�u rs�8��3��5�u�rs�8��/��5�u,rs�8��5�u rs�8��!��5�u�rs�8��/��5�u,rs�8��5�u rs�8��

��!+11�1F++1 ,+1�,!��!++ .� -�F1�+F1 !��!��00 .-�1 1+102

er of subsequent iterations depends on the tendency of the value of the error for the next iteration to increase or decrease and on the set limit. The rotation matrix enables straightforward calculation of the rotation about the X, Y and Z axes. Fig. 5 shofrom a camera with superimposed coordinate system.

Fig. 5. Pose of the tetragon (2x2 sector of a checkerboard) with superimposed coordinate system

The method of calculating the position and orientation of tetragons in threedimensional space, implemented by the OBRUM's Simulator Department, enables constructing a system for positioning virtual objects based on markers (Fig. 6), the location of homography of templates from a number of methods for detecting and determining the features of characteristic points (methods without express markers), such as: SIFT [8], BRISK

KAZE, etc. It can also be used as a calibration step in SLAM algorithms [2]. Confronting the method with more recent P4P developments, lik

for the next iteration, a rotation matrix and translation vector with a smaller error was selected (the error can be calculated based on the average distance

for the subsequent iteration can be written in the form:

rs�8��!�

��!+11�1F++1 ,+1�,!��!++ .� -�F1�+F1 !��!��00 .-�1 1+102

er of subsequent iterations depends on the tendency of the value of the error for the next iteration to increase or decrease and on the set limit. The rotation matrix enables straightforward calculation of the rotation about the X, Y and Z axes. Fig. 5 shows the image

Fig. 5. Pose of the tetragon (2x2 sector of a checkerboard) with superimposed coordinate

The method of calculating the position and orientation of tetragons in three-dimensional space, implemented by the OBRUM's Simulator Department, enables constructing a system for positioning virtual objects based on markers (Fig. 6), the location of homography of templates from a number of methods for detecting and determining the features of characteristic points (methods without express markers), such as: SIFT [8], BRISK

KAZE, etc. It can also be used as a calibration step in SLAM algorithms [2]. Confronting the method with more recent P4P developments, like Fast and

118 119

robust pose estimation using coplanar feature pointsinteresting prospect for further research.

Detection of markers is not the subject of this article, however, it is easy to imagine the possibilities of applying the algorithm described here upon detecting a potential marker. (e.g. location of the walls on the basis of symmetry of windows enabling the application of virtual advertisements).

Fig. 6. Positioning according to a marker (RGBY points)

Reliability of the method in case of temporary lack of reference points and eliminating vibration caused by inaccurate detection of a marker can be achieved by using a Kalman filter (Fig. 7).

Fig. 7. Marker-based position tracking using a Kalman filter (areference points does not cause loss of the position of a virtual object)

Fig. 8 illustrates the concept of using homography of one of the methods and determining features of characteristic points in order to calculaorientation of a coffee machine. Locating homography of a comparative template in the camera image enables calculating the position and orientation of the entire object when the actual position of the characteristic section of surface the object is known.

robust pose estimation using coplanar feature points by J. Xia, Xu X, Xiong J. [11], makes an interesting prospect for further research.

Detection of markers is not the subject of this article, however, it is easy to imagine of applying the algorithm described here upon detecting a potential marker.

(e.g. location of the walls on the basis of symmetry of windows enabling the application of


liability of the method in case of temporary lack of reference points and eliminating vibration caused by inaccurate detection of a marker can be achieved by using a Kalman filter

based position tracking using a Kalman filter (absence of one of the four reference points does not cause loss of the position of a virtual object)

Fig. 8 illustrates the concept of using homography of one of the methods and determining features of characteristic points in order to calculate the

coffee machine. Locating homography of a comparative template in the camera image enables calculating the position and orientation of the entire object when the actual position of the characteristic section of surface (template) in relation to the position of

Michał BUGAŁA

by J. Xia, Xu X, Xiong J. [11], makes an

Detection of markers is not the subject of this article, however, it is easy to imagine of applying the algorithm described here upon detecting a potential marker.

(e.g. location of the walls on the basis of symmetry of windows enabling the application of


liability of the method in case of temporary lack of reference points and eliminating vibration caused by inaccurate detection of a marker can be achieved by using a Kalman filter

bsence of one of the four reference points does not cause loss of the position of a virtual object)

Fig. 8 illustrates the concept of using homography of one of the methods of detecting te the position and

coffee machine. Locating homography of a comparative template in the camera image enables calculating the position and orientation of the entire object when the

(template) in relation to the position of


Fig. 8. Homography of characteristic section of surface (template) obtained using the

The algorithm described here proved to be reliable also in the case of determining camera gestures. A comparative template is in this case one of the previous camera frames (Fig. 9), while the algorithm is responsible for calculating the trajectory of the camera.

Image analysis and augmented reality is a fipotential for development and scientific research. Devices that use knowledge and technology of augmented reality have a great chance to become an attractive product on the markets of consumer goods (e.g. Google Gla


Fig. 8. Homography of characteristic section of surface (template) obtained using the FAST method

The algorithm described here proved to be reliable also in the case of determining era gestures. A comparative template is in this case one of the previous camera frames

9), while the algorithm is responsible for calculating the trajectory of the camera.

Fig. 9. Homography of camera

Image analysis and augmented reality is a field of computer science with a very high potential for development and scientific research. Devices that use knowledge and technology of augmented reality have a great chance to become an attractive product on the markets of

Google Glass) and industrial goods (perception of robots).

Fig. 8. Homography of characteristic section of surface (template) obtained using the

The algorithm described here proved to be reliable also in the case of determining era gestures. A comparative template is in this case one of the previous camera frames

9), while the algorithm is responsible for calculating the trajectory of the camera.

eld of computer science with a very high potential for development and scientific research. Devices that use knowledge and technology of augmented reality have a great chance to become an attractive product on the markets of

) and industrial goods (perception of robots).


4. REFERENCES

[1] Bazargani H., Laganière T.: Camera Calibration and Pose Estimation from Planes, https://www.researchgate.net/publication/284786954_Camera_Calibration_and_Pose_Estimation_from_Planes [Access: 29.08.2016].

[2] Castle R.O., Murray D. W.: Combining monoSLAM with Object Recognition for Scene Augmentation using a Wearable Camera http://www.robots.ox.ac.uk/~bob/ publications/castle_etal_ivc2010/castle_etal_ivc2010.pdf [Access: 29.08.2016].

[3] Lepetit V., Moreno-Noguer F., and Fua P.: EPnP: Efficient Perspective-n-Point Camera Pose Estimation, https://infoscience.epfl.ch/record/160138/files/top.pdf?version=1 [Access: 16.08.2016].

[4] Oberkampf D., DeMenton D., Davis L.S.: Iterative Pose Estimation Using Coplanar Feature Points,. Computer Vision and Image Understanding (63) no. 3, 1996 (pp. 495–511), ISSN: 1077-3142. Elsevier.

[5] Pa�czyk B., Łukasik E., Sikora J., Guziak T.: Metody numeryczne w przykładach, 2012, Politechnika Lubelska, ISBN: 978-83-63569-14-3.

[6] Borowicz G., Pikor M., Selwa S., Szczygieł W.: Systemy akwizycji i przesyłania informacji, http://www.prz.rzeszow.pl/kpe/materialy/astadler/DAQWWW/LabVIEW/ Dataprocessing/Algebraliniowa/algebra%20linoiwa.htm [Access: 10.05.2016].

[7] Clark D.: A note on the pseudoinverse, http://faculty.rmc.edu/davidclark/math/papers/ pseudoinverse.pdf [Access: 10.05.2016].

[8] Lowe D. G.: Object Recognition from Local Scale-Invariant Features: http://www.cs.ubc.ca/~lowe/papers/iccv99.pdf [Access: 11.06.2016].

[9] Leutenegger S., Margarita Ch., Siegwart R. Y.: BRISK: Binary Robust Invariant Scalable Keypoints, https://www.researchgate.net/publication/221110715_BRISK_ Binary_Robust_invariant_scalable_keypoints [Access: 12.06.2016].

[10] Rosten E., Drummond T.: Machine learning for high-speed corner detection: https://www.edwardrosten.com/work/rosten_2006_machine.pdf [Access: 12.06.2016].

[11] Xia J., Xu X., Xiong J.: Fast and robust pose estimation using coplanar feature points, https://www.researchgate.net/publication/261228857_Fast_and_robust_pose_ estimation_using_coplanar_feature_points [Access: 13.06.2016].

augmented reality dimensional space - key issues in … · identifying the location of tetragons in...

Documents