An Embedded ROS-based Machine Vision SystemImplemented on the Autonomous Surface Vehicle

Zhi LiIEEE Student Member

Autonomous Ocean Systems LaboratoryFaculty of Engineering and Applied Science

Memorial University of Newfoundlandzhil@mun.ca

Ralf BachmayerIEEE Member

Autonomous Ocean Systems LaboratoryFaculty of Engineering and Applied Science

Memorial University of Newfoundlandbachmayer@mun.ca

Abstract—This paper describes the integration of an em-bedded machine vision system based on the Robotic OperatingSystem (ROS) for an Autonomous Surface Vehicle (ASV). Thisvision system is integrated using a low-cost Universal Serial Bus(USB) camera connected to an ARMTM processor based single-board computer (SBC) running the Ubuntu distribution of theLinux operating system. The ROS is installed on the SBC inorder to benefit from the large amount of image processingalgorithms available for ROS. The goal of the implementationof an image processing system is to test and evaluate visuallyaided navigation and tracking algorithms for ASVs. In orderto evaluate the system, a core algorithm is realized using theARToolbox [2] and the specific marks from ARTags [3]. Althoughonly one camera is used, the distance and orientation includingroll, pitch and yaw between the camera and the marks can becalculated. The implementation of the ROS based vision systemconcludes with its integration into the controller area network(CAN) based distributed communication and control frameworkof the ASV.

Keywords—Robot Operating System (ROS); target tracking;ARToolbox

I. INTRODUCTION

The area of Autonomous Surface Vehicle (ASV) underwentsignificant developments in the past few years. A numberof prototypes with different configurations and applications[4]–[7] have been developed. Naval applications ranging fromharbour security to diesel electric submarine trailing over longdistances have been proposed and are currently in variousstates of realizations. The civilian applications are driven bysurvey tasks, such as bathymetric mapping or oil spill trackingor scientific tasks such as current profiling and water surfaceoceanographic and atmospheric measurements.

Depending on the particular mission the vehicles are usedin a semi-autonomous mode or are remotely controlled, butmost ASVs share the basic waypoint/trajectory following ca-pabilities. Most civilian and scientific operators are workingwithin line of sight of the ASV, however floating obstacles,such as tree trunks and other debris pose a veritable threat tothe vessels and are often not visible for the operator. On boardvision and vision processing could help to reduce the risk offouling or in some cases more severe collisions, without thenecessity of large investments into high bandwidth long-rangevideo transmission infrastructure. In addition the vision sensormight be explored for additional capabilities, such as sea state

estimates or shore line imaging. In the annual International“RoboBoat” Competition hosted by AUVSI Foundation [8],participating teams were required to equip their ASV withvision systems in order to detect different colour buoys whichdefined the path for the ASV to follow. In another vision ap-plication the Jet Propulsion Laboratory (JPL) [9] implementeda 360-degree visual detection system for their ASV to identifymoving ships.

In this article we report on our progress toward the im-plementation of the Robot Operating System (ROS) basedmachine vision system on the ASV. Section II provides anoverview of the ASV’s control and communication system.Section III introduces the machine vision system as it isimplemented on a single board computer, suitable for instal-lation onboard the ASV. The following section focuses onthe software implementation of this vision system. Section Vreports on preliminary test results of the vision system whileSection VI provides conclusions and future work.

II. THE AUTONOMOUS SURFACE VEHICLE

A. System Overview

The Autonomous Surface Vehicle (ASV) SeaCat was usedin this project (Fig. 1). The SeaCat was built in the Au-tonomous Ocean Systems Laboratory (AOSL) at MemorialUniversity. As can be seen from Fig. 1, this is a displacementstyle catamaran-type ASV. The two streamline bodies are theASV hulls, and two parallel connected battery systems arestored inside each hull. The ASV propulsion system consistsof two commercially available outboard electric motors, andthey are fixed at the rear part of each hull. The aluminum su-perstructure is used to install the antenna system and necessaryelectrical components.

The ASV onboard communication and control system wasbuilt based on the Controller Area Network (CAN) protocol[10]. Using the Micro-type cables and connectors [11] awaterproof and robust on-board wiring system was realized.The primary parameters of this ASV system are summarizedin Table I.

B. System Architecture

The vessel’s onboard communication and control systemis built using the Controller Area Network (CAN) protocol,

Fig. 1. The ASV SeaCat

TABLE I. THE SEACAT ASV MAIN SPECIFICATION

Parameters ValueLength 1.5 mWidth 1.0 mHeight 1.25 mDraft 0.37 mMass 146 kg

Max. Speed 1.0 m/s

a distributed industrial standard communication and controlsystem. Figure 2 depicts the top view of the layout of this dis-tributed system. As can be seen, the CAN network starts frominside one hull, and it passes through the superstructure andthen into the other hull. The initial configuration has integratedthe following four CAN nodes (labeled from 1 to 4) as shownin both Fig. 1 and Fig. 2. Two actuator CAN nodes (labeled 3and 4) are used for the motor control; the controller CAN node(label 1) exchanges the data with the dock-side computer usinga 900MHz RF-modem; the navigation CAN node (label 2) isresponsible for the collection of data from the Global PositionSystem (GPS) and the Attitude and Heading Reference System(AHRS). The actual position of these modules can also bevisualized in Fig. 1.

Fig. 2. CAN bus layout on the SeaCat

C. Controller Area Network (CAN)

The Controller Area Network (CAN) protocol was createdby Bosch Inc. in 1986. As a robust communication protocol,it was extensively used in building the communication andcontrol system on-board the automobiles.

The CAN protocol is based on a message-orientated com-munication mechanism [10]. Each transmitted CAN messageis started with a unique identifier (ID), and a message filteringsystem inside every CAN node decides if this unique IDmessage should be accepted. This unique identifier is aninteger, and its value denotes the priority that a messagecan access the CAN-bus (low value with high priority). In amessage transmission conflict, the high priority message willbe transmitted instantly, while the transmission of the lowpriority message will be delayed. In this case, the high prioritymessages will be guaranteed with the shortest transmissionlatency.

III. MACHINE VISION SYSTEM

The machine vision system integrates a low-cost UniversalSerial Bus (USB) webcam and an embedded single boardcomputer, i.e. PandaBoard ESTM . The webcam provides 640X 480 pixel color images, with a maximum update rate of 30frames per second. All images captured by the camera can bestreamed to the connected computer through the USB connec-tion. The PandaBoardTM uses dual-core ARMTM CortexTM-A9 processor with the maximum processing speed of 1.2 GHzeach, and it also integrates a 1 GB low power DDR2 RAM.The PandaBoardTM is a relatively powerful platform for anSBC, and for development purposes can be used like a desktopcomputer with monitor, Ethernet 10/100 Mbps as well as802.11 b/g/n wireless communications. Additionally, the USB,RS232 and I2C interfaces are provided to assist its connectionand communication with the peripheral devices [12].

The PandaBoardTM boots from the preloaded operatingsystem image in its SD card. In this application, the Ubuntu12.04 LTS desktop image was used. Based on this embeddedLinux operating system, the Robot Operating System (ROS)[13] software package was installed to provide the requiredhardware drivers and the required image processing functions.More details will be provided in the following sections. Fig-ure 3 shows the realized machine vision system architecture(highlighted in blue) and how this vision system is connectedto the ASV on-board communication and control system.

As can be seen from Fig. 3, the machine vision system (orthe vision CAN node) is connected with the CAN networkusing a CAN to USB converter. This converter is used totransmit and receive the CAN messages following the CANprotocol standard, and on the other side, it can exchange thedata with the PandaBoardTM using the USB connection.

Figure 4 shows the final realized machine vision systemsetup. As can be seen in 4(a), all electrical components (exceptthe camera) are fixed inside an enclosure that follows the IP 67standard [14]. The webcam is fixed on top of the enclosure lid,and the Micro-type connectors are used to enable convenientconnection with the main CAN network. Inside the enclosure(Fig. 4(b)) are the power system and the PandaBoardTM . TheUSB hub is used for connection of the camera, CANUSBconverter, keyboard and the mouse.

Fig. 3. Machine vision system connected with CAN network on SeaCat

(a) (b)

Fig. 4. Realized machine vision system a) Whole system configuration; b)Inside box system architecture.

IV. SOFTWARE ARCHITECTURE

A. Robot Operating System (ROS)

The Robot Operating System (ROS) was originally de-signed by Stanford Artificial Intelligence Laboratory in 2007,but from 2008 the main contribution to the ROS developmentcame from the not-for-profit organization Willow Garage [15]and more recently from the Open Source Robotics Foundation(OSRF). As a result ROS has been extensively used in mobilerobotics including land robots, quadcopters, surface crafts andautonomous underwater vehicles.

ROS is designed for robotics software development. Allfiles in ROS are organized based on its file system architecture.A “Package” is a basic unit for organizing files in ROS, andinside each package the ROS executable files, some dependentlibraries, configuration documents are included. Above that,“Stacks” can be regarded as a collection of “Packages” whichhave associated functionalities.

The ROS executable files are located inside each “Pack-age”, and they are generated by compiling the whole “Pack-age” with associated libraries and other dependencies. Mostexecutable files (or named as nodes in ROS) are created usingC++ or Python programming language. These files can then beran by the users to accomplish different tasks such as control,image processing, data transmission, parsing and so forth. Amain advantage of ROS is that each ROS node can be designedindividually and ran separately. The communication betweenROS nodes is used for exchange of data. However, this node-to-node communication has to be published to a specific ROStopic. The ROS topic defines the contents in a ROS message[13].

As an open source robotics software platform, ROS isextensively used by research institutions and laboratories. The

ROS core libraries provide the file management tools, compiletools and some useful services. By using these tools, otherROS packages can be developed. Since all packages arecreated following the same rules, the collaboration and sharingbetween different developers are convenient.

B. ROS-based Machine Vision System

So far ROS has been mainly implemented on IntelTM orAMDTM processor based computers, with significantly lessimplementations of ARMTM based SBCs. This caused someproblems in the implementation, since most packages werenot developed for the ARM processors. As a result duringthe first stage of this project, a lot of time was spent tosuccessfully build the ROS core libraries and install it on theARM based PandaBoard using the Ubuntu 12.04 operatingsystem. In addition the following packages were needed forthis project:

1) USB webcam driver, which can acquire the rawimages data from the webcam.

2) Image display package, which can be used to receivethe raw image data and display it on the screen.

3) Image processing package, which can process theraw images and calculate for the desired distance andorientation information.

4) CANUSB driver, which can build the communicationbetween ROS and the SeaCat CAN network.

5) Message conversion package, which on the one sideacquires the distance and orientation informationfrom the image processing node, and convert the datato send it on the CAN network.

Since it took significant time and effort to find the packagesa summary of these packages is provided:

For the USB webcam driver, the “gscam” package fromBrown University was used [16], and it was successfully builtand ran on the PandaBoardTM . The image display module“image view” was installed with the ROS core library, butthe author had no success in compiling this package at thefirst point because some dependent libraries could not beinstalled on PandaBoardTM . The same situation happened onthe image processing package the “ar recog” package [16].The author did not have any success until the post in [17] wasfound. By following the instructions from [17], the OpenCV[18] software package was successfully built and installedon the PandaBoardTM . After that, both the “image view”and the “ar recog” packages could be compiled. For theCANUSB device driver, the author implemented a packagecalled “can communication” [13]. 1

The last package needed was the message conversionpackage. Since there was no existed packages that couldbe used in this specific application, this package had to becreated. The package was written using the Python program-ming language. The package was named ASCvision, and anexecutable file named “ar listener.py”. The package subscribesto the image processing node messages and publishes to the

1This package was not maintained very well, some syntax errors occurredduring the initial compilation which needed to be fixed. Contact the mainauthor for more details.

CAN communication node, providing an interface between thevision system and the vehicle control network.

The“ar recog” ROS package includes the core image pro-cessing algorithms used in this machine vision system. Thisalgorithm is supported by the ARToolbox [2] and the OpenCV[18] software package, and it calculates the distance andorientation between the webcam and the ARTags [3]. Someexamples of the ARTags are shown in Fig. 5. The ARTagsconsist of a symbol which is in the center and it is surroundedby a black frame. The algorithm basically tries to identifythe region that can be fitted by four line segments (the blackframe), and then the parameters of the four line segmentsand their intersections are extracted. The region is normalized,and its included symbol is identified and compared with thepreloaded template. The details of performing the coordinatetransformation and calculating the distance and rotation anglecan be found in [19].

Fig. 5. Some examples of the ARTags

Figure 6 shows the realized software topology. Each ROSnode is developed separately, but they can communicate witheach other using the ROS messages. This feature can bevisualized clearly from Fig. 6. The “/gscam” node is re-sponsible for collecting the data from the USB webcam,and then the image and camera information is transmittedto the “/image listener” node. This “/image listener” node iscreated by the “ar recog” package, and the core algorithminside this node will calculate the distance and orientationbetween the camera and the marks. These information will bepublished under the “/tags” topic, and the author created node“ar listener” will repackage the received data and transmit therequired data under the “/can bus tx” topic to the CANUSBconverter for transmission. The “/image view” node is used forvisualized display the identified ARTags images. The “/rosout”node is used for logging all the transmitted messages and topicsfrom different nodes.

Fig. 6. Machine vision system software architecture

V. EVALUATION TESTS

Figure 7 shows the planned system setup. The machinevision system will be installed on the top front part of theASV superstructure. Then, a mark will be fixed at the towcarriage with the objective of the ASV tacking the mark. Byprocessing the image of the marks, the distance and orientationinformation between the ASV and the marks will be calculated.Based on the reference distance and orientation, the machinevision system will send the required commands to the ASV

propulsion systems. Therefore, the ASV will be driven forwardto follow the marks or backward to keep the required distance.By applying the differential thrust, the ASV heading can beadjusted to align with the marks.

Fig. 7. The SeaCat with realized machine vision system tested in tow tank

Prior to these experiments we have been evaluating theperformance of the described machine vision system modulein stand alone mode. In the evaluation test, the mark size waschosen as 150 mm X 150 mm, and the camera angle of viewwas set as 0.5. The distance test results are shown in Fig. 8.It can be concluded that the effective measurement distanceusing the realized machine vision system is within 4 meters,and above 4 meters the algorithm provides 0.5 m discrepancywith the accurate tape measurement. Within 4 meters, however,the camera measurement provides the acceptable accuracy.

Fig. 8. Upper plot shows the distance measured by the vision systemcompared with manual measurements, while the lower plot shows the errorbetween the two measurements.

The Y rotation angel (ASV yaw angle) and Z rotation angle(ASV roll angle) evaluation results are shown in Fig. 9 andFig. 10. The Y rotation angle was measured from -75 to 75degrees, and the angle measurement discrepancy is from 4 to11 degree, which seems to have a stable offset angle of 6degrees. This offset angle can be taken into account in thecalibration process, and after performing this transformation,the discrepancy is within ±5 degrees. The Z rotation angle wasevaluated from 0 to 75 degrees, and the absolute discrepancyis within 5 degree.

VI. CONCLUSION AND FUTURE WORK

In this project, an embedded ROS enabled machine vi-sion system that is capable of calculating the distance andorientation between an ARTag and a low-cost webcam has

Fig. 9. Y rotation measurement evaluation test results compared withprotractor measurement

Fig. 10. Z rotation measurement evaluation test results compared withprotractor measurement

been realized. Based on ROS, the realized system could useonly one USB webcam to successfully acquire the distance andorientation information from the ARTags and send the requiredCAN messages to the ASV system to control its motors. Thebuilt vision system is SeaCat compatible and potentially canbe used for the target tracking tasks (with ARtags). The builtmachine vision system was supported by the ARToolbox [2],ARTags [3] and OpenCV [18], and the measurement accuracywas evaluated. The CAN communication interface within theROS package still needs work prior to a full implementationon the ASV.

The implemented machine vision system was proven to befunctional and is able to provide the basis for further researchin vision assisted navigation and control. Using the availablevision and other algorithms that the ROS framework has tooffer we are now able to explore a wide variety of moreintelligent image processing algorithms for a variety of tasks.

VII. ACKNOWLEDGEMENT

This work was supported by the Natural Sciences andEngineering Research Council (NSERC) through the NSERC

Canadian Field Robotics Network (NCFRN) and MemorialUniversity.

REFERENCES

[1] http://www.can232.com/?page id=16.[2] http://www.hitl.washington.edu/artoolkit/.[3] http://www.artag.net/.[4] J. Manley, “Unmanned Surface Vehicles, 15 Years of Development,”

Proceedings of Oceans ’08 MTS/IEEE, 2008.[5] J. Manley, “Development of the Autonomous Surface Craft ACES,”

Proceedings of Oceans ’97 MTS/IEEE, October, 1997.[6] M. Caccia, G. Bruzzone, and R. Bono, “A Practical Approach to Mod-

eling and Identification of Small Autonomous Surface Craft,” OceanicEngineering, IEEE Journal of, vol.33, no.2, pp.133-145, April 2008.

[7] W. Naeem, T. Xu, R. Sutton, and A. Tiano, “The design of a navigation,guidance, and control system for an unmanned surface vehicle forenvironmental monitoring,” Proceedings of the Institution of MechanicalEngineers, Part M: Journal of Engineering for the Maritime Environ-ment, June 1, 2008.

[8] http://www.auvsi.org/Home/.[9] M. T. Wolf, C. Assad, Y. Kuwata, A. Howard, H. Aghazarian, D. Zhu, T.

Lu, A. Trebi-Ollennu and T. Huntsberger, “360-degree visual detectionand target tracking on an autonomous surface vehicle,” J. Field Robotics,27: 819833. doi: 10.1002/rob.20371, 2010.

[10] -Ing. K. Etschberger, “Controller Area Network: Basics, Protocols,Chips and Applications,” IXXAT Press, 88250 Weingarten, Germany,2001.

[11] S. Spitzer, L. Luft, and D. Morschhauser, “NMEA 2000 Past, Presentand Future,” RTCM 2009 Annual Conference, 2009.

[12] http://pandaboard.org/.[13] http://www.ros.org/wiki/.[14] https://en.wikipedia.org/wiki/IP Code.[15] http://www.willowgarage.com/.[16] https://code.google.com/p/brown-ros-pkg/.[17] http://www.ozbotz.org/opencv-installation/.[18] http://opencv.org/[19] H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration

for a Video-based Augmented Reality Conferencing System,” AugmentedReality, 1999. (IWAR ’99) Proceedings. 2nd IEEE and ACM InternationalWorkshop on , vol., no., pp. 85, 94, 1999.