intelligent robotics and embedded systems at the

Presenting the TIERS Lab University of Turku, Finland

Intelligent Robotics and Embedded Systems at theUniversity of Turku

Turku Intelligent Embedded and Robotic Systems (TIERS) LabDepartment of Computing

Faculty of TechnologyUniversity of Turku, Finland

https://tiers.utu.fi

The Turku Intelligent Embedded and Robotic Systems (TIERS) Lab was established at the University of Turkuin 2018 with the objective of initiating a new research group at the intersection of edge computing, distributedautonomous systems and autonomous robots. These built together towards robust and resilient multi-robotsystems. The TIERS Lab works in algorithmic design of autonomous systems, with the constant aim of puttingresearch in action. This is possible with extensive equipment: unmanned aerial, ground and surface vehicles(UAVs. UGVs and USVs), from micro-aerial vehicles (MAVs) to state-of-the-art multirotor platform (e.g., DJI’sMatrice 300). These are supported by a wide array of sensors, including multi-spectral and thermal cameras,2D and 3D rotating lidars, solid-state 3D lidars, radars and other types of wireless sensors, among others.

Abstract: This report briefly introduces the key research directions and recent developmentswithin the TIERS Lab at the Faculty of Technology, University of Turku, Finland. These areas are,namely, edge computing, autonomous robots and multi-robot systems. Within all this areas, a keyfocus is on embedding intelligence through lightweight machine learning and dynamic offloading inedge devices and mobile robots. We develop algorithms for collaborative and decentralized decisionmaking, distributed control of multiple mobile robots, and collaborative sensing. Key technologiescurrently in use are distributed ledger technologies (DLTs), among them blockchain, and ultra-wideband (UWB) wireless communication and ranging. Since it was established, the TIERS labhas initiated research collaboration in three national Finnish projects: AutoSOS, RoboMesh, andForesail.

Index Terms: Robotics; Autonomous systems; Multi-robot systems (MRS); Edge Computing;Edge AI; Deep learning (DL); Robot learning; Reinforcement learning (RL); Federated learning(FL); Navigation; Mapping; Localization; Search and Rescue (SAR).

1. IntroductionAt the TIERS lab we carry out interdisciplinary research in topics ranging from the designof autonomous systems and multi-robot systems to FPGA-based hardware accelerators or thedefinition of novel edge computing architectures. In multi-robot systems, our main interests arein the areas of decentralized control, collaborative and heterogeneous multi-robot systems, andblockchain for distributed robotic systems. From the perspective of autonomous robotic solutions,our research focuses in localization and mapping in dense urban environments, mapping of un-structured environments, computational offloading techniques, and hardware accelerators for ROS.We work at the edge, with research covering topics from the definition of novel edge computingarchitectures to the deployment of artificial intelligence at the edge. Our focus is on embedded anddistributed intelligence.

Turku Intelligent Embedded and Robotic Systems (TIERS) Lab

https://tiers.utu.fi


2. Autonomous Drones Supporting Maritime Search and Rescue (AutoSOS)Rescue vessels are the main actors in maritime safety and rescue operations. Aerial drones bring asignificant advantage into this scenario. Therefore, AutoSOS develops an autonomous multi-robotsearch and rescue assistance platform capable of sensor fusion and object detection in embeddeddevices using novel lightweight AI models. The platform performs reconnaissance missions for initialassessment of the environment using novel adaptive deep learning algorithms that efficiently usethe available sensors and computational resources on drones and rescue vessel. When drones findpotential objects, they will send their sensor data to the vessel to verity the findings with increasedaccuracy. The actual rescue and treatment operation are left as the responsibility of the rescuepersonnel. The drones will autonomously reconfigure their spatial distribution to enable multi-hopcommunication, when a direct connection between a drone transmitting information and the vesselis unavailable (see Figures 1 and 2).

The AutoSOS project will base its research exploration on the previous work in the development ofthe world’s first autonomous ferry (Brighthouse Intelligence Oy), research on hybrid aerial-surface-underwater autonomous systems for rescue operations (Tampere University and Alamarin-Jet Oy,aCOLOR project), and algorithms for drones in the areas of formation control and autonomouscooperation in multi-agent systems (University of Turku).

3. Beyond 5G Distributed Ledger Technology driven Mesh for Industrial RobotCollaboration (RoboMesh)The robotization of industry is one of the key drivers behind the Industry 4.0 revolution. Collabora-tive robots are becoming a reality across the manufacturing industry, autonomous robots are alreadya key asset in the logistics sector, and UAVs are being used for inspection and monitoring in diversedomains. Ubiquitous robots with augmented connectivity are merging into the Industrial Internet ofThings, enabling higher degrees of intelligence through computational offloading. RoboMesh delvesinto the design and development of a framework for collaboration and long-term autonomy indistributed and heterogeneous multi-robot systems based on a Beyond-5G wireless mesh networkwith built-in distributed ledger technology. This framework involves data sharing, collaborativedecision making, and dynamic and adaptive computational offloading, while it serves as the basisfor interaction between robots and infrastructure with collaborative sensing and multi-modal sensorfusion approaches (see Figure 3).

The multidisciplinary nature and the ambitious targets set by the project requires collaborationbetween partners having strong know-how and complementary expertise. RoboMesh brings together



Figure 3: Illustration of potential applications of the RoboMesh framework.

the competence and effort of the recognized experts in edge computing and distributed processingfor multi-robot systems (TIERS/Univ. Turku), wireless communications (CWC/Univ. of Oulu),and advanced robotic systems and solutions (BISG/Univ. Oulu).

4. Active research areasActive research areas in TIERS include multi-robot coordination [1], [2], [3], [4], [5], swarm design [6],[7], [8], [9], UWB-based localization [10], [11], [12], [13], [14], [15], localization and navigation inunstructured environments [16], [17], [18], lightweight AI at the edge [19], [20], [21], [22], [23],distributed ledger technologies at the edge [24], [25], [26], [27], [28], [29], edge architectures [30],[31], [32], [33], [34], [35], offloading for mobile robots [36], [37], [38], [39], [40], [41], [42], LPWANnetworks [43], [44], [45], [46], sensor fusion algorithms [47], [48], [49], and reinforcement and federatedlearning for multi-robot systems [50], [51], [52], [53].

References[1] Jorge Peña Queralta, Jussi Taipalmaa, Bilge Can Pullinen, Victor Kathan Sarker, Tuan Nguyen Gia, Hannu

Tenhunen, Moncef Gabbouj, Jenni Raitoharju, and Tomi Westerlund. Collaborative multi-robot search andrescue: Planning, coordination, perception and active vision. IEEE Access, 8:191617–191643, 2020.

[2] Jorge Peña Queralta, Cassandra McCord, Tuan Nguyen Gia, Hannu Tenhunen, and Tomi Westerlund.Communication-free and index-free distributed formation control algorithm for multi-robot systems. ProcediaComputer Science, 2019. The 10th International Conference on Ambient Systems, Networks and Technologies(ANT).

[3] Cassandra McCord, Jorge Peña Queralta, Tuan Nguyen Gia, and Tomi Westerlund. Distributed progressiveformation control for multi-agent systems: 2D and 3D deployment of UAVs in ROS/Gazebo with rotors. pages1–6. IEEE, 2019.

[4] Jorge Peña Queralta, Li Qingqing, Tuan Nguyen Gia, Zhuo Zou, Hannu Tenhunen, and Tomi Westerlund.Distributed progressive formation control with one-way communication for multi-agent systems. pages 2012–2019. IEEE, 2019.

[5] Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, and Tomi Westerlund. Collaborative mapping withioe-based heterogeneous vehicles for enhanced situational awareness. IEEE, 2019.

[6] Jorge Peña Queralta and Tomi Westerlund. Blockchain-powered collaboration in heterogeneous swarms of robots.Preprint, 2020. Presented at the 2019 Symposium on Blockchain for Robotics and AI Systems, MIT Media Lab.

[7] Jorge Peña Queralta, Li Qingqing, Tuan Nguyen Gia, Hong-Linh Truong, and Tomi Westerlund. End-to-enddesign for self-reconfigurable heterogeneous robotic swarms. IEEE, 2020.

[8] Jorge Peña Queralta, Yu Xianjia, Li Qingqing, and Tomi Westerlund. Towards large-scale scalable MAV swarmswith ROS2 and UWB-based situated communication. 2021. Best paper award.



[9] Jorge Peña Queralta, Jenni Raitoharju, Tuan Nguyen Gia, Nikolaos Passalis, and Tomi Westerlund. Autosos:Towards multi-UAV systems supporting maritime search and rescue with lightweight AI and edge computing.arXiv, 2020.

[10] Jorge Peña Queralta, Li Qingqing, Fabrizio Schiano, and Tomi Westerlund. Vio-UWB-based collaborativelocalization and dense scene reconstruction within heterogeneous multi-robot systems. arXiv Preprint, 2020.

[11] Jorge Peña Queralta, Carmen Martínez Almansa, Fabrizio Schiano, Dario Floreano, and Tomi Westerlund.UWB-based system for UAV localization in GNSS-denied environments: Characterization and dataset. IEEE,2020.

[12] Carmen Martínez Almansa, Wang Shule, Jorge Peña Queralta, and Tomi Westerlund. Autocalibration of amobile UWB localization system for ad-hoc multi-robot deployments in GNSS-denied environments. volume2626, pages 1–10. CEUR WS Proceedings, 2020.

[13] Wang Shule, Carmen Martínez Almansa, Jorge Peña Queralta, Zhuo Zou, and Tomi Westerlund. UWB-based localization for multi-UAV systems and collaborative heterogeneous multi-robot systems: a survey.Procedia Computer Science, 175:357–364, 2020. The 15th International Conference on Future Networks andCommunications.

[14] Yu Xianjia, Li Qingqing, Jorge Peña Queralta, Jukka Heikkonen, and Tomi Westerlund. Applications of UWBnetworks and positioning to autonomous robots and industrial systems. arXiv preprint, 2021.

[15] Yu Xianjia, Li Qingqing, Jorge Peña Queralta, Jukka Heikkonen, and Tomi Westerlund. Cooperative UWB-based localization for outdoors positioning and navigation of UAVs aided by ground robots. arXiv preprint,2021.

[16] Li Qingqing, Paavo Nevalainen, Jorge Peña Queralta, Jukka Heikkonen, and Tomi Westerlund. Localization inunstructured environments: Towards autonomous robots in forests with delaunay triangulation. Remote Sensing,2020.

[17] Paavo Nevalainen, Qingqing LI, Timo Melkas, Kirsi Riekki, Tomi Westerlund, and Jukka Heikkonen. Navigationand mapping in forest environment using sparse point clouds. Remote Sensing, 2020.

[18] Paavo Nevalainen, Parisa Movahedi, Jorge Peña Queralta, Tomi Westerlund, and Jukka Heikkonen. Long-termautonomy in forest environment using self-corrective slam. arXiv preprint, 2020. Presented at FinDrones 2020.

[19] Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, and Tomi Westerlund. Edge-AI in lora-based healthmonitoring: Fall detection system with fog computing and lstm recurrent neural networks. pages 601–604. IEEE,2019.

[20] Li Qingqing, Jussi Taipalmaa, Jorge Peña Queralta, Tuan Nguyen Gia, Moncef Gabbouj, Hannu Tenhunen, JenniRaitoharju, and Tomi Westerlund. Towards active vision with UAVs in marine search and rescue: Analyzinghuman detection at variable altitudes. IEEE, 2020.

[21] Zhuo Zou, Yi Jin, Paavo Nevalainen, Yuxiang Huan, Jukka Heikkonen, and Tomi Westerlund. Edge and fogcomputing enabled AI for IoT - an overview. IEEE, 2019.

[22] Tuan Nguyen Gia, Li Qingqing, Jorge Peña Queralta, Zhuo Zou, Hannu Tenhunen, and Tomi Westerlund. EdgeAI in smart farming IoT: Cnns at the edge and fog computing with lora. IEEE, 2019.

[23] Aly Metwaly, Jorge Peña Queralta, Victor Kathan Sarker, Tuan Nguyen Gia, Omar Nasir, and Tomi Westerlund.Edge computing with embedded AI: Thermal image analysis for occupancy estimation in intelligent buildings.page 1–6. ACM, 2019.

[24] Jorge Peña Queralta and Tomi Westerlund. Blockchain for mobile edge computing: Consensus mechanisms andscalability. Mobile Edge Computing (Book Chapter), 2020.

[25] Jorge Peña Queralta, Li Qingqing, Eduardo Castelló Ferrer, and Tomi Westerlund. Secure encoded instructiongraphs for end-to-end data validation in autonomous robots. arXiv preprint, 2020.

[26] Jorge Peña Queralta, Li Qingqing, Zhuo Zou, , and Tomi Westerlund. Enhancing autonomy with blockchainand multi-access edge computing in distributed robotic systems. IEEE, 2020.

[27] Anum Nawaz, Jorge Peña Queralta, Jixin Guan, Muhammad Awais, Tuan Nguyen Gia, Ali Kashif, Haibin Kan,and Tomi Westerlund. Edge computing to secure IoT data ownership and trade with the ethereum blockchain.Sensors, 2020.

[28] Anum Nawaz, Tuan Nguyen Gia, Jorge Peña Queralta, and Tomi Westerlund. Edge AI and blockchain forprivacy-critical and data-sensitive applications. IEEE, 2019.

[29] Tuan Nguyen Gia, Anum Nawaz, Jorge Peña Queralta, Hannu Tenhunen, and Tomi Westerlund. Artificialintelligence at the edge in the blockchain of things. volume 320. Springer, 2019. Presented at the 8th EAIInternational Conference on Wireless Mobile Communication and Healthcare.

[30] Behailu Negash, Tomi Westerlund, and Hannu Tenhunen. Towards an interoperable internet of things througha web of virtual things at the fog layer. Future Generation Computer Systems, 2019.

[31] Syed Kakakhel, Tomi Westerlund, Masoud Daneshtalab, Zhuo Zou, Juha Plosila, and Hannu Tenhunen. Aqualitative comparison model for application layer IoT protocols. IEEE, 2019.

[32] Imed Saad Ben Dhaou, Aron Kondoro, Syed Rameez Ullah Kakakhel, Tomi Westerlund, and Hannu Tenhunen.Internet of things technologies for smart grid. 2020.

[33] Syed Rameez Ullah Kakakhel, Aron Kondoro, Tomi Westerlund, Imed Ben Dhaou, and Juha Plosila. Enhancingsmart grids via advanced metering infrastructure and fog computing fusion. IEEE, 2020.

[34] Mehar Ullah, Syed Rameez Ullah Kakakhel, Tomi Westerlund, Annika Wolff, Dick Carrillo, Juha Plosila, andPedro H. J. Nardelli. IoT protocol selection for smart grid applications: Merging qualitative and quantitativemetrics. MIPRO Croatian ICT Society (MIPRO), 2020.

[35] Diana Rwegasira, Imed Ben Dhaou, Syed Kakakhel, Tomi Westerlund, and Hannu Tenhunen. Distributed loadshedding algorithm for islanded microgrid using fog computing paradigm. IEEE, 2020.

[36] Li Qingqing, Jorge Peña Queralta, Tuan Nguyen Gia, and Tomi Westerlund. Offloading monocular visualodometry with edge computing: Optimizing image quality in multi-robot systems. pages 22–26. ACM, 2019.



[37] Li Qingqing, Fu Yuhong, Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, Zhou Zou, and TomiWesterlund. Edge computing for mobile robots:multi-robot feature-based lidar odometry with fpgas. pages 1–2,2019.

[38] Victor Kathan Sarker, Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, and Tomi Westerlund.Offloading slam for indoor mobile robots with edge-fog-cloud computing. IEEE, 2019.

[39] Li Qingqing, Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, Zhou Zou, and Tomi Westerlund. Visualodometry offloading in internet of vehicles with compression at the edge of the network. pages 1–2, 2019.

[40] Tuan Nguyen Gia, Qingqing Li, Jorge Peña Queralta, Zhuo zou, Hannu Tenhunen, and Tomi Westerlund. Losslesscompression techniques in edge computing for mission-critical applications in the IoT. IEEE, 2019.

[41] Jorge Peña Queralta, Fu Yuhong, Lassi Salomaa, Li Qingqing, Tuan Nguyen Gia, Zhuo Zou, Hannu Tenhunen, ,and Tomi Westerlund. Fpga-based architecture for a low-cost 3D lidar design and implementation from multiplerotating 2D lidars with ROS. IEEE, 2019.

[42] Victor Kathan Sarker, Prateeti Mukherjee, and Tomi Westerlund. Enhanced reliability of mobile robots withsensor data estimation at edge. IEEE, 2020.

[43] Jorge Peña Queralta, Tuan Nguyen Gia, Zhuo Zou, Hannu Tenhunen, and Tomi Westerlund. Comparativestudy of lpwan technologies on unlicensed bands for m2m communication in the IoT: beyond lora and lorawan.Procedia Computer Science, 2019. The 14th International Conference on Future Networks and Communications(FNC).

[44] Victor Kathan Sarker, Jorge Peña Queralta, Tuan Nguyen Gia, Hannu Tenhunen, and Tomi Westerlund. Asurvey on lora for IoT: Integrating edge computing. IEEE, 2019.

[45] Victor Kathan Sarker, Tuan Nguyen Gia, Imed Ben Dhaou, and Tomi Westerlund. Smart parking system withdynamic pricing, edge-cloud computing and lora. volume 20. MDPI, 2020.

[46] Tuan Nguyen Gia, Jorge Peña Queralta, and Tomi Westerlund. Exploiting lora, edge, and fog computing fortraffic monitoring in smart cities. LPWAN Technologies for IoT and M2M Applications, 2020.

[47] Li Qingqing, Jorge Peña Queralta, Tuan Nguyen Gia, Zhuo Zou, and Tomi Westerlund. Multi sensor fusionfor navigation and mapping in autonomous vehicles: Accurate localization in urban environments. UnmannedSystems, 2020. The 9th IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and the9th IEEE International Conference on Robotics, Automation and Mechatronics (RAM).

[48] Li Qingqing, Jorge Peña Queralta, Tuan Nguyen Gia, Zhuo Zou, Hannu Tenhunen, and Tomi Westerlund.Detecting water reflection symmetries in point clouds for camera position calibration in unmanned surfacevehicles. pages 507–512. IEEE, 2019.

[49] Li Qingqing, Yu Xianjia, Jorge Peña Queralta, and Tomi Westerlund. Adaptive lidar scan frame integration:Tracking known MAVs in 3D point clouds. arXiv preprint, 2021.

[50] Wenshuai Zhao, Jorge Peña Queralta, Li Qingqing, and Tomi Westerlund. Towards closing the sim-to-real gapin collaborative multi-robot deep reinforcement learning. IEEE, 2020.

[51] Wenshuai Zhao, Jorge Peña Queralta, Li Qingqing, and Tomi Westerlund. Ubiquitous distributed deepreinforcement learning at the edge: Analyzing byzantine agents in discrete action spaces. Elsevier, 2020.

[52] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learningfor robotics: a survey. IEEE, 2020.

[53] Yu Xianjia, Jorge Peña Queralta, Jukka Heikkonen, and Tomi Westerlund. An overview of federated learningat the edge and distributed ledger technologies for robotic and autonomous systems. arXiv preprint, 2021.


intelligent robotics and embedded systems at the

Documents