a study on the tele-medicine robot system with face to

A Study on the Tele-medicine Robot System with

Face to Face Interaction

Dae Seob Shin*★

Abstract

Consultation with the patient and doctor is very important in the examination. However, if the consultation

cannot be done directly, such as corona virus, it is difficult for the doctor to determine the patient’s condition

more accurately. Recently, an image counseling system has been developed based on the Internet, but in the case

of heart disease, remote medical counseling cannot be performed because it is not possible to stethoscope the

heart sounds remotely. In order to solve this problem, it is necessary to develop an interactive mobile robot

capable of remote medical consultation, and a doctor and a patient should be able to set a planting sound during

consultation and transmit it in real time. In this paper, we developed a robot that can remotely control a medical

counseling robot to move to a hospital room where patients are hospitalized, and to consult a patient in the room

remotely from a doctor’s office. A remote medical imaging stethoscope system for real-time heart sound

transmission is presented. The proposed system is a kind of P2P communication that transmits video information,

audio information, and control signal independently through webRTC platform, so that there is no data loss.

Consults and sees doctors in real time and finds it more effective than traditional methods for patient security.

The system implemented in this paper will be able to perform remote medical care in the place where the

spread of diseases between humans like the recent corona 19 as well as the remote medical care of heart disease

patients in the future.

Key words：Telemedicine, Face-to-face, WebRTC, Stethoscope, Mobile robot

* Dept. of Electronic Information Communication Division, Shin Ansan University

★ Corresponding author

E-mail：[email protected], Tel：+82-2-2679-8556

※ Acknowledgment

Manuscript received Mar. 10, 2020; revised Mar. 19, 2020; accepted Mar. 24, 2020.

This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License

(http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction

in any medium, provided the original work is properly cited.

Ⅰ. Introduction

Recently, with the emergence of smartphones

and tablet PCs in line with 5G technology, as the

information communication technology has been

developed and advanced, the interface technology

between human and computer has been developed,

and the interface technology between human and

computer has been diversified in various forms.

As 5G technology develops, u-health technology

is attracting attention as it is being combined

with an increase in the elderly population. Advances

in communication also mean that there is a

foundation for providing health care anywhere,

anytime[1][2].

As communication technology improves, efforts

to create added value in telemedicine using video

communication have been attempted in various

fields. In recent years, telemedicine has been

revisited due to u-health. There is a need for a

service that can receive medical services anywhere

outside the hospital, measure biometric information,

ISSN：1226-7244 (Print)ISSN：2288-243X (Online) j.inst.Korean.electr.electron.eng.Vol.24,No.1,293～301,March 2020논문번호 20-01-40 http://dx.doi.org/10.7471/ikeee.2020.24.1.293

293

(293)

294 j.inst.Korean.electr.electron.eng.Vol.24,No.1,293～301,March 2020

and provide consultation with a doctor without

going to the hospital.

In particular, as the elderly population increases,

the demand for remote monitoring of patients

without going to the hospital is increasing. In

addition, when a virus that spreads between humans

has appeared recently, such as a corona virus,

there is a need for a system that enables a doctor

to perform a medical examination remotely without

directly meeting a patient.

Even in hospitals, doctors need a system that

enables them to remotely manage their patients

without visiting the patient’s room. Types of

telemedicine are shown in Fig. 1. It can be divided

into three as follows. (a) may be a medical

consultation between the medical institution’s

ward and the physician. (b) may provide consultation

between school facilities and physicians. (c) enables

telemedicine, such as homes and health care

facilities. This study is an effective robotic system

for conducting medical consultations with patients

and medical institutions such as (a) and (c).

Fig. 1. Telemedicine Service Classification and Structure.

Fig. 2. Appearance of Remote Video Robot.

In medical consultations, doctors need a minimum

of photos and a stethoscope to accurately determine

a patient’s condition. Observation of the trauma

of the patient is possible through a remote imaging

robot. However, patients with heart disease are

difficult to consult because they cannot remotely

listen to heart sounds. It should be possible to

measure the heart sounds during the consultation

between the doctor and the patient and transmit

them in real time. If a system capable of measuring

such sound can be transmitted in conjunction

with a remote video robot, effective telemedicine

will be possible.

In this paper, we propose an video robot system

that can be operated remotely to check the condition

of the patient. Remote control is possible not

only from PC but also from smart device, and it

is configured so that doctors can conveniently

control the image robot using various sensors

and multi-touch technology built in smartphone

or tablet PC as well as from fixed PC[3]. In

addition, we designed, implemented and implemented

a telemedicine video consultation system for

heart sound transmission over the Internet. For

video communication, we use WebRTC platform,

which is more effective than the existing P2P

communication, to transmit video information, audio

information, and data information separately[4].

Ⅱ. Related Work

1. Remote Video Robot System Configuration

The remote image robot system is designed

and manufactured to be composed of three types.

* Robot Mechanism Design

First of all, the design of the robot’s external

mechanical part were carried out. In general, the

robot was designed with a low cost and simple

structure so that it can be used from homes to

medical institutions without burden.

In addition, the robot has two wheels, a 12-inch

tablet PC monitor with a height of 120cm, and

(294)

A Study on the Tele-medicine Robot System with Face to Face Interaction 295

Fig. 5. Robot base manufacturing process.

the screen can be moved by Fan / Tilt. Fig. 3

shows the drawings of the remote imaging robot

mechanism and modeled in 3D. In Fig. 4, the

robots have a friendly feeling by covering various

types of character dolls on the remote video robot.

Fig. 3. designed remote video robot base.

Fig. 4. Appearance design on remote video robot.

* Robot base production

The base part of the robot was manufactured

to fit the designed robot. Two motors and ball

casters were used. Fig. 5 shows the manufacturing

process of the robot base.

We manufactured the robot base, developed the

robot drive unit, and installed the ultrasonic

sensor around the robot to move the robot safely.

We also developed a control algorithm to drive

two motors.

① Development of Autonomous Driving Control

Algorithm for Mobile Robot

- Intelligent control algorithm and control method

for remote control driving of Mobile Robot

driven by 2 axis DC motor of Differential

Driving type were developed.

- The sensor part of Mobile Robot adopts IR

sensor and ultrasonic sensor to enable real-time

collision avoidance while the robot is driving,

and the camera is mounted on the upper part

of the robot so that the image of the robot in

front of the robot can be remotely monitored

It was sent to the operator.

Fig. 6. Autonomous Driving and Collision Avoidance Structure

of Mobile Robot.

* Robot Monitor Structure

LCD monitor for video communication was

designed. The monitor was designed with the

fan / tilt moving structure and driven by two

servomotors.

(295)


Fig. 7. Robot Monitor Fan / Tilt Design.

Fig. 8. Fan / Tilt using servo motor.

Fig. 9. Remote Video robot Appearance.

We designed and built the robot to the world

level to move the robot safely, and experimented

to confirm the range of motion.

(Main Function Spec) Unit World LevelDeveloped

Level

Speed control error cm/sec within ±10 ±10

Positioning errors cm within ±15 ±10

Ultrasonic sensor interface cm 5 2

Light sensor interface o o

Collision avoidance o o

Table 1. Acdeptable Goal

2. Electronic Stethoscope System

The remote image transmission robot was

designed to work with a stethoscope to check

not only the image but also the patient’s condition.

Auscultation is a diagnostic procedure performed

to detect heart failure and digestive status defects.

The doctor listens to the sound of the stethoscope

on the heart, lungs or intestine, the source of the

sound. Fig. 10 shows the structure of the

stethoscope.

Fig. 10. the structure of the stethoscope.

The stethoscope is transmitted to the ear

through the earpieces as the sound source

captured by the diaphragm moves up the tube.

This measured heart rate goes up the tube and

you hear only low notes due to the HPF

(Hight-Pass Filter). As a result, doctors cannot

hear the original sound of the patient’s measured

heart sound, and the sound of the low frequency

band is mixed with the heart sound every time.

Fig. 11. control structure of the electronic stethoscope.

Recently, the electronic stethoscope is widely

used. It was solved by filtering and amplification

(296)


Fig. 14. Working Process of WebRTC Peer to Peer

Communication.

as shown in Fig. 11. And while traditional

stethoscopes only listen to doctors, electronic

stethoscopes have the advantage of being able to

store and record.

Fig. 12. Connect to the tablet of the electronic stethoscope.

The existing electronic stethoscope was connected

to the robot and transmitted in real time as an

video, and the receiver realized the medical

consultation at the receiving end to realize a

remote medical service requiring a stethoscope.

Fig. 12 shows the structure of an electronic

stethoscope connected to a tablet. This system is

installed on the video transmission robot manufactured

earlier and transmits the video and the audio

signal of the electronic stethoscope to the doctor.

3. Program development

It is important to obtain accurate patient’s

medical information in real time for the treatment

between the patient and the doctor with a remote

video robot. Hardware configuration is important

for accurate medical information, but software

configuration is most important. Existing video

communication is the structure that sends and

receives the control signal of the robot through

the TCP Socket while controlling by UDP Socket

communication. However, it is true that accurate

examination is difficult because of various

problems in communication. However, in this

study, we conducted experiments by securing

secure communication using WebRTC platform.

Fig. 13 shows the structure of WebRTC.

Fig. 13. Architecture of WebRTC.

Browsers and mobile applications use Audio

and Video RTC (real-time communication) using

WebRTC through a simple API. The WebRTC

component has been optimized to best suit this

purpose. WebRTC-based web applications provide

rich real-time multimedia capabilities (think video

chat) on the web without plug-ins, downloads, or

installations, and help build a robust WebRTC

platform that works across platforms in multiple

(297)


web browsers. Fig. 14 shows a tank that transmits

video and audio signals with the WebRTC Peer

to Peer Communication procedure.

WebRTC Benefits:

• WebRTC is In-built in Firefox browser.

• Improved video and audio streaming.

• VP8 video codec and OPUS audio codec

provides much less data transmission without

packet loss.

Fig. 15 shows the process of running an Android

program using Eclipse.

Fig. 15. Android video control program development.

Fig. 16. Android tablet Control screen.

Fig. 17. Android smartphone screen.

Fig. 18. video transfer experiment after mounting on robot.

After completing the development of the remote

video robot, we made a character with a doll to

have a friendly relationship with the patient when

performing remote medical examination using a

real robot.

Fig. 19. Equipped the character on the developed guide

robot.

Ⅲ. Implementation and evaluation

In order to perform the experiment of the

implemented system, the experiment is composed

of the experiment of the robot control unit and

the hardware control experiment.

1. Control board and video transmission experiment

In order to test the robot controller, a program

written in C language was downloaded to the

ATmega128 board. After the Bluetooth pairing

was performed, the program was set up to 0×31,

0×32, 0×34 ... 0×38 and the servo was driven

(298)


using data such as 0×40, 0×41, 0×42, 0×43. Then,

a motion control experiment was performed using

a remote control app.

Fig. 20. Experimental procedure of video transmission and

motor drive.

2. Evaluation item experiment of development

technology

For the evaluation of the developed product, the

experiment was carried out at the temperature:

(20 ± 2) ℃ and humidity: (56 ± 5)% R. H. In this

section, we will briefly summarize the results and

data of the experiment.

Table 2-1. Experiment item and evaluation method.

No. Test Item Evaluation mothod

1Video transmission

speed⦁Video transmission time

measurement.

2 Sound size capacity⦁Speaker output noise

measurement.⦁Measure 1m from the front.

3 Operating time⦁Check the remaining charge

after operation / operation when fully charged.

4 Max Speed⦁Maximum moving speed

measurement

5 Battery usage indicator⦁LED indication.⦁Display remaining battery

Check.

6 weight ⦁Weight measurement.

7 Robot height ⦁Robot height measurement.

8After sensor response

Stop speed

⦁Stop speed measurement after sensor response.⦁Stop after detecting a forward

object while moving

9Stethoscope Frequency

MeasurementStored Stethoscope

Measurements

Sample photo (front) Sample photo (rear)

Table 2-2. Evaluation results.

No. Test Items Target value Evaluation results

1Video transmission

speed20 Frame

Video call with WebRTC platform

2 Sound size capacity 60 dBA/m 70.8 dBA

3 Operating time 8H 10H

4 Max Speed 30 cm/s 40.62 cm/s

5Battery usage

indicatorLED display LED display

6 weight 38 kg 17.80 kg

7 Robot height 120 cm 132.6 cm

8After sensor response

Stop speed1 s 0.20 s

9Stethoscope Frequency

Measurement8000Hz 8000Hz

3-3. Experiment content

The experiment was carried out according to

the evaluation method for each item, and the

evaluation of the sound and speed of the robot

and the frequency of the stethoscope sound were

summarized.

3-3-1. Robot output sound measurement

experiment

Speaker maximum output noise was measured

during video call with tablet PC. The noise

measuring room (width×length×height) was 4.98

×3.1×3,42 m and the distance from the robot was

measured at 1 m from the front. The noise

measuring instrument was performed by NA-27

(RION) and there were 10 measuring functions,

which made it easy to measure. The experiment

was performed three times to find the average

value.

(299)


[Measurement Data]

NO Room noise (dBA) Noise measurement (dBA)

1

41.4

70.1

2 69.5

3 71.4

Average value 70.8

[Evaluation results]

- Speaker maximum output noise result (69.5～

71.4) dBA.

Noise meter Noise measurement pictures

Fig. 21. Experimental process photo.

3-3-2. Stop speed experiment after sensor response

In order to improve the interaction between the

patient and the doctor through the robot, the

doctor responds quickly to the robot’s sensor

response to speed up the interaction and environment

recognition of the remote robot. The instrument

used was a stopwatch and a HS-6 (CASIO)

instrument.

[Measurement Data]

NO Reaction time (s) etc

1 0.22

2 0.18

3 0.17

Average value 0.20

[Evaluation results]

- Result of measurement of downtime after reaction

of object detection sensor (0.17～0.22) s.

Photos before the moveStill photo after detecting an

object while moving


3-3-3. Stethoscope storage experiment

Received the stethoscope sound transmitted from

the remote video robot and added the function to

store and play the stethoscope sound on the tablet

PC, and experimented through the storage and

playback screen on the screen. Fig. 23 shows a

screen for storing and playing stethoscope

sounds on the tablet screen.


Since the stethoscope sound must be transmitted

and stored, it is encoded to 8000Hz 8bit mono

type and it is configured to provide the function

of playing and recording the stethoscope sound

received by the doctor. The stethoscope sound

sent from the patient can be stored in the doctor’s

monitor and managed for video consultation

history. The following figure shows the playing

and stopping of the stored stethoscope. Through

experiments, we could hear and diagnose a

stethoscope sound through a remote imaging

robot. In addition, since the images are transmitted

using WebRTC, more than 20 frames of images

have been acquired for remote medical examination,

and it was confirmed through experiments that

the images were transmitted safely without any

breaks in the image system.

Ⅳ. Conclusions

In this study, we developed an video transmission

robot system that can move freely in the structure

proposed in the fixed PC for telemedicine service.

Even if the doctor and the patient are far away,

the video call is performed in real time as if they

are close, and the doctor can operate the robot

remotely to examine the patient’s condition in

various ways. For the configuration of the video

(300)


system, real-time video transmission is implemented

using the H263 codec, which is a video communication.

Unlike the method of providing the login function

to the server agent by using the TCP socket, the

webRTC platform is configured to separate the

video signal, the audio signal, and the control

signal so that the video is not interrupted and is

transmitted stably at a transmission speed of 20

frames or more per second. It confirmed that it

became. In addition, since the electronic stethoscope

system is mounted on the medical robot, the

doctor will consult with the patient to check the

planting in real time and receive and examine the

patient.

Using the system implemented in this study, it

is possible to monitor and remotely refer to

elderly patients at home or patients discharged

after heart disease surgery. Since the minimum

sound quality required for accurate diagnosis by

medical staff should be guaranteed, future research

projects will compare the sound quality before

and after the transmission of the stethoscope

sound, and conduct continuous experiments and

analysis for the effectiveness of the stethoscope

image counseling system.

In addition, various researches will be needed,

such as interactions that can safely control remote

robots using various sensors and multi-touch

technology mounted on smartphones or tablet PCs,

and environmental awareness technology of remote

robots.

References

[1] T. Cohen, “Medical and Information technologies.

converge,” IEEE Engineering in Medicine and

Biology Magazine, Vol.23, No.3 pp.59-65, 2004.

DOI: 10.1109/MEMB.2004.1317983

[2] W. Yuan, D. H. Guan, S. Y. Lee, and H. J. Lee,

“Using Reputation System in Ubiquitous Healthcare,”

The Proc. of 9th IEEE Int’l Conference on

e-Health Networking, pp.182-186, 2007.

DOI: 10.1109/HEALTH.2007.381626

[3] E. Pacchierotti, H. I. Christensen and P. Jensfelt,

“Human-Robot Embodied Interaction in Hallway

Settings: a Pilot User Study,” IEEE International

Workshop on Robots and Human Interactive

Comm, pp.164-171, 2005.

DOI: 10.1109/ROMAN.2005.1513774

[4] S. J Baek, R. H. Lee, C. S. Yi, “Design and

Development of A Systemic Structure to Ensure

the Interoperability between the WebRTC-based

Video Conferencing Systems and Heterogeneous

Terminals,” KIISE Transactions on Computing

Practices, Vol.23, No.4, pp.238-243. 2017.

DOI: 10.5626/KTCP.2017.23.4.238

BIOGRAPHY

Dae Seob Shin (Member)

1996：BS degree in Electronics

Engineering, Howon University.

1998：MS degree in Electronics

Engineering, Inha University.

2014：Ph. D. degree in Electrical and

Biomedical Engineering, Hanyang

University

2019～Present：adjunct Professor, Shin Ansan University

<Research Interests>

Image processing, neural network, Adaptive control, Signal

Processing, Embedded Control, Rehabilitation robots.

(301)

a study on the tele-medicine robot system with face to

Documents