development of living body information and behavior ... · lying in bed and measures his...

15

Journal of Engineering Education ResearchVol. 17, No. 4, pp. 15~20, July 2014

Development of living body information and behavior monitoring system for nursing personAi Ichiki*,† ․Hidetoshi Sakamoto* ․ Yoshifumi Ohbuchi**Kumamoto University

ABSTRACTThe non-contact easy detecting system of nursing person's body vital information and their behaviors monitoring system are

developed, which consist of “Kinect” sensor and thermography camera. The “Kinect” sensor can catch the body contour and the body moving behavior, and output their imaging data realtime. The thermography camera can detect respiration state and body temperature, etc. In this study, the practicability of this system was verified.

Keywords: Engineering Education

I. Introduction1)

The items of the respiratory state, the body temperature

and the behaviors for the nursing are very important

observation items. Especially, the state of respiratory

monitoring is an important item which is indispensable　in

the detection of the “Sudden Infant Death Syndrome”(SIDS)

and the apnea syndrome. Polysomnograph(PSG) is widely

used in determining respiratory states. However, the largest

shortcoming of PSG is that it is expensive and its low

tolerance for the nursing patients and infants　by relatively

high invasiveness of the PSG .

In this study, the easy monitoring system of non-contact

respiratory state and body temperature for early detecting

the SIDS and the apnea syndrome was proposed .

The validation for practical use as follows was carried out.

1) Verification of the posture detecting performance of

“Kinect” sensor.

2) Automatically detecting the state of respiratory by

thermography camera.

3) Development and verification of the new monitoring

system with “Kinect” sensor.

Received 21 June, 2014; Revised 21 June, 2014Accepted 30 July, 2014† Corresponding Author: [email protected]

II. Non-contact monitoring system of state of

respiratory state

The monitoring system outline of non-contact respiration

is shown in Fig 1. The initial system consists of the

infrared camera, CCD camera, room temperature measurement

unit and two personal computers. The CCD camera

detects a patient posture and his face contour, who is

lying in bed. The camera'a informations are used for the

thermography camera control. The thermography camera

measures his respiratory and his body temperature.

Fig. 2 shows an example of the tracking image of the

face detected with CCD camera. Recognizing his head area

by CCD camera, the face area is automatically displayed by

image processing. Next, the face area is extracted from

head area by skin color image matching. The mouth area

is initially set up manually. The face position is decided

Fig. 1 The body vital information monitoring system

Ai Ichiki․ Hidetoshi Sakamoto․ Yoshifumi Ohbuchi

공학교육연구 제17권 제4호, 201416

Fig. 2 The example of Posture tracking system by CCD camera

on position of mouth area by the face area's pattern

matching.

Fig. 3 shows the picture of image processing of

infrared image. The infrared camera detects the thermal

information around the face. By image processing of these

thermal information, the state of respiratory and the body

temperature can be obtained. The head region and face

area are detected by binary pictures of the face in infrared

camera. In setting up the mouth area in the face, the

face digital image obtained from CCD camera was used.

The state of respiratory is detected by using temperatures

fluctuate in the current of air by breath around the mouth.

However, this CCD camera has some problems about

the face recognition and the posture detection. So, we

replaced this camera with “Kinect” sensor for solving these

problems. Because this sensor can recognize the shape

and depth of the head, the image of the head can be

quickly monitored. The “Kinect” sensor is shown in Fig. 4.

Fig. 3 The picture of image processing by infrared camera

Fig. 4 “Kinect” sensor

III. Validation of the monitoring system

1. Verification of the posture detecting performance

of “Kinect” sensor

a. Detection ability of posture by “Kinect” sensor.

In this study, CCD camera was replaced with “Kinect”

sensor, and the verification about two items, that is, the

face recognition and the posture detection, were carried

out. Table 1 shows the comparison between CCD camera

and “Kinect” sensor. The front face tracking image is

shown in Fig. 5 By using “Kinect” sensor, the following

three items were improved.

• Auto detection of mouth area.

• Skip of skin color image process.

• Wide traceability of face rotation.

Table 1 Comparison between CCD camera and “Kinect” sensor

CCD camera "Kinect" sensor

Face recognitionSetting skin color

extractionInfrared distance sensor

Posture detectionRelative position in

mouth area to face area

The angle of face

rotation

Fig. 5 The front face tracking image by "Kinect" sensor

Development of living body information and behavior monitoring system for nursing person

Journal of Engineering Education Research, 17(4), 2014 17

Table 2 Face recognition angle

Distance from sensor and face/

Rotation velocity60cm 70cm 80cm 90cm 100cm 110cm 120cm

45deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

90deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

180deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

b. Evaluation of posture recognition ability by “Kinect”

sensor

We evaluated relationship of distance from “Kinect”

sensor to his face. The distance from "Kinect" sensor to

the face was changed, and the face rotation speed and

the face recognition angle were examined. Table 2

shows the results. From this table, the recognition angle

is independent of the distance from sensor and face.

2. Evaluation of automatically respiratory state

detecting by infrared camera.

a. Automatic setting of threshold by infrared camera.

• Threshold 1(two-valued of head area)

The histogram of two-valued image shows the

bimodality. The two-valued image separate the face area

and the background clearly. We calculated of threshold of

the valley part in histogram with mode method. Fig. 6 shows

an image processing as an example of thermography

camera. Fig. 6 (a) and (b) show the origin picture and the

picture after image processing respectively.

• Threshold 3( Breath detection)

The value of threshold 3 fluctuates widely by room

temperature and body temperature. We tried every threshold

value from 100 to 200. When the breath detection flag

appeared, we decided the value of threshold 3 and

determined as threshold. Fig. 7 shows an example of

image processing in infrared camera.

b. Validation of threshold 1 by infrared camera.

Here, when the body temperature change, the variance

of the threshold 1 was examined. In threshold 1

obtained by above trial, the face recognition ratio is

100% when the body temperatures are from 30 to 37

degree Celsius.

(a) The face origin picture

(b) The picture after face image processing

Fig. 6 Image processing example of thermo camera

Fig. 7 Nasal breathing



(a) RGB image

(b) Depth image

(c) Infrared image

Fig. 8 Image processing example of “Kinect” sensor

IV. Development of the new monitoring system

by using “Kinect” sensor

In this research, CCD camera was replaced with “Kinect”

sensor (4.1.1) for the system improvement because of

there were some problems in the face recognition and

the posture detection. In the result, using “Kinect” sensor,

it was found that the face recognition, posture and

respiration are non-contact measurable by non-contact .

So, the new monitoring system by using only “Kinect” sensor

was developed(System 2) and compared with conventional

monitoring system (System1).

Fig. 8 shows image processing of “Kinect” sensor. Fig. 8(a)

is RGB image, Fig. 8(b) is Depth image and Fig. 8(c) is

Infrared image, respectively.

1. The monitoring system of “Kinect” sensor

“Kinect” sensor detects the patient's posture who is

lying in bed and measures his respiratory state with RGB

camera and depth sensor. In the posture measurement,

the multi-“Kinect” sensors are available. This system

(a) Posture measurement

(b) Respiration measurement

Fig. 9 The body vital information monitoring system (System2)

use two sensor and one is set at foot side and the other

is head side. Fig. 9 shows the body vital information

monitoring system (System 2). Fig. 9(a) is posture measurement,

Fig. 9(b) is respiration measuring, respectively.

2. Development of the monitoring system with

“Kinect” sensor(System2)

The posture measurement system tracks the body

motion and saves the image when the body moves.

Respiration measuring system detects the breathing state

and records the breathing count every minute.

a. Development of the posture measurement system

The posture is obtained by measuring the shortest

distance from “Kinect” sensor to nursing person. This

detection use the distance measuring function of “Kinect”

sensor. Fig. 10 shows the image processing of the

posture measurement system. The RGB camera of “Kinect”

sensor measures the body from foot side and head side.

The light blue point of the depth image is the minimum

Fig. 10 The image processing of the posture measurement system

Development of living body information and behavior monitoring system for nursing person

Journal of Engineering Education Research, 17(4), 2014 19

Fig. 11 The night measuring with the infrared camera

Fig. 12 The recoded data

depth and detects his posture. In the measurement at

night, RGB camera changes into the infrared camera

equipped with “Kinect” sensor. The night measuring

image with the infrared camera is shown in Fig. 11.

Also, this system records the RGB processing image

when the body moves. RGB image recording is decided

by the movement of the minimum depth. The recording

data is recorded the current data and time. Fig. 12 shows

the recorded data.

b. Development of the respiration measuring system

We developed the respiration measuring system by

using RGB camera and depth camera of “Kinect” sensor.

The image processing is shown in Fig. 13. The breathing

state is detected by movement of depth change of the

chest. Fig. 14 shows the flowchart of the breathing detection.

In the measurement at night, RGB camera changes into

the infrared camera. In the night measuring, RGB camera

is switched the infrared camera. The night measuring

with the infrared camera is shown in Fig. 15. Also, the

system records the breathing count every minute. Fig. 16

shows the logging data of the starting measurement time

and the breathing count.

Fig. 13 The image processing

Fig. 14 The flowchart of the breathing detection

Fig. 15 The night measuring with the infrared thermo camera

Fig. 16 The logging data of the breathing count



V. The comparison of system 1 with system 2

In the posture measurement of the system 1, the traceable

limit of face rotation is 45～55 degree. In the system 2,

the traceable limit of face and body rotation are 90 degree.

In the respiration measuring system, we evaluated the

precision of the breathing state. The breathing data of system

2 corresponded to the standard breathing. The accuracy of

measuring the breathing improved by using “Kinect” sensor.

VI. Conclusions

<System 1>

• Using “Kinect” sensor instead of CCD camera, the system

improvement was carried out for practical use as follows.

(a) Automatic detection of mouth area and face area.

　 (b) Skip of the skin color detection process .

　 (c) Improvement of traceable limit of face rotation.

• By calculating threshold 1 and 3 automatically, the setting

time for measurement has been greatly shortened.

<System 2>

• “Kinect” sensor is an effective sensor to the posture

measurement and the respiration measurement.

• The performance of the posture measurement and the

respiration measuring is improved by replacing the

CCD camera into “Kinect” sensor.

• By combining the system 1 and system 2 organically, the

higher precise non-contact monitoring system is obtained.

References

1. Nobuhiro YOSHITAKE, An Implementation of Posture

Detection Functions to Inpatient Monitoring Systems using

Kinect, IPSJ SIG Technical Report(2013), 1－8

2. Yuhki TAKAHASHI, Development of System for Prevention

of Midnight Prowl Using Kinect, 2013 The Institute of

Industrial Applications Engineers Japan(2013), 42-43

3. Mariko AKIMOTO, A sheet-type device for home-monitoring

sleep apneas in children, Sleep and Biological Rhythms

(2011), 103-111

4. Tomohito HAYASHI, Study of Non-Restrictive Sleep Monitor

With Air-Matt Sensor, The Japan Society Mechanical Engineers

(2002), 71-74

5. Shunji HYUGA,”Let’s make the Kinect for Windows

application! ”(2012)

6. Tomoaki UEDA, The sensing world changing by “Kinect”

sensor, http://www.neo-tech-lab.co.uk/arsensing/

Ai Ichiki

Engineer of TOSHIBA Cooperation LTd.

Received BS (2012), MS (2014) in Mechanical System

Engineering from Kumamoto University, Japan. Her work

experiences are Product Engineer of TOSIBA Coopration LTd.

(2014), Her current research focuses on the production

technology of turbine for power electronics.

Phone: +81963423735

Fax: +81963423729

E-mail: [email protected]

Hidetoshi Sakamoto

Professor, Doctor of Engineering, at the Mechanical System

Division, Graduate School of Science and Technology, Kumamoto

University, Japan. He is received Master degree of Mechanical

Engineering by Kumamoto University, Japan in 1977, and got

his doctor's degree of Engineering from Kyushu University, Japan

in 1991. He was the Kyushu Branch Head of The Society of

Materials Science, Japan, 2006-2008 and now the Director of Infrared Thermography

Committee of The Society of Experimental Mechanics, Japan. He is also a member

of WIT international Science committee of “Computational Mechanics and Experimental

Methods”, “High performance Structures and Materials”, “Contact and Surface

Treatments”, and an editorial board member of WIT International Journal of Modeling

and Simulation He is the director of International conference on Far East Fracture

of Strength. The field of his research includes Solid Mechanics, Computer mechanics,

Sheet metal forming, High-speed fracture and deformation analysis, Biomaterial materials

strength evaluation and Engineering educational support program development, etc.

Phone: +81-96-342-3735

Fax: +81-96-342-3729


Yoshifumi Ohbuchi

Associate Professor, Doctor of Engineering, Creative Engineering

and Design Education Center Faculty of Engineering, Kumamoto

University, Japan. He is received Master degree of Mechanical

Engineering by Kumamoto University, Japan in 1985, and got

his doctor's degree of Engineering from Tokyo Institute of

Technology, Japan in 2002. He was a member of Kumamoto

University from 1985 to 1993. He was Visiting Researcher of Tokyo Institute of Technology

from 1993 to 1994. He was Associate Professor of Fukuoka Institute of Technology,

Japan from 2003 to 2005. Since 2005, he is a Creative Engineering & Design Education

Center Faculty of Engineering, Kumamoto University, Japan. Interesting Research Area

are Engineering Education, Creative Engineering and Design, Succession Methods of

Traditional Craftsmanship and Skill, Metal cutting and grinding simulation.

Phone: +81-96-342-3732

Fax: +81-96-342-3729


development of living body information and behavior ... · lying in bed and measures his...

Documents