Drowsiness Detection System UsingHeartbeatRate in Android-based Handheld Devices

Drowsiness Detection System usingHeart BeatRate in Android-based Handheld Devices

Advisor : Dr. Kai-Wei KePresenter : D. JayasakthiWireless and Broadband Networks Lab,Department of Electrical Engineering and Computer Science,National Taipei University of Technology, Taiwan-106. 1ContentsIntroductionObjectiveDesign and AnalysisImplementation Experimental ResultsConclusions

22IntroductionA common activity in most peoples life is driving; therefore, making driving safe is an important issue in everyday life.

Even though the drivers safety is improving in road and vehicle design, the total number of serious crashes is still increasing.

Most of these crashes result from impairments of the drivers attention.

33Drowsy driving is a serious issue in our society not only because it affects those who are driving while drowsy, but also it puts all other road users in danger.

Therefore, the use of assisting systems that monitor a drivers level of vigilance is important to prevent road accidents.

These systems should then alert the driver in the case of drowsiness or inattention.

44Drowsiness detection can be done in various ways based on the results of different researchers.

The most accurate technique towards driver fatigue detection is dependent on physiological phenomena like brain waves, heart rate etc.

Also different techniques based on the behaviors can be used, which are natural and non-intrusive.

These techniques focus on observable visual behaviors from changes in a humans facial features like eyes, head and face.

55ObjectiveThe aim of the thesis is develop a prototype for drowsiness detection system.

The application is developed using the android SDK and it will detect the heart beat signals from a heart rate monitoring device.

Also using this heart rate monitoring device, the ECG signals are obtained .

As a result of the electrical stimulation a change in potential of the order of 1mV can be measured during the cardiac cycle. This signal is known as the Electrocardiogram (ECG).

66ECG signal obtained from the sensor is analyzed in frequency domain.

In frequency domain, the power spectral density (PSD) is found.

From the PSD the Low Frequency(LF) to High Frequency(HF) ratio is estimated.

It is found that the LF/HF ratio decreases as the person becomes drowsy.

As a result the drowsiness of a person can be detected from this power ratio.

77How it WorksAutonomic Nervous System (ANS) activity presents alterations during stress, extreme fatigue and drowsiness.

Wakefulness states are characterized by an increase of sympathetic activity and/or a decrease of parasympathetic activity.

Extreme relaxation states are characterized by an increase of parasympathetic activity and/or a decrease of sympathetic activity.8

8The ANS activity can be measured non-invasively from the Heart Rate Variability (HRV) signal obtained from ECG.

Power on low frequency (LF) band (0.04-0.15 Hz) is considered as a measure of sympathetic activity.

Power on high frequency (HF) band (0.15-0.4 Hz) is considered of parasympathetic origin in classical HRV analysis.

Balance between sympathetic and parasympathetic systems is measured by the LF/HF ratio.

99Design and AnalysisVarious Signal Processing Methods needed to apply to the ECG signals are:DecimationHamming WindowFast Fourier TransformCalculate the low to high frequency ratio

1010DecimationConsider a band-limited discrete-time signal x(n) with a base-band spectrum X(f).

The sampling rate can be decreased by a factor of M through discarding of M1 samples for every M samples of x(n).

11h[k] Mx[n]v[n]y[n]FilterSampling Rate CompressorFsFs /M11Decimation by a factor of M can be achieved through a two-stage process of:

(a) Low-pass filtering of the zero-inserted signal by a filter with a cutoff frequency of Fs/M, where Fs is the sampling rate.

(b) Discarding of L1 samples for every L samples

The decimation factor is simply the ratio of the input rate to the output rate. It is usually symbolized by "M", so input rate / output rate=M.

1212The sampling frequency of the sensor was 250 Hz which means 250 samples per second.

It was very high to process the ECG signals.

So the sampling frequency was reduced by 50 Hz.

The decimation was done using a low pass filter technique.1313Hamming Window TechniqueWindowing functions, enhances the ability of an FFT to extract spectral data from signals.

Windowing functions act on raw data to reduce the effects of the leakage that occurs during an FFT of the data.

There are many window functions available.

For an ECG signal the appropriate window function is the Hamming Window.

The formula for Hamming window is w(n)=0.540.46cos(2n/N1).

If x(n) is the signal ,then we get the windowed signal by multiplying x(n) with the w(n) .

1414Fast Fourier Transform(FFT)The FFT is a highly elegant and efficient algorithm, which is still one of the most used algorithms in speech processing, communications, frequency estimation, etc.,

Basic radix-2 algorithm is used which requires N to be a power of 2.

FFT is applied to the windowed ECG signal.

By applying FFT , the power spectrum is found .

LF/HF ratio is calculated every 1 minute .

If this ratio decreases then the person is becoming drowsy.

15

15ImplementationVarious methods that has been implemented are:Bluetooth ModuleECG for Drowsiness DetectionHeart Beat MeasurementHeart Rate Variability

1616I-Mami HRM2 and Android Phone17

I-Mami HRM2 sensor from Microtime Computer Inc.Garmin Asus A5017Pairing Sensor with the MobileFirst the device discovery is done in order to connect the sensor with the mobile.

If a device is discoverable, it will respond to the discovery request by sharing some information, such as the device name and its unique MAC address.

Once a connection is made with a remote device for the first time, a pairing request will be automatically presented to the user.

The user must enter a 4 digit pin number for the device to be paired.1818Scan for Bluetooth DevicesPairing RequestSensor has been paired with the mobile

1919Bluetooth Communication20StartBluetooth ModuleObtain bluetooth device objectUse this object to acquire bluetooth socketInitialize bluetooth socketPerform lookup on remote device in order to match UUIDMatch UUID?Channels will not be openedShare RFCOMM ChanelObtain signals from sensorNoYesUUID - Universally Unique Identifier201. Main Screen with all Modules2.Bluetooth Module3. List of Paired Devices4. Sensor Connected to the Mobile5. Device Not Connected

Bluetooth Communication2121ECG SignalsElectrocardiography is the interpretation of the electrical activity of the heart over a period of time .

The ECG detector works mostly by detecting and amplifying the tiny electrical changes on the skin that are caused during each heartbeat.

The I-Mami HRM2 heart rate monitoring device is used to fetch the heart rate of a person and it is displayed in the android mobile with the help of programmable application, developed by using android SDK.

22221. Select Sensor from menu2. Displays paired devices3. Displays ECG Signals

ECG Signals2323Heart Rate MeasurementThe heart rate is the number of heart beats per minute.

Normal heart rate of a human being depends on the age. For example, children will have higher heart rates comparing with the adults.

This measurement can be done in various ways with respect to time.60 seconds (no calculation needed) - most accurate15 seconds (multiply by 4)10 seconds (multiply by 6)Less than 10 seconds = less precise

24241.Heart Rate Measurement Module2. Select a Device from Menu3. Lists the Paired Device4. Displays the Heart Rate and other values

2525Heart Rate VariabilityHeart rate variability (HRV), known as the variation of the period between consecutive heartbeats over time.In time domain analysis, based on beat to beat or NN intervals some variables are analyzed. They areSDNN: Standard Deviation of all normal to normal intervals index. Often calculated over a 24-hour period. SDANN, the standard deviation of the average NN intervals calculated over short periods, usually 5 minutes. SDANN is therefore a measure of changes in heart rate due to cycles longer than 5 minutes. NN50: Number of pairs of adjacent NN intervals differing by more than 50 ms in the entire recordingpNN50: The proportion of NN50 divided by total number of NNs.AVNN: Average of all NN intervals.

26261. Select a Device from Menu2. Select the Sensor 3.Displays the HRV

2727Implementation of Drowsiness Detection Technique28Obtain ECG signal from sensorReduce the sampling rate to 50 HzApply Hamming WindowApply FFTCalculate LF/HF ratioIs Ratio Decreasing?Person is not drowsyPerson Becomes DrowsyCollect next 512 samples of dataStartDrowsiness Detection Is max_count?Alert the person NoYesYesNoAAAA28Experimental SetupThe testing was performed on two persons, one male and one female. Data was collected when the persons were awake and asleep.Two hours of data was collected from each of them.The testing was repeated 10 times for both male and female and also when they were awake and asleep.

2929Experimental Results for Male, State: Awake30 Time Ratio Time Ratio Time Ratio Time Ratio10.2421310.2642610.2468910.233220.2388320.2425620.2638920.205530.2661330.2577630.2222930.169340.2537340.2153640.362940.231450.2305350.2087650.2022950.200160.2606360.1642660.2601960.203270.2823370.2302670.2147970.240380.2402380.1696680.2585980.220890.193390.2242690.243990.208100.2287400.2226700.20571000.1926110.1986410.2303710.19171010.235120.2254420.2488720.21461020.2028130.235430.2537730.24831030.2497140.2333440.2143740.22731040.2472150.244450.1815750.23851050.2351160.2342460.2065760.25841060.269170.2308470.2356770.20571070.2331180.2424480.2066780.20591080.227190.243490.2509790.22071090.2168200.2487500.3267800.21461100.2563210.2151510.1954810.18631110.2498220.2485520.1994820.25151120.2436230.3733530.2196830.20241130.258240.2247540.2394840.19721140.2463250.1823550.2682850.20341150.248260.2164560.1931860.311160.2664270.2235570.2305870.43131170.1511280.2562580.2599880.24211180.2676290.2353590.2595890.16871190.2349300.2079600.2168900.2371200.237130 Time Ratio Time Ratio Time Ratio Time Ratio10.2101310.2395610.4464910.233720.2054320.1931620.2369920.245930.2498330.2616630.2071930.250940.2184340.2512640.2206940.278950.2022350.2453650.2386950.258260.2252360.1892660.1996960.258470.3057370.2647670.1926970.234180.2415380.2533680.2202980.213290.2363390.1892690.2341990.2471100.2142400.2386700.25511000.224110.3224410.3341710.1941010.2278120.2765420.2438720.22121020.2573130.3478430.2074730.25121030.2221140.2392440.2519740.21681040.2475150.214450.1945750.23431050.2409160.2252460.2723760.24941060.2105170.1949470.2179770.2071070.1766180.2536480.2199780.21991080.2454190.2302490.2474790.24211090.2403200.2191500.2229800.20991100.2519210.2778510.2615810.27031110.2407220.2513520.2286820.19931120.2164230.2589530.2519830.22691130.2379240.2352540.2884840.26751140.2229250.2203550.2614850.25461150.3005260.242560.2437860.21481160.2508270.2518570.3833870.21341170.2846280.1995580.2126880.24251180.2016290.2414590.2533890.23921190.2406300.3818600.2311900.1971200.2311Experimental Results for Male, State: Awake3131Experimental Results for Male, State: AsleepTimeRatioTimeRatioTimeRatioTimeRatio10.2565310.1882610.1447910.121220.2506320.1889620.1555920.129330.2573330.1783630.1505930.127740.2033340.1788640.1487940.114450.1949350.1742650.1575950.112760.1995360.1837660.1515960.117770.1964370.1896670.1402970.125480.1902380.1811680.1443980.192290.2493390.1784690.1587990.1224100.2393400.1774700.14931000.1473110.2236410.1702710.15261010.1132120.213420.1832720.15411020.1166130.1981430.176730.14241030.1143140.2287440.1805740.15061040.1139150.1925450.1774750.13011050.1157160.1881460.1506760.13341060.1019170.1932470.1654770.13421070.1064180.1936480.1468780.14831080.1052190.1903490.1626790.13641090.1003200.1953500.1531800.13491100.1075210.1862510.1509810.14591110.1025220.1844520.1686820.14931120.1041230.1803530.1683830.13471130.1004240.1862540.1638840.13611140.1032250.1911550.1541850.12181150.1065260.194560.1592860.12821160.1007270.1815570.1501870.13541170.103280.1859580.1532880.13661180.182290.1871590.1426890.12031190.1142300.1797600.1412900.12571200.10923232Experimental Results for Male, State: AsleepTimeRatioTimeRatioTimeRatioTimeRatio10.2325310.1696610.1201910.102220.2216320.1702620.1291920.117830.2187330.1604630.1289930.118240.2122340.1576640.1241940.112650.2248350.1783650.1224950.104160.2483360.1545660.1273960.107370.2188370.1417670.112970.109580.1958380.1566680.1238980.116290.1727390.1678690.1125990.1192100.1969400.1459700.12781000.1163110.1575410.1542710.1291010.1121120.1521420.152720.11981020.1033130.1906430.1675730.10451030.1045140.1853440.1742740.11671040.1075150.1789450.1572750.10231050.1149160.1594460.17760.10671060.1131170.1957470.1591770.10341070.1173180.1884480.1458780.10231080.1182190.1822490.1556790.10931090.1267200.1718500.1494800.10011100.114210.1778510.1434810.1031110.1271220.1886520.1533820.1021120.1143230.1728530.1499830.10261130.1132240.1781540.1407840.11891140.1184250.1835550.1317850.11211150.1149260.1659560.1264860.10561160.1163270.1696570.1434870.10291170.1131280.1875580.1366880.11341180.1047290.1771590.1222890.11441190.1093300.1615600.1212900.1781200.10743333Experimental Results for Female, State: Awake Time Ratio Time Ratio Time Ratio Time Ratio10.3062310.4238610.2181910.246920.2559320.3814620.5302920.201330.2189330.1987630.2424930.24640.3066340.5112640.3827940.229850.2503350.2059650.3045950.227660.3134360.2616660.2575960.261670.2286370.2665670.2192970.238280.255380.2669680.4336980.238890.3032390.2048690.5218990.2829100.2448400.2394700.44831000.2018110.2244410.2523710.35621010.2648120.264420.2848720.22251020.238130.2231430.308730.27221030.211140.1928440.222740.22091040.2597150.1939450.2982750.24831050.2362160.3443460.3239760.25691060.2214170.2239470.2601770.26341070.246180.393480.2683780.2511080.248190.5402490.241790.26761090.2581200.2433500.2222800.19311100.2702210.249510.2652810.26221110.6572220.219520.2782820.20071120.3003230.2463530.2574830.23571130.2403240.2929540.4996840.27391140.2261250.275550.2294850.27061150.273260.6102560.3677860.25491160.235270.1954570.636870.26421170.259280.6796580.2986880.43031180.2495290.3054590.3553890.23391190.2603300.2364600.261900.21741200.23423434Experimental Results for Female, State: AwakeTimeRatioTimeRatioTimeRatioTimeRatio10.4251310.2311610.3823910.268120.3031320.2329620.2595920.223930.2604330.2434630.1994930.232440.2725340.2558640.2499940.244250.3506350.28650.2434950.223860.3885360.2521660.2428960.254770.2435370.2556670.2336970.2380.2413380.2468680.2392980.258290.2435390.2382690.7867990.2346100.2199400.2119700.20911000.2434110.2596410.2347710.24261010.2349120.4623420.2477720.23811020.263130.2209430.2291730.24511030.408140.2442440.2487740.23881040.2528150.2463450.2462750.25661050.2335160.2715460.1949760.22991060.254170.252470.2348770.23261070.2513180.2545480.2502780.23231080.2375190.273490.2565790.39331090.2241200.2423500.2327800.2571100.2356210.212510.2817810.23661110.2347220.2755520.2667820.2531120.1881230.2692530.222830.22671130.2278240.256540.36840.23551140.2328250.2186550.2389850.21671150.2262260.2419560.2326860.24691160.2824270.1862570.2554870.25851170.2763280.2272580.2509880.24251180.2341290.2145590.2453890.2771190.3311300.2594600.217900.2521200.29453535Experimental Results for Female, State: Asleep Time Ratio Time Ratio Time Ratio Time Ratio10.3251310.1811610.1623910.118120.3031320.1829620.1595920.113930.2604330.1934630.1694930.112440.2725340.1858640.1599940.124250.2506350.1928650.1434950.123860.2885360.1921660.1428960.124770.2435370.1856670.1596970.12380.2413380.1968680.1592980.128290.2435390.1882690.1567990.1246100.2199400.1919700.16911000.1244110.1996410.1847710.15261010.1149120.1923420.1777720.14811020.1163130.2209430.1891730.14511030.1108140.2442440.1887740.13881040.1128150.2463450.1762750.14661050.1035160.2715460.1949760.14991060.1054170.1952470.1848770.13261070.1013180.1945480.1702780.13231080.1075190.203490.1665790.12331090.1041200.2423500.1727800.12571100.1056210.212510.1817810.12661110.1047220.2755520.1767820.12531120.1081230.2692530.1622830.12671130.1078240.256540.1636840.13551140.1028250.2186550.1789850.11671150.1062260.1919560.1626860.13691160.1124270.1862570.1654870.12851170.1163280.1972580.1509880.12251180.1152290.1945590.1453890.12771190.1056300.1994600.1517900.12521200.10323636Experimental Results for Female, State: Asleep Time Ratio Time Ratio Time Ratio Time Ratio10.2487310.1934610.1479910.108420.2531320.1877620.1477920.109730.2663330.1786630.1506930.104540.2385340.1777640.1427940.101850.251350.1702650.1455950.109360.2474360.1588660.1329960.108770.233370.1618670.1499970.108680.2284380.1657680.1495980.112390.2392390.1725690.1564990.1182100.2621400.1723700.15471000.1045110.2381410.1712710.14741010.1189120.2052420.1638720.13861020.1225130.1914430.1651730.13021030.1131140.2076440.1653740.12431040.1064150.1893450.1581750.12881050.1012160.1608460.1785760.13081060.1041170.1925470.1605770.12981070.1053180.1996480.1742780.13491080.1005190.1992490.1542790.12061090.102200.1983500.1537800.12281100.1009210.1846510.1467810.10661110.1067220.1992520.1542820.11011120.1056230.1898530.1524830.11191130.1029240.1924540.1431840.11851140.1034250.1915550.1578850.11381150.1053260.1968560.1632860.12021160.1155270.1926570.1543870.10161170.1063280.1825580.1408880.11521180.1002290.1821590.1453890.10581190.1005300.1865600.1431900.10071200.11143737Comparison of Power Ratio for Awake and Sleep States

3838Comparison of Power Ratio for Awake and Sleep States

390.11905339ConclusionsIn this research work, drowsiness detection has been analyzed based on the ECG signal obtained from the sensor.

The application is successfully able to detect the heart beat accurately and it also displays the heart rate variability and the ECG signals in the android devices, respectively.

By applying FFT to the obtained ECG signal, the power on the low and high frequency components were measured. Then the LF/HF ratio was calculated for every one minute.

4040From the graph it is clear that the HRV analysis on the two hour heart rate time series showed that LF/HF ratio had a decreasing trend when they were asleep or feeling drowsy.

When a decreasing trend is identified below 0.17 and for a max_count value of 3, if the value continuously decreases, then an alarm will be invoked automatically in order to alert the driver.

4141My contributions for the thesis are Display the ECG signal successfully on the android mobile.Apply the signal processing techniques and process the obtained ECG signal.Find the power spectrum and calculate the LF/HF ratio and save in database for analysis purpose.Collect data when awake and asleep from both male and female for 2 hours.Analyze the power ratio values in both the cases.Detect the drowsy state from the collected data.Compare the power ratio values for both male and female.

4242Future WorksThe efficiency of this system could be further improved by employing the sensors on the seat belt to achieve better accuracy.

In addition more data must be collected from human test in order to improve the accuracy of drowsiness judgement.

Further with the help of the ECG, we can also analyze the persons mood on a daily basis. 434344

44