cat fatigue technology report 2008

OperatOr Fatigue DetectiOn technOlOgy review

© 2008 Caterpillar All Rights Reserved. CAT, CATERPILLAR, their respective logos,“Caterpillar Yellow,” and the POWER EDGE trade dress as well as corporate and product identity used herein, are trademarks of Caterpillar and may not be used without permission. Cat and Caterpillar are registered trademarks of Caterpillar Inc., 100 N.E. Adams, Peoria IL 61629.

ExEcutivE Summary » introduction » tEchnology rEviEw » driving Simulation Study

» ProjEct rEcommEndationS » acknowlEdgEmEntS » aPPEndix 1: tEam mEmbEr bioS »

aPPEndix 2: ExPErt rEviEwEr bioS » aPPEndix 3: Product SummariES


1 / OperatOr Fatigue DetectiOn technOlOgy review

The technology review identified 22 technologies that were commercially available or considered as emerging technologies with potential for near term commercialization. The team then proceeded to gather data from the technology supplier, the scientific literature and customers (if available) to gather a good understanding of the background of the technology. The team then created a list of technology features and capabilities that would be used for evaluating the 22 technologies. The set of features was developed into a rating matrix consisting of 16 feature categories with 93 total features. The team provided a 1 to 10 weighting to each category and feature based on the importance of that category or feature to the mining customer. In addition to weighting the matrix for the mining customer, a set of weights was also established through discussions with fatigue industry experts in order to represent the population needs in general. To provide an unbiased, objective assessment of these technologies the team invited 5 international experts from a variety of related industries (transportation research, mining research, biosignal analysis, human factors and ergonomics research) to provide their input to the matrix for all of the technologies. The input from all experts was consolidated and the technologies were ordered from best to worst in terms of the experts’ scoring. The top 5 technologies based on the mining industry weightings were: ASTiD™ (Pernix), FaceLab (Seeing Machines), HaulCheck (Accumine), Optalert™ (Sleep Diagnostics) and the Driver State Monitor (Delphi).

At the present time ASTiD™, HaulCheck, and Optalert have been trialed in a mining application. Results from

the ASTiD™ and Optalert™ field trial are encouraging. FaceLab and the Driver State Monitor have only been trialed in on-highways applications. Plans are in place for a mining trial of the Seeing Machines product in the coming months. To compliment the field trial results from the three technologies mentioned above, the team conducted trials of the Driver State Monitor (Delphi) and FaceLab (Seeing Machines) using an interactive driving simulator. Simulator trials of the Delphi and Seeing Machines devices showed very good correlation with driving errors and fatigue; however the data suggested that more work could be done to increase the robustness of both systems before they would be ready for commercial release. In response, both Delphi and Seeing Machines are now examining what additional development would be required to adapt their technologies for the mining industry customer.

An additional outcome of the technology review and simulator research was an in-depth look at the use of head-nod sensors for fatigue detection. Two head-nod sensor products were included in the technology review, both of which had been used previously in the mining industry. The two products received extremely low scores from the expert reviewers. Use of one of these devices was included in the interactive driving simulator study to provide an objective measure of the head-nod sensors’ effectiveness and to determine if the low ratings are warranted. The device was plagued with numerous false alarms due to typical driving-related head movements and true alarms only accounted for 1% of the fatigue related driving errors.

Caterpillar launched the fatigue technology review project in January 2006. Since that time Caterpillar has conducted an in depth review of available and emerging fatigue detection technologies. In addition to the original scope of simply reviewing fatigue detection technologies, the team included follow-up evaluations of 3 of the top 6 technologies using an interactive driving simulator. The results provided in this report summarize the activities from both the technology review and the follow-up evaluation.

i. executive Summary



In summary, of the 22 technologies only three of the top rated technologies are immediately available: ASTiD™ (Pernix), HaulCheck (Accumine) and Optalert™ (Sleep Diagnostics). Of these technologies only ASTiD™ and Optalert™ can be considered as fatigue detection technologies. The HaulCheck product only measures lane position and vehicle proximity; notifying the operator only after they have deviated dangerously out of their lane regardless of their level of fatigue. Alternatively, ASTiD™ and Optalert™ are both designed to detect the early effects of drowsiness, both have gone through field trials with promising results and they are both established on sound scientific research from well respected fatigue laboratories. Therefore, for the purposes of identifying effective and predictive technologies for drowsiness detection, only ASTiD™ and Optalert™ are recommended. Both technologies are viable options for use as supporting technologies to an asset’s overall fatigue management program.

When implementing new technologies, it is easy to forget or ignore the most important aspects of a successful fatigue management program; ensuring that people recognize and take responsibility for their own fitness for work, taking into consideration the frontline supervisors and their understanding and management of their workgroups, and the development of a culture within our businesses that encourages reporting of and action on drowsiness and fatigue risks. To improve the likelihood for success of both the new technology and the accompanying fatigue management programs, it is important to utilize the appropriate change management process. This process ensures that end user contribution is sourced, intervention strategies are agreed on and timely, and that appropriate communications and support are established prior to, during and following the implementation.

When discussing the potential for decreasing the operational risk of 24/7 operations through the implementation of safety-enhancement technologies, it is easy to allow technology to supersede and

overshadow the importance of good people-centric policies. These devices should not be relied on as the panacea for managing fatigue. In fact, these technologies only serve as a last line of protection when all other fatigue management policies and procedures have been put into place.

executive Summary



ii. intrODuctiOn

Within the public sector, driver drowsiness and inattention are significant factors contributing to commercial truck crashes accounting for 1,200 deaths and 76,000 injuries annually at an estimated cost of $12.4 billion to the commercial trucking industry. In the surface mining industry, 93% of haulage truck accidents are due to human error. 60-70% of human error accidents were found to be fatigue-related. Fatigued drivers are often not aware of their condition, frequently driving for 3-30 seconds with their eyes totally closed.

While operator fatigue is predominantly a people management issue, there is available technology that can be adapted to assist in the detection of the onset

of fatigue and interface with the operator to prevent an incident, and subsequently, allow remedial actions to be taken. The on-highway trucking, automotive, and mining industry have looked to technology to provide supplementary solutions to the driver/operator fatigue issue. Numerous technologies have surfaced, but none have been clearly identified as the ideal solution in terms of accuracy or wide spread operator acceptance.

This project contained two major components: 1) An in-depth technology review and 2) A driving simulation study of leading fatigue technologies.

Operator fatigue is one of the most prevalent root causes of earth moving equipment accidents within the mining industry. Sleep deprivation, fatigue and drowsiness decrease awareness, attention, and increase reaction time.



iii. technOlOgy review

A. Objectives

The purpose of this project was to conduct a comprehensive review of all existing and emerging fatigue detection technologies. This is the first time that a wide range of experts from various backgrounds has used the same methodology to quantitatively and objectively assess fatigue management technologies. The end result of this project was a comprehensive objective assessment of the available fatigue/alertness technologies as well as identifying the merits of emerging technologies that may become available in the future. The goals of the technology review project are to:

• Identify and conduct a detailed review of all available fatigue and alertness technologies currently being utilized for detecting driver fatigue worldwide.

• Identify what research is being considered or is in fact underway in regards to detection technology.

• Identify the gaps within available detection technology compared to current research and recommend methods to resolve this variance.

• Provide a detailed report and summary to mining industry companies on the application and effectiveness of identified detection technology systems that may support the mining industry. (May include technology currently being utilized in the aviation, military, commercial trucks and/or motor vehicles industries)

B. Team Members

The technology review project included individuals from a customer, Caterpillar and from the shift work consulting firm, CIRCADIAN™. Detailed team member biographies are included in the appendices.

Team Members:

David Edwards, Caterpillar Inc.Acacia Aguirre, CIRCADIAN™ Bill Davis, CIRCADIAN™ Todd Dawson, CIRCADIAN™ Udo Trutschel, CIRCADIAN™

C. Methods

The methodology was very structured to maintain a high degree of objectivity. Below is the outline of the main tasks for completing the technology review.

1. Identify all available technologies. This was done through web searches, interviews with fatigue industry experts and mining customers, prior team experience and an extensive patent review on alertness/fatigue technologies.

2. Identify most promising technologies. This shorter list of technologies was determined by availability of system, previous or current experience of the technology in the mining industry, potential for use in mining, and current stage of development

3. Gather information on most promising technologies. Whenever possible, users in the mining industry were contacted for input regarding the technology and any outcomes or data that was available. The information was gathered from, but not limited to public domain reports, interviews with users and interviews with technology suppliers.

4. Develop diagnostic Objective Matrix Tool. Experts at CIRCADIAN™ with input and advice from both customer and CAT developed the matrix.

5. Score each fatigue/alertness technology. Using the matrix, each technology was scored by internal CIRCADIAN™ experts.

6. External expert scoring. External experts were used to broaden the arena of expertise to leaders in optics (which are often utilized in fatigue technologies), psychology, ergonomics, medicine, mining and transportation. These external experts also used the matrix for technology scoring.

7. Mining weights. To close the circle of experience, representatives from a customer were asked to identify the most important aspects of a fatigue detection device with regard to the mining industry.

8. Develop composite scores. All of the data gathered was incorporated into a final composite score for each technology. This provides an overall view of the most promising fatigue and alertness technologies.




D. Product list and reviews

IdentIfyIng commercIally avaIlable and emergIng technologIes

Commercially available products and emerging technologies were identified primarily through web searches, literature reviews, interviews with subject matter experts and a patent search on alertness/fatigue technologies. This process resulted in identifying 35 technologies. Each product or technology was then evaluated based on its current availability, history of experience in the mining industry (past, present or future trials), feasibility of implementation within a heavy mining equipment operator station and the technology’s current

stage of development if it was not commercially available. This process narrowed the original 35 technologies down to a much smaller list of 22 products and/or technologies (Table 1). Table 1 shows a complete list of the 22 reviewed technologies as well as their developer and their general technology category. A quick breakdown of the reviewed technologies shows eye feature analysis as the predominant technology type followed by lane deviation systems (Figure 1).

gather InformatIon on most promIsIng technologIes

Once the final 22 technologies were identified, a more thorough investigation was conducted to ensure the

Company

AcuMine

Advanced Safety Concepts

ARRB Transport Research

AssistWare Technologies

Atlas Research Ltd

Attention Technologies

Delphi Corporation

International Mining Technologies

Iteris Inc

MCJ

Mobileye NV

Neurocom

Ospat Pty

Pernix

Precision Control Design Inc

Muirhead/Remote Control Tech.

Security Electronic Systems

Seeing Machines

Sleep Diagnostics

Smart Eye

SMI

Welkin

HaulCheck

PASS

Fatigue Management System

SafeTrac

NOV Alert

Driver Fatigue Monitor

Driver State Monitor

Voice Commander System

Lane Departure

EyeCheck

Vision/Radar Sensor

EDVTCS

OSPAT

ASTID™

SleepWatch

Fatigue Warning System

Sleep Control Helmet System

Facelab

Optalert™

AntiSleep

InSight

Nap Zapper

Lane Deviation

Head Nodding Detection

Mental Reaction Time

Lane Deviation

Muscle Tone Analysis

Eye Blink Detection

Eye Blink Detection


Lane Deviation

Fitness for Duty System

Lane Deviation

Skin Conductance

Fitness for Duty System

Steering/Machine Movement

Activity Monitor



Eye Feature Monitoring

Eye Feature Monitoring

Eye and Head Monitoring

Eye and Head Monitoring


produCt teChnology

table 1: Final teChnology list



evaluators had as much technical information as was publicly available for each of the technologies. Users of these technologies in the mining industry were contacted for input regarding the technology and any outcomes from technology trials if data was available. Additional information was gathered from, but not limited to the following sources: public domain reports and interviews with users and vendors.

Product briefs were developed by the team as a means of quickly educating stakeholders and our external experts on the 22 technologies can be found in the appendices.

E. Assessment Matrix

development of matrIx/feature crIterIa

Through interviews with mining customers and fatigue experts a list of technology features was created including descriptive, technical and functional criteria. A total of 93 features were selected across 16 categories. Features and feature categories were finalized using input from both the fatigue and mining industry.

1.0 focus of technology—This category describes what the technology is monitoring. Recent studies suggest that users strongly prefer systems that require as little personal monitoring and contact with the technology as possible. The preference is for systems that monitor vehicles instead of people. 1.1 Vehicle Monitoring 1.2 Operator Monitoring

2.0 system capabilities—System capabilities are important to identify the spectrum of metrics that can

be measured or are included as part of the technology. 2.1 Accident Mitigation (lane deviation, passing instructor) 2.2 Collision Warning 2.3 Operator Performance Evaluation 2.4 Operator Fatigue Prediction 2.5 Microsleep Detection 2.6 Operator/Dispatch Assistant

3.0 primary sensor technology—This section identifies the sensor(s) primarily used by the system. The sensor may or may not currently be used to track fatigue. However, most of these sensors have been used to track fatigue to some degree in different environments. 3.1 Machine Vision (Digital Video) 3.2 Infrared (IR) camera 3.3 Visible light camera 3.4 IR illumination and sensors 3.5 Equipment / Electrodes attached to Body 3.6 GPS 3.7 Laser Scanning 3.8 Accelerometry 3.9 Motion detection (gyro sensor) 3.10 Timer Switch / button

4.0 primary measures (eye)—This is one of several sections that identify the primary measure of the system. In this case, the primary measure is the eye. All the subcategories are generally accepted measures for the eye that can be linked to alertness and fatigue.


Figure 1: teChnologies reviewed by type

Muscle Tone Skin Conductance

Activity Monitor


Head Nod Lane Deviation/Steering Analysis

Eye Feature Analysis

Fitness for Duty

Num

ber

of P

rodu

cts 7

6

5

4

3

2

1

0



4.1 Gaze tracking 4.2 PERCLOS/AVECLOS 4.3 Blink rate 4.4 Prolonged eye closure 4.5 Eye Features (pupil, eyelids) 4.6 Pupil Reactivity

5.0 primary measures (behavior/physiology)— Primary measure for behavior and physiology including heart rate, body movements and brain activity. 5.1 Head Movement (Head nodding) 5.2 Facial Features (yawning, muscle movements, muscle tone) 5.3 Heart rate 5.4 Electroencephalogram (EEG) 5.5 Electromyography (EMG) 5.6 Electroocculogram (EOG) 5.7 Grip force (Steering Wheel) 5.8 Skin Resistance 5.9 Body Movement (Posture)

6.0 primary measures (operator performance)—Primary measure based on operator performance. This category includes characteristics about the operator’s driving performance and quality of work that can be measured to indicate fatigue. 6.1 Microsteering corrections 6.2 Variation Steering Angle 6.3 Variation Steering Angular Velocity 6.4 Lane Deviation 6.5 Distance to right/left lane 6.6 Time to Line Crossing 6.7 Operator reaction time to artificial stimulus (mental reaction time) 6.8 Position relative to Objects (GPS, radar, laser)

7.0 primary system characteristics—These are capabilities of the system for detecting fatigue and alertness or characteristics that indicate fatigue. 7.1 Sensor Fusion (using multiple sensors and hardware)

7.2 Use of Alertness Models in Algorithm (CIRCADIAN™ model, sleep model) 7.3 Ability to detect operator state in real time 7.4 Automated feature extraction (Identifying portions of data for immediate analysis and storage, e.g. capturing the analysis of hundreds of images rather than each individual image to speed up the decision making process) 7.5 Ability to store and retrieve data wirelessly (wireless communication with device to/from dispatch) 7.6 Ability to store and retrieve data locally (device stores data locally on the truck)

8.0 system integration requirements 8.1 Ease of installation in vehicle 8.2 Takes status of vehicle into account (transmission, speed, state, etc.) 8.3 Permanent integration ability (how easy it would be to integrate the system into the dashboard and/or with other machine systems (radio dispatch, GPS, Machine health monitor, etc.)) 8.4 Installation and integration.

9.0 fatigue countermeasure 9.1 Countermeasure (is there any kind of countermeasure that is initiated when a certain condition is detected) 9.2 Multiple Countermeasures (does the system use multiple countermeasures like lights, audible alarms, seat vibrations, vehicle interventions, etc) 9.3 Adaptive Countermeasures (does the countermeasure change in frequency, intensity, etc. when certain conditions occur, e.g. alarm frequency increases or decreases depending on predicted fatigue level or change in reaction to stimulus) 9.4 Online Feedback about Alertness Level (i.e. the operator is given information about his current fatigue state) 9.5 External alarm (Alarms surrounding operators about operator state)




9.6 Dispatcher Interaction (does the system communicate with both the operator and dispatcher, e.g. notifies dispatcher about operator state) 9.7 Identifying fatigue countermeasures that are used in the system.

10.0 environmental requirements for technology—Are there any environmental conditions that might interfere with the system? 10.1 Dust 10.2 Vibration 10.3 Weather conditions 10.4 Lighting 10.5 Road Conditions (rough bumpy roads, muddy/slick roads, etc.)

11.0 data evaluation, recording, reporting methods—Data analysis methods used in signal analysis. 11.1 Conventional Statistics (threshold) 11.2 Intelligent Adaptive Data Analysis Methods (fuzzy logic, neural nets) 11.3 System Reporting (the system provides an easy to use interface that provides meaningful reports)

12.0 validation and system accuracy—Has the system been tested in different environments to ensure accuracy? 12.1 Objective Validation (the system has been validated using scientifically accepted objective measures) 12.2 Subjective Validation (the system has been validated using subjective measures such as opinion surveys, sleepiness scales, etc) 12.3 Validation in the Laboratory 12.4 Validation in the Field (On-highway) 12.5 Validation in the Field (Mining Environment) 12.6 Avoids False Positives (How often does the system trigger an alarm that was incorrect?) 12.7 Avoids False Negatives (How often does the system not trigger an alarm when one is required?)

13.0 technologies integration ability—Integration with the mining industry as well as other fatigue detection systems. 13.1 Ease of integration of other measures into the data analysis (e.g. if a system is collecting eye feature data, how easily could it also assess eye gaze?) 13.2 Ease of Integration with other alertness/fatigue products (How easy would it be to integrate this system with other systems to complement it?) 13.3 Long-term future of the system (the device is funded well and has great support from investors or have a long term life) 13.4 Compatibility with future safety technologies (is system capable of integration with other safety technologies?)

14.0 operator acceptance—Issues around interface with the operator/user and whether or not the technology will likely be accepted. 14.1 General User Acceptability (How well does the user accept the system?) 14.2 Mining User Acceptability (How well does the mining user accept the system?) 14.3 Union Acceptance of technology (field trial reports) 14.4 Robustness to Operator Manipulation (how easily can the operator manipulate the system? E.g. turn it off, avoid detection, etc) 14.5 Robustness to Operator destruction (how easily can the operator physically damage the system to make it inoperative?) 14.6 Robustness to individual differences (The system handles individual differences easily and with little time requirement) 14.7 Not mentally invasive (does not require additional work for the operator e.g. no response required?) 14.8 Not physically invasive (does not require contact with the operator) 14.9 Easy integration into Mining Culture




15.0 system data integration, calibration,

maintenance and infrastructure costs - What additional effort is required to implement the system and maintain it? 15.1 Integration Ability to Mining Operations 15.2 (The system tracks data that is objective and can be correlated to issues that mine managers are concerned with) 15.3 Ease of System Calibration (Is it easy to calibrate for individual operators or for each truck) 15.4 Ease of System Maintenance (how much maintenance is required for general upkeep) 15.5 Does the System avoid additional expensive Infrastructure (GPS, lane markers, radio frequency identification, etc.)? 15.6 Ratio Accident Reduction/System Costs (Cost efficiency is defined as the ratio between ability of the system to reduce fatigue related incidents costs and the overall cost of the system and its maintenance.)

16.0 technology readiness—Is the system commercially available and timeframe for optimal usefulness. 16.1 Is the product commercially available and used? 16.2 Has the product potential to be used in the short-term (6 mos-1 yr) in mining operation? 16.3 Has the product potential to be used in the middle-term (1-4 years) in mining operation with some modifications? 16.4 Has the product potential long-term (4-6 years) use in mining operation when directly integrated in the mining truck?

development of weIghtIng and scorIng system

A weighting system was developed to allow users of the matrix to rate each feature and feature category on a 0-10 level of importance of that feature or category. Each category’s weightings were then normalized across all categories to ensure categories were equally represented regardless of the number of features in each category. Scoring was based on a 6-point scale (none, potential, possible, low, medium, high) to allow a

score of each technology feature based on the degree to which that feature was applicable to the technology or the likelihood that a feature could be incorporated into the technology through additional development. Numerical values were assigned to each score (0.00, 0.10, 0.20, 0.25, 0.75, 1.00, respectively) for purposes of calculating the overall scores for each technology.

scorIng and weIghtIng of each fatIgue/alertness technology

To ensure that the scores generated for these technologies were as objective as possible, ratings for technologies were conducted both by the authors as well as several experts from throughout the fatigue research community from disciplines including optics, occupational medicine, human factors, mining, and transportation research. Experts had no known conflicts of interest with any of the technologies included in the matrix. In addition to providing actual product ratings, the experts were given the opportunity to provide their weightings to the features and feature categories, as they deemed appropriate. In addition to the weightings provided by the fatigue experts, input from a major global mining company was provided to produce a set of weightings for the features and feature categories based on mining specific applications. Each expert was also given the opportunity to subjectively score each technology on a simple 1-10 scale, with 10 being the highest rating. Table 2 shows the category weights for the team, all experts, and the mining representatives.




Of interest were the distinct differences in the category and feature weights given by the fatigue experts compared to the weights provide by the mining industry representatives. Average weights from fatigue experts remained fairly consistent between 4 and 8. Mining weights differed by in large showing more importance on system capabilities,

operator acceptance and technology readiness. Lowest scoring categories were: Focus of Technology and Primary Measures (Eye, Behavior, and Operator Performance). This difference is a reflection of the mining representatives’ not placing much value on how the technology detects fatigue, but more so on the system’s operator acceptance and availability.


table 2: summary oF all expert and mining weights

Fmt produCt evaluation

Focus of Technology 8 5 3 5 1 10 5.3 0

System Capabilities 10 10 6 7 1 10 7.3 10

Primary Sensor Technology 6 6 4 5 1 5 4.5 3

Primary Measures (Eye) 0 8 8 7 1 8 5.3 1

Primary Measures 0 8 6 2 1 10 4.5 1

(Behavior/Physiology)

Primary Measures 0 6 4 6 1 10 4.5 2

(Operator Performance)

Primary System 8 7 8 6 1 7 6.2 6

Characteristics

System Integration 6 3 4 3 1 7 4.0 6

Requirements

Fatigue Countermeasure 10 10 8 7 1 10 7.7 5

Environmental 7 8 8 10 1 10 7.3 8

Data Evaluation, 8 4 8 5 1 10 6.0 4

Recording, Reporting

System Accuracy 8 8 8 10 1 10 7.5 4

Technologies Integration 8 5 6 5 1 7 5.3 4

Operator Acceptance 7 10 5 10 1 10 7.2 10

Data Int., Cal., Maint. & 7 3 4 6 1 10 5.2 4

Infra. Costs

Readiness 9 4 6 5 1 7 5.3 9

team weights

expert 1 expert 2 expert 3 expert 4 expert 5average expert

weights

mining weights



F. Matrix Results

The completed matrix included final composite scores for each technology along with subjective technology ratings (Figure 3 and 4). Product composite scores were influenced by the differing weights applied by fatigue and mining experts. To simplify the review of the scores technologies were divided into three tiers. Using the fatigue expert weights, top tiered products consisted of FaceLab, ASTiD™, Optalert™, HaulCheck, Delphi’s Driver State Monitor and SmartEye. Of the top six scoring products, four are eye feature detection systems and two (ASTiD™ and Haulcheck) are vehicle-monitoring

systems. The second tier consists of 12 products. Of these 12, two are eye feature detection systems, four are physiology/behavioral devices (DVTCS, SleepWatch, NovAlert, and PASS) and the three are mental reaction time tests (Voice Commander, ARRB, Muirhead/RCT) and three are vehicle-monitoring systems (SafeTrac, MobileEye, and AutoVue). The bottom tier of products consists of two pre-shift fitness-for-duty tests (Ospat and Eyecheck) and two head worn head nod sensors (Sleep Helmet and NapZapper).

When mining weights are applied the top tier of 6 technologies remain unchanged, however the ordering


Figure 2: Fatigue expert weights versus mining weights

Focu

s of T

echn

ology

Syste

m C

apab

ilities

Primary

Sen

sor T

echn

ology

Primary

Mea

sure

s (Eye

)

Primary

Mea

sure

s (Beh

avior

/Phy

siolog

y)

Primary

Mea

sure

s (Ope

rator

Per

form

ance

)

Primary

Sys

tem

Cha

racte

ristic

s

Fatig

ue C

ount

erm

easu

re

Enviro

nmen

tal

Data E

valua

tion,

Recor

ding,

Repor

ting

Syste

m A

ccur

acy

Tech

nolog

ies In

tegr

ation

Operat

or A

ccep

tance

Data In

t., C

al., M

aint.

and I

nfra.

Cos

ts

Readin

ess

Syste

m In

tegr

ation

Req

uirem

ents

10.0

9.0

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.0

0.0

Wei

ghts

Average Expert Weights

Mining Weights



differs slightly. The second tier sees significantly more shifting of technologies and finds 2 technologies dropping into the third tier (both lane deviation systems) and 2 tier three technologies climbing into tier two (both fitness-for-duty tests). These findings are understandable as the lane deviation technologies relied heavily on image recognition of painted roadway markings, which by definition are non-existent in off-road environments. Further support for the technology shift is apparent in that fitness-for-duty tests increased in value under the mining weights. Mines typically have controlled access, which could allow for pre-shift testing; something far less practical in more general applications like automotive driving. The head-worn, head-nod sensor scores remained in the bottom tier regardless of which system of weights was applied.

Subjective scoring of the technologies demonstrates a clear trend for the expert reviewers towards eye feature detection systems for both the general and mining industry weights. Due to the popularity and preponderance of research funding on methods for automatically tracking percent eye closure (PERCLOS) these subjective scores are not surprising. Further evaluation of the general fatigue subjective scores show that next most highly rated technologies were those that monitored lane deviation utilizing image recognition. The lowest general fatigue subjective scores were for the mental reaction time and head nod sensing technologies. In contrast, when considering mining industry requirements, following the automatic PERCLOS detection systems, the same expert reviewers subjectively scored the two lane deviation/steering deviation devices that do not depend on image recognition (ASTiD™ and Haulcheck) followed by mental reaction time technologies and head nod sensors. Image recognition lane deviation technologies, based on mining industry requirements, were subjectively scored the lowest.

The highest ranking products displayed the following characteristics: (1) Multiple sensors or ability to process multiple features; (2) Multiple means of alerting the operator of impending fatigue and signaling the

supervisors and/or dispatchers; (3) Previous validation tests in the field, particularly in rough environments; (4) The capability to be customized to the individual; and (5) Required little or no operator input. With regard to the features of user/operator acceptance, devices that focused primarily on the vehicle tended to score higher. Some of the technologies that did not score in the upper rankings were nonetheless promising for the long term. These included products like NOVAlert and SleepWatch. The ideal solution to managing fatigue in the mining industry will likely be comprised of several different technologies working together. These will likely but not necessarily include pre-shift assessments such as Ospat or EyeCheck, vehicle-monitoring systems like HaulCheck or ASTiD™, and operator monitoring systems like Optalert™.

The usage of the scoring matrix is critical to providing an unbiased and equitable evaluation of all the technologies. Comparing the subjective and objective scores using the two weighting systems emphasizes the importance of such a matrix. Were these experts to merely provide their professional opinion on which technologies they were to recommend, their response could be largely biased on the knowledge of their own particular industry and field of study. However, when industries other than the on-highway transportation industry seek advice from fatigue experts or the scientific literature, not having a way to account for the experts’ inherent bias towards their industry’s particular needs could lead to drastically different and potentially inappropriate recommendations. This matrix provides each industry a way to leverage the knowledge of the fatigue research community by tailoring it to each industry’s specific needs through the use of the matrix weighting system.

The technology review project is significant in that it brought together a wide range of experts from various backgrounds and used the same methodology to objectively and subjectively assess several commercially available and emerging fatigue management technologies. The end result of this collaboration and methodology was not only an objective assessment of the currently available technologies, but it also assessed the merits of emerging technologies that may become available in the near future.





FaceLab

ASTiD

Optalert

Delphi

SmartEye

SMI

DVTCS

SleepWatch

CoPilot

Voice Comm.

ARRB

PASS

NovAlert

SafeTrac

MobileEye

Autovue

Muirhead

Ospat

EyeCheck

Sleep Helmet

Nap Zapper

Figure 3: produCt and teChnology ratings with Fatigue experts’ weights

1

General Objective

Tier 1

Tier 2

Tier 3

General Subjective

0.90.5 0.70.3 0.80.4 0.60.20.10

ASTiD

FaceLab

HaulCheck

Optalert

Delphi

SmartEye

DVTCS

PASS

SleepWatch

Voice Comm.

ARRB

SMI

CoPilot

NovAlert

Ospat

Muirhead

EyeCheck

SafeTrac

Autovue

Sleep Helmet

MobileEye

Nap Zapper

Figure 4: produCt and teChnology ratings with mining weights

1

General Objective

Tier 1

Tier 2

Tier 3

General Subjective

0.90.5 0.70.3 0.80.4 0.60.20.10




G. Gap-Analysis

The best in class technologies were selected from the matrix scores using a Pareto plot to identify the top 25% technologies (Figure 5). Based on this information the team recommended a strategy for addressing the gaps in both the available and emerging technologies and what if any action should be taken to help move the best in class technologies forward. Technologies identified as needing additional investigation included: FaceLab (Seeing Machines), HaulCheck (Accumine), Optalert™ (Sleep Diagnostics) and the Driver State Monitor (Delphi). Additional technologies were included in the gap analysis discussion because of their extremely low scores on the matrix and the fact that they are currently in use in the mining industry (Sleep Helmet and NapZapper). The EDVTCS (Neurocom) system also was included as a technology worth further investigation simply because it used a less intrusive sensor (wrist watch) that might make for an easier mining implementation were it found to be effective.

ASTiD

Promising Predictive Technology

Face

Lab

Haulch

eck

Optale

rt

Delphi

DSM

Smart

Eye

EDVTCS

PASSARRB

SMI

Copilo

t

Novale

rt

OSPAT

Muir

head

Eyech

eck

SafeTr

ac

Auto-V

ue

Sleep H

elmet

Mob

ileEye

Sleepw

atch

Napp Z

appe

r

Voice C

omm

ande

r

Figure 5: pareto plot oF teChnologies (top 25% of technologies to left of orange line)

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

No Further Activities RequiredAdditional Validation Required




Of the technologies identified as “promising” or “requiring additional validation”, the technologies were plotted on a time to delivery axis by their matrix score (Figure 6). This activity demonstrates graphically what the relative value would be when additional time and money is invested to bring future promising technologies to market versus those that were already immediately available. A summary of each of these products’ current state, long-term possibility, development time and estimated cost can be seen in Table 3 and Table 4 for reactive and predictive technologies. To conclude the gap-analysis activities, a list of action items was created to provide any additional information or justification to enable the team to formulate its final recommendation.

One action item was clearly identified with needing additional effort. The two head nod sensors included in the evaluation scored extremely low by the fatigue industry experts. Of concern to the team was that these products have in the past been used at some mine sites and could potentially be used by sites in the future. Due to the fact that these devices were being used in the field and their significantly low scores, it was determined that the team should conduct a validation study of these products to determine if the lower scores are justified. This validation study was added to the original project scope. The details and results of this study are discussed in the following section.

Now

ASTiDHaulcheck

OptalertDelphi

EDVTCS

Sleep Helmet

Nap Zapper

Facelab

Time to Product Availability

3yrs+1.5-3yrs

Figure 6: matrix sCore versus time to availability

table 3: available reaCtive teChnologies

0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

system

Haulcheck

Nap Zapper & Sleep Helmet

Available in mining industry


Effective collision prevention through “berm sensing”

Incremental improvements

Now

Now

$50,000 AU per truck. (High on-going maintenance costs)

$10/$1000 per truck and/or operator

Current state long term possibility time Cost




table 4: promising prediCtive teChnologies

system

ASTiD™

Optalert™

Facelab

Delphi

EDVTCS


Available

Favorable results in lab setting

Favorable results in lab setting

Favorable results in railroad and trucking

Incremental improvements

Accurate JDS readings in mine setting

Readings analogous to EEG from wristband

Accurate PERCLOS readings & alarming from camera

Accurate PERCLOS readings & alarming from camera

Now

Now

3-5 yrs

1/2-3 yrs

1/2-3 yrs

$6,000 AU per truck. (High on-going maintenance costs)

$16,000 AU per unit -includes 1 pair of glasses (Volume pricing possible)

Unknown

$500-$3000 per OHT (based on volume)

$8,000 per OHT

Current state long term possibility time system Cost



iv. Driving SimulatiOn StuDy

A. Objectives

This study represents the continuation of the technology review study conducted in 2006, which evaluated Fatigue Management Technologies (FMT) using a comprehensive matrix. The objective of this study is to perform an in-depth evaluation of the performance of head nod sensors compared with the tier 1 technologies identified with the matrix. The selected FMT’s were evaluated using a driving simulator in addition to a host of physiological, behavioral and performance measurements during an overnight driving protocol.

B. Team Members

David Edwards, Caterpillar Acacia Aguirre, CIRCADIAN™ Bill Sirois, CIRCADIAN™.Udo Trutschel, CIRCADIAN™Dave Sommer, University of Applied Science, SchmalkaldenMartin Golz, University of Applied Science, Schmalkalden

C. Method

The study was conducted in the Department of Adaptive Biosignal Analysis at the University of Applied Sciences, Schmalkalden. The department has a sophisticated driving simulator and ample experience conducting this type of study. Figure 7 shows the layout of the lab, with driving simulator and the observation and data collection areas. The lab is fully controlled by specialized software, allowing regulation of light levels, car environment (temperature, noise level, humidity), and of controls and instruments. The communication between volunteers and experimenters is conducted by interphone. The driving simulator includes an Opel Corsa cab to provide a realistic driving experience. The driving scenarios software is also very realistic. Figure 8 shows an example of the driving simulator driving scenario. Drives were conducted on a rural road with no other traffic on the roads. The monotony of the driving task ensured that drivers would become fatigued throughout the driving session.

Figure 7: lab layout; (1) landscape generation, (2) video Capture,(3) driver state sensor,(4) experimental Control,(5) video Capture,(6) electrophysiological recording, (7) Car hardware Control,(8) eye-tracking recording,(9) video projector,(10) digital video Cameras, (11) hd video eye Cam, (12) projection screen, (13) Car opel Corsa

Figure 8: simulator driving environment: top down (left) and screen image (right)




measurements

Data collected for this study included the following:

• Driving parameters: lane deviation, steering angle, driving errors (defined as all wheels out of the lane)

• Driver behavior: video recordings of driver’s eyes, face and head position

• Physiological parameters: EEG (9 channels), EOG (2 channels), EMG, ECG

• Driver subjective alertness: Karolinska Sleepiness Scale (KSS), Samn-Perelli, alertness and micro sleeps self assessment

• Performance tests: Continuous Tracking Tasks (CTT) and PVT

The following FMT suppliers made their products available for testing: Seeing Machines, Delphi, SMI and NapZapper. SeeingMachine’s DSSR records the following parameters: head position, head rotation, and eye closure (PERCLOS). Delphi Driver State Monitor (DSM) records head movement, distraction, eye closure (AVECLOS). The system also provides eye closure and eye closure duration warnings. SMI Insight Eye Tracker records head movement, point of gaze, eye closure (PERCLOS 70, PERCLOS 80). NapZapper records head nodding. The following figures show the devices and data collection screens of Seeing Machines and Delphi (Figure 9 and 10).

volunteers

Volunteers were recruited among Schmalkalden University students. Students were informed about the study by information posted in the University website. All volunteers were interviewed before the experimental night. They were informed about the study protocol and signed an informed consent. Sixteen volunteers participated in the study, ten men and six women. The average age was 22 (range 18 – 31). They were all healthy and had regular sleep/wake schedules.

protocol

The study consisted of two overnight driving simulation sessions. Before the experimental nights, volunteers were trained in the different

tests, and wore an activity monitor and completed a sleep/wake log for at least 24 hours prior to the experiment. On both nights, volunteers arrived at the lab at 10 PM. After wire-up, checking logs and activity monitors, and retraining, the experimental sessions started at roughly 11:30 PM. There were eight experimental sessions, each one lasting one-hour with the last session being finished at 8:30 AM. Volunteers had a 1-h break at 3:30 AM. Each session included: 40-minutes driving session, 10-minutes CCT performance test, and 10-minutes PVT. Alertness self-assessment and Samn-Perelli questionnaires were performed at the end of the driving task. KSS and brief alertness assessments were performed at regular intervals during the driving task, as well as before and after the task. To minimize distractions during the driving task, the experimenter would ask volunteers about their alertness using both KSS and the self-assessment questionnaire. Figure 11 shows the experimental protocol.

Volunteers were asked to complete two experimental nights, several weeks apart, so that they would fully

Figure 9: seeing maChines (dssr)

Figure 10: delphi (dsm)




recover from sleep deprivation. The first night, FMT were used without alarms, and the second night with alarms. To study included twelve volunteers completing both experimental nights, and six others completed only the first experimental night.

D. Results

In this report, the following analyses are presented:

• Fatigue assessment using Delphi, Seeing Machines and NapZapper

• Correlation between Delphi and Seeing Machines fatigue assessment and subjective alertness (KSS) and driving performance

• Volunteers evaluation of Delphi, Seeing Machines and NapZapper (operator acceptance)

alertness, fatIgue and drIvIng performance progressIon throughout the nIght

As expected, sleepiness as measured by the KSS, DSM, DSSR increased, and driving performance deteriorated progressively throughout the night. Driving performance and KSS showed a strong correlation (91%) overall throughout the entire night, meaning that as the driver became more fatigued the number of driving errors increased throughout the night.

10:00pm

1:30am 1:40am

(Re)calibration Self-Assessment(Questionnaire)

Samn Perelli(Questionnaire)

CTT: Continuous Tracking TaskPVT: Psychomotoric Vigilance Test

KSS+Awareness(Questionnaire)

1:50am 2:00am 2:10am 2:20am 2:30am

Preparation 1st Session 2nd Session 3rd Session 4th Session 5th Session 6th Session 7th Session 8th SessionBreak

CTT PVT Break

11:30pm 0:30am 1:30am 2:30am 3:30am 4:30am 5:30am 6:30am 7:30am 8:30am

Driving in the Simulator

Figure 11: experimental protoCol

Figure 12: average session values during experimental night (leFt: seeing maChines[dssr] right: delphi[dsm])

Session Session

Sca

led

Par

amet

ers

Sca

led

Par

amet

ers

Mean Driving Errors (yellow), Mean KSS (black) and DSSR PERCLOS (gray) Mean Driving Errors (yellow), Mean KSS (black) and DSM AVERCLOS (gray)

Drive DriveKSS KSSDSSR DSM

1 12 23 34 45 56 67 78 8

0.25

0.2

0.15

0.1

0.05

0

0.25

0.2

0.15

0.1

0.05

0



36%58% 11% 66% 48% 73% 47% 74%


fatIgue assessment usIng delphI and seeIng machInes

Since subjective alertness (KSS) and driving performance show a well-known pattern of increased deterioration throughout the night, these values were used as benchmarks to evaluate Delphi (DSM) and Seeing Machines (DSSR) fatigue measurements. Correlations between subjective alertness (KSS) and driving performance and Delphi (DSM) and Seeing Machines (DSSR) as an overall correlation across all volunteers and all sessions.

Individual correlations were calculated for each system, (DSSR and DSM) and each driving session throughout

the night. The overall correlation for all sessions are shown in Table 5. Both Delphi (DSM) and Seeing Machines (DSSR) showed strong correlations with (KSS) subjective alertness (90% and 89% respectively). The correlation with driving performance (Table 5) was also extremely high for the DSM (98%) and the DSSR (84%).

Time

TimeFigure 14: session by session Correlations For seeing maChines (dssr) and driving perFormanCe

12pm

12pm

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0.4

0.3

0.2

0.1

0

1am

1am

2am

2am

3am

3am

4am

4am

5am

5am

6am

6am

7am

7am

8am

8am

Sca

led

Par

amet

ers

Sca

led

Par

amet

ers

mean Kss (green) and mean dssr perClos (red) for 14 subjects; numbers show Correlations

mean driving errors (blue) and mean dssr perClos (red) for 14 subjects; numbers show Correlations

sess 8sess 1 sess 2 sess 3 sess 4 sess 5 sess 6 sess 7


Kss (subjeCtive Fatigue)

DSM

DSSR

90%

89%

98%

84%

driving perFormanCe

table 5: overall Correlations

Figure 13: session by session Correlations For seeing maChines (dssr) and subjeCtive alertness




Both technologies appear to do a good job of tracking the progression of fatigue throughout the night according to individuals’ subjective assessment (KSS) and their driving performance.

A more detailed analysis evaluated the systems’ performance session by session (all volunteers averaged). Figures 13 and 14 show the results and correlations between Seeing Machines (DSSR) and subjective alertness and driving performance, and Figure 15 and 16 show the same plots for Delphi (DSM) respectively.

The two systems performed similarly when looking at the subjective alertness and driving performance session by session. In general it could be stated about both technologies that in the early driving sessions (up until 3 AM) KSS increased dramatically but the fatigue measures from DSSR and DSM showed very little change overall. Similarly, the number of driving errors remained relatively flat during the same time period, not beginning to increase until after 3:00 AM. Therefore subjectively, volunteers were “feeling” tired during the early sessions, but their body wasn’t showing the fatigue related increases in percent eye closure nor was their driving behavior getting much worse.

Time

Time

Figure 15: session by session Correlations between delphi (dsm) and subjeCtive alertness

Figure 16: session by session Correlations between delphi (dsm) and driving perFormanCe

12pm

12pm

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0.4

0.3

0.2

0.1

0

1am

1am

2am

2am

3am

3am

4am

4am

5am

5am

6am

6am

7am

7am

8am

8am

Sca

led

Par

amet

ers

Sca

led

Par

amet

ers

mean Kss (green) and mean dsm aveClos (red) for 15 subjects; numbers show Correlations

mean driving errors (blue) and mean dsm aveClos (red) for 15 subjects; numbers show Correlations






It was not until the fourth driving session (3:00 - 4:00 AM) that the fatigue measures started to elevate and driving performance started to decline. Driving performance values did rise and fall from minute to minute, but in general, throughout the early morning hours driving errors were frequent. Likewise, similar to driving performance, the fatigue metric was highly variable from moment to moment, but there was an overall upward trend with the fatigue values remaining elevated throughout the early morning driving sessions. Throughout all the early morning sessions

it is clear that both the DSSR and DSM were correlating well with driving performance. As the fatigue monitoring devices were showing high levels of fatigue, the volunteers were experiencing high numbers of driving errors.

Even though the overall correlations shown above were strong, if you look at the data on a person-by-person basis there are some interesting differences. Figure 17 and 18 show excellent data from a single individual during a single session where the fatigue device correlated very well with driving performance. Lane deviation in the driving simulator

Minutes

Figure 17: example oF delphi dsm perFormanCe with numerous driving errors

0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

05 10 15 20 25 30 35 40

Sca

led

Par

amet

ers

threshhold DSM

Minutes

Figure 18: example oF delphi dsm perFormanCe with no driving errors

0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

05 10 15 20 25 30 35 40

Sca

led

Par

amet

ers




is shown as the dark blue line with an average lane position in light blue. Significant driving errors are marked as yellow vertical lines. DSM and DSSR output are shown in red with an average output in pink. The volunteer commits many driving errors and fatigue is detected (Figure 17). A volunteer commits no driving errors and no fatigue is detected (Figure 18).

However, not all individuals’ data were this clear. Figures 19 and 20 show examples of individual volunteers and

sessions where the FMT did not work well, that is, the volunteer committed errors and fatigue was not detected, or the driver committed no errors and the FMT detected fatigue (Figure 19 and 20 respectively). The present study was not able to determine the direct cause of these broad differences between volunteers, however the reasons behind these major differences will be critically important for technology developers to understand to ensure that their systems are equally effective for all individuals.

Minutes

Sca

led

Par

amet

ers

Figure 19: example oF seeing maChines dssr perFormanCe with many driving errors

0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

05 10 15 20 25 30 35 40

Minutes

Sca

led

Par

amet

ers

Figure 20: example oF seeing maChines dssr perFormanCe with no driving errors

0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

05 10 15 20 25 30 35 40

threshhold DSM




In summary, both Seeing Machines (DSSR) and Delphi (DSM) show, overall, strong correlations with subjective alertness and driving performance. However, correlation level varies by subject and session (time of day). With both systems, there are cases when the volunteer commits errors and fatigue is not detected, or the volunteer commits no errors and the FMT detects fatigue. This finding supports the accepted belief in the fatigue research community that there are individual differences in the population with some individual’s performance being very tolerant to the CIRCADIAN™ cycle and the physiological effects of fatigue, while others are extremely susceptible to both. There is room, however, for fatigue algorithms to be improved for both systems so that they are less affected by individual differences.

napZapper and drIvIng performance

That NapZapper is a small device worn over the ear that triggers an alarm when head nodding is detected (Figure 21). It has been sold to numerous industries as an accident prevention device. The NapZapper seldom was activated during our driving simulation sessions. Below is an example session from one individual who experienced numerous alarms. In this example you see that the majority of the driving error events show that the NapZapper was not activated until after the car was already completely outside of the lane after the driving error occurred (Figure 22).

The Nap Zapper performed poorly overall. In Figure 23 we see the number or correct and false alarms. Correct alarms are those instances where the NapZapper alarm went off and a driving error was committed within the next few seconds. False alarms were those were the alarm went off but no driving errors occurred. For all drivers combined, there were 1633 out of road errors. According to the data, the NapZapper only alarmed 50 times for all subject for all events.

Figure 21: napZapper deviCe.

Figure 22: individual example oF napZapper driving error deteCtion




16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

00 42 6 91 5 83 7 10

Number of Alarms

Figure 23: napZapper CorreCt and False alarms by volunteer

Figure 24: total number oF driving errors per subjeCt

Sub

ject

15 Correct Alarms35 False Alarms

16-2 16-1

15-1

14-2 14-1

13-2 13-1

12-1

11-2 11-1

10-2 10-1

9-2 9-1

8-2 8-1

7-2 7-1

6-1

5-2 5-1

4-2 4-1

3-2 3-1

2-2 2-1

1-2 1-1

Nig

ht o

f S

ubje

ct

Number of 4 Weels Out

0 50 100 150 200 250

Sess1 Sess2 Sess3 Sess4 Sess5 Sess6 Sess7 Sess8



Of those 50 alarms, only 15 were authentic and associated with a subsequent driving error and 35 were considered false alarms. This leaves 1618 driving error events that were completely missed by the device, less than a 1% success rate

For example, looking at the number of driving errors by subject (Figure 24) subject number 16 had over 200 driving errors never triggering a single correct alarm from the device. Other subjects showed this as well, but to a lesser extent. The individuals that committed the most number of driving errors (subject 12) with approximately 275 out-of-road errors had the most number of head nods (8 correct and 3 incorrect alarms). The accuracy for this individual was around 2%. The only other individuals who experienced numerous alarms (subjects 3, 5, 11, and 13) all had more than twice the number of false alarms as correct alarms. Using head-nod as measured by the NapZapper to predict and alarm for driving errors is not effective based on the evidence provided here. The correlation between alarms and accidents for the NapZapper was less than 1% compared to similar overall correlations for the DSSR and the DSM, which were above 80% correlated with driving performance.




A. Short-term

Short-term recommendations are based on looking at only technologies that are currently commercially available for use. Three of the top 5 rated technologies in the technology assessment matrix are currently available: ASTiD™ (Pernix), Haulcheck (Accumine) and Optalert™ (Sleep Diagnostics). All of these technologies have been trialed at mine sites. There are two fundamental differences between these three technologies.

Haulcheck merely looks at the trucks lane position and alarms solely when that deviation passes a set threshold that is considered dangerous. This system should help prevent accidents so long as the operator reacts quickly and appropriately to the alarm. Unfortunately, the operators’ state at the time of the alarm cannot be known, hence their reaction to the alarm may not be timely enough to prevent accidents with other vehicles, machines, or infrastructure. As a last line of defense this product will assist in reducing the number of collisions at a mine site, however, in Haulcheck’s current design, it is unable to identify behavioral or physiological signs of drowsiness at an early stage.

ASTiD™ has gone through very detailed on-site testing accumulating almost 1000 hours of field usage. Results from this trial demonstrated that when operators are willing and cooperative the system is capable of providing excellent feedback throughout a shift (both day and night) on the operators’ state of drowsiness. With ASTiD’s new analysis techniques, drowsiness detection is getting more accurate and predictive. The latest generation of ASTiD™ now includes an integrated dispatch system that provides up to the minute information on all operators in all trucks in the fleet with a simple Red, Yellow or Green rating. This real-time feedback can allow the dispatcher to strategically alter truck assignments or crib breaks to accommodate operators who are showing signs of increased drowsiness compared to the rest of the shift. ASTiD’s major shortfall comes with its need

for operators to enter their sleep quality and quantity for the previous 24-hour period. So long as operators are cooperative this system allows the system to fine tune its algorithm to increase its sensitivity and reduce false alarms. However, if operators do not understand how to correctly enter this information or purposefully enter in incorrect information, then this could result in either increasing or decreasing the sensitivity of the system. Either of which could lead to undesired results. The supplier has suggested that future versions of the device will no longer require this input for accurate detection.

The benefit of this system is that it requires no direct contact with the operator and only evaluates the operation of the truck. Operators tend to feel more comfortable with technologies that focus on the vehicle rather than themselves. With proper training and implementation ASTiD™ could provide a very effective supporting technology to a site’s fatigue management program.

As with ASTiD™, Optalert™ has also undergone field trials. During these field trials excellent information was gathered which allowed for hardware and software changes to be made to accommodate the mining environment. Specifically the system takes transmission and speed information to dynamically activate or deactivate the system based on whether the machine is parked or whether it is in motion. Optalert™ has a very effective algorithm that evaluates the operators’ state of drowsiness on a continuous basis. The nature of this system requires the operator to wear a pair of sensor glasses, which provides the algorithm with the drowsiness measures. The glasses are attractive and can accommodate prescription lenses or tinted lenses. Shortfalls for this system are that each pair of glasses requires professional fitting and adjustment to ensure consistent and accurate data collection. As was found in the field trials, slippage or misalignment of the glasses can negatively affect the systems performance or cause the system not to work entirely. As with the ASTiD™ system, success

v. prOject recOmmenDatiOnS



v. prOject recOmmenDatiOnS

of the system requires operator cooperation. If the glasses are not worn correctly or are not worn at all, the system is compromised. Optalert™ is also developing dispatch software similar to the ASTiD™ that will inform a remote location the drowsiness level of each operator.

The benefit of this system is that is uses an extremely sensitive algorithm that is personalized to each operator. This personalized calibration leads to improved accuracy and better prediction of drowsiness for the user.

For the purposes of immediate implementation, customers only have three real options: Haulcheck, ASTiD™ and Optalert™. Each of the technologies has pros and cons that are not straightforward in terms of identifying a clear leader. The strengths and weaknesses of the technologies differ across systems. The decision as to which technology would provide the best solution for a particular site will require assessing each site’s cultural and organizational makeup. Based on the expert ratings and the field trials ASTiD™ and Optalert™ have consistently performed well and supplier interviews have shown each company is dedicated to producing and providing their customers with a drowsiness detection solution geared specifically towards the mining industries needs. It is the recommendation of this team that Optalert™ and ASTiD™ both can provide an immediate assistance in identifying and warning operators of eminent drowsiness.

B. Intermediate-term

Other companies are developing detection systems that do not require operators to wear glasses, as required by the Optalert™ system. These newer technologies will also not require input from the operator on sleep quality or quantity. These have been the major concerns with the technologies currently available to the market. The industry should encourage and support companies such as Sleep Diagnostics Pty Ltd, Pernix Ltd, Seeing Machines and Delphi in their development of machine-

integrated systems in the hope that an equally effective solution can be developed that doesn’t require the up-front and on-going costs associated with providing individualized glasses, placing PVC pipes along all haul roads or operator input.

C. Long-term

The outlook long-term for fatigue technologies would be to have sensors completely off of the operator using both operator and machine performance variables as part of a combined fatigue/drowsiness detection system. This system could be dynamically linked with the dispatch system and a collision warning system to provide multiple layers of protection for the operator regardless of their level of fatigue. It is recommended that suppliers look at combining the best elements of their respective technologies together. This would provide the quickest and best chance for success in moving the state of fatigue/drowsiness detection technologies forward.



We would like to thank all the technology suppliers for their willingness to share their products with our team. Specifically, Delphi, Pernix Ltd, Sleep Diagnostics Pty Ltd, SMI and Seeing Machines all provided unprecedented access to their engineers and scientists in support of this project.

vi. acknOwleDgementS



David Edwards, Ph.D.

senior engineering speCialist and 6 sigma blaCK belt, Caterpillar

David Edwards is a Sr. Engineering Specialist and a former 6 Sigma Black Belt with Caterpillar Inc. David holds a doctorate degree in Behavioral Neuroscience from the University of Alabama at Birmingham. Dr. Edwards specialized in cognitive ergonomics and transportation safety examining age-related effects on visual function, attention and driving performance. Following his education, Dr. Edwards worked for Hyperion Technologies, one of the world’s leading makers of automobile simulators for research and driver training as a product development engineer developing simulation software for novice driver-training.

Dr. Edwards has been with Caterpillar’s Technology & Solutions Division in the Ergonomics Technology Group for the past 6 years. During this time Dr. Edwards has focused his research on the development of analysis tools for studying operator mental and physical workload. Highlights include implementing portable EEG and eye tracking technology on earth-moving machines to analyze operator mental workload and visual behavior during earth moving applications. Dr. Edwards is now coordinator for safety research and development with Caterpillar’s newly formed Customer Safety Services division.

Dr. Edwards began studying operator fatigue in 2001 working on a joint study with the National Institute of Occupational Safety and Health (NIOSH) to examine the state of fatigue detection technology and it’s potential use in mining operations. This research led to technology trials in U.S. and Indonesian mines.

Dr. Edwards has been an invited speaker on the topics of driver safety, fatigue and collision warning by mining companies, NIOSH and professional research organizations.

appenDix 1: team member biOS



William G. Sirois

William G. Sirois

senior viCe president & Coo, CirCadian™

Bill Sirois is Vice President and Chief Operating Officer for CIRCADIAN™. He is responsible for all CIRCADIAN™ services in North America and Europe, including development of Industrial Shift work Strategies, Alertness Assurance programs, Human Alertness Technologies, Ergonomics programs, Industrial Engineering, Pre-employment Screening, Behavioral Safety Development, and Bio-compatible Shift work Scheduling and support training on Managing a Shift work Lifestyle.

By addressing human limitations and capabilities from a holistic perspective (i.e., operational, physiological, and sociological), Mr. Sirois has demonstrated that a new frontier of opportunity exists for human asset utilization and continuous improvement in overall employee health, safety, and operational performance for all types of business.

Mr. Sirois has also published and lectured extensively as a featured speaker at numerous corporate meetings and international conferences, including the National Association of Manufacturers, The Society of Plastics Engineers, National Ergonomics Conference, The American Petroleum Institute, The American Shipping Club, International Semiconductor Safety Association, Canadian Electric Association, the National Food Processors Association, the National Transportation Safety Board, the Puerto Rico Health and Safety Conference, and the Institute of Mining Health, Safety and Research. Mr. Sirois holds a degree in Chemical Engineering from the University of New Hampshire.

William Davis

viCe president oF operations, CirCadian™

Bill Davis joined CIRCADIAN™ as a former client and now serves as Vice President of Operations for CTI.

He is an industrial safety manager with a broad-based and unique operational background that spans nearly 20 years.

This has included production experience at the facility, divisional and corporate levels at International Paper and other leading pulp & paper companies.

Beginning as a shift worker in the Pennsylvanian steel mills, Bill has held both plant management and corporate safety positions in the paper and specialty board industries. He has extensive experience working with a variety of unions and governmental safety and health regulatory agencies, as well as first-hand experience with high performance & self-directed work environments. His real-world industrial background affords a natural rapport with managers, union representatives and employees at all organizational levels.

Todd A. Dawson, M.S.

direCtor oF researCh, grants & speCial projeCts, CirCadian™

Mr. Dawson graduated from Harvard University with a BA in biological anthropology. While at Harvard, he focused his studies on the biological rhythms of human hormones with a special focus on cortisol. After joining CIRCADIAN™ in 1994, Mr. Dawson was part of several Fatigue Risk Assessments in which he investigated the sources of fatigue and proposed fatigue countermeasures for industries including commuter rail operations, manufacturing, and marine transport.

Mr. Dawson spent nearly three years as a project manager for the Canadian National Rail While on the Canadian National Rail project, his focus was strictly on the train crews. The combination of these two projects has provided him with excellent understanding of the freight rail operation. Mr. Dawson has managed projects at companies including ChevronTexaco, Roadway Express, Tidewater Marine, Hutchison Port Holdings, and GO Transit.

As the Director of Research, Grants and Special Projects at CIRCADIAN™, Mr. Dawson leads CIRCADIAN’s world-class research team in developing assessments and solutions for the wide range of challenges confronting




companies with extended hours operations. Current research focus areas include employee health, employee demographics, and operations management best practices. Mr. Dawson also oversees the development of new technologies, including the Flexible Workforce Management System (FWMS), Freight Rail Crew Optimization System (FRCOS), CIRCADIAN™ Alertness Simulation (CAS), Shift work Adaptation Testing System (SATs), and microsleep prediction systems (including alertness monitoring technologies).

Udo Trutschel, PH.D.

senior researCh sCientist, CirCadian™

Udo Trutschel graduated with a degree in theoretical physics from the Institute for Solid State Physics and Theoretical Optics from Friedrich-Schiller-University (Germany). He received his doctoral degree in applied physics from the Physical Institute, Technical University Ilmenau (Germany). After leaving Germany in 1991, Dr. Trutschel worked for 18 months as research assistant at Tufts University, Boston in the Electro-Optics Technology Center. Afterwards he took a position as visiting Professor at the Electrical Engineering Department, Laval University, Quebec for 3 months.

Dr. Trutschel joined CIRCADIAN™ in 1995. He pursued research on automatic detection of microsleeps / drowsiness from electrophysiological recordings, time series modeling, and the development of algorithms for alertness simulation and prediction, resulting in the development of the CIRCADIAN™ Alertness Software (CAS).

More recently, Dr. Trutschel supervised the development of the Optimization-Simulation System for biocompatible crew scheduling and the Freight Rail Crew Optimization System (FRCOS) software used in railroads. He supervised several research projects focusing on the characterization and detection of microsleeps based on EEG, EOG and eye-parameter measures using supervised and unsupervised neural network techniques.

Dr. Trutschel currently serves as Senior Research Consultant and focuses on the development of software system based on a Flexible Workforce Management Approach (FWMA). Other current activities include the design of Shift work Adaptation Testing-System (SATS), microsleep detection technologies and knowledge-based systems for alertness prediction.

Dr. Trutschel has published his research in over 50 scientific publications and currently holds 8 patents.

Dr. Acacia Aguirre

mediCal direCtor, CirCadian™

Dr. Aguirre has over fifteen years experience in sleep and alertness research, focusing on factors affecting shift workers alertness, safety and health, and the development of fatigue countermeasures. She also has extensive clinical experience in the field of sleep medicine, having practiced as a sleep disorders specialist at one of the major teaching hospitals in Paris, France.

After receiving her MD degree, Dr Aguirre completed her D.M.Sc., which obtained the mention Summa cum Laude. She completed her graduate research work in at the University of Paris VI (France), where she received her PhD in Neuroscience.

Dr. Aguirre’s work at CIRCADIAN™ includes providing training and consulting support on major client engagements, such as fatigue risk assessments, workload analysis, evaluation of employees’ alertness health and safety, scheduling and implementation of fatigue countermeasures. She is also involved in the design of educational materials for shift workers and developed CIRCADIAN’s sleep disorders screening and treatment program.

Dr. Aguirre is actively involved in the scientific community and participates regularly in specialized scientific meetings and symposia. She is member of the European Sleep Research Society, and is also in





the Editorial Board of the Journal of the Spanish Sleep Research Society. She has published over 50 scientific articles and book chapters.

David Sommer

university oF applied sCienCes, sChmalKalden,

germany

David Sommer received his Master’s degree in Computer Science in 1998 from University of Applied Sciences, Schmalkalden, Germany. Since 1998, he has been a scientific co-worker at the Department of Computer Science and an Associate Lecturer in neural networks and pattern recognition.

David has written over 50 publications on neural networks, evolutionary algorithms, nonlinear signal processing, data fusion and pattern recognition in different areas of applications, such as driver fatigue, posturography and sleep physiology.



appenDix 2: expert panel reviewerS

Dr. Larry Barr

u.s. department oF transportation volpe national transportation systems Center advanCed saFety teChnology division

Dr. Lawrence Barr is a senior research engineer in the Advanced Safety Technology Division of the U.S. Department of Transportation’s Volpe National Transportation Systems Center. He has conducted numerous safety-related programs and research studies for the Federal Highway Administration, the National Highway Traffic Safety Administration, the Federal Motor Carrier Safety Administration, and the National Aeronautics and Space Administration. These include conducting motor vehicle crash causation studies as well as a comprehensive benefit-cost study of crash countermeasure systems for all major crash types and vehicle platforms in support of the Intelligent Vehicle Initiative, completing a detailed analysis of naturalistic driving data to develop an understanding of the nature and extent of driver fatigue and driver distraction among truck drivers, providing technical support to the evaluation plans for the Drowsy Driver Warning System and Road Departure Crash Warning System field operational tests, and developing investment analysis and risk models of advanced aviation safety and security technologies for the NASA Aviation Safety and Security Program.

Dr. Barr recently completed a comprehensive survey study on emerging vehicle-based driver drowsiness detection and alertness monitoring technologies for the Federal Motor Carrier Safety Administration. The major objective of the study was to review and discuss many of the activities currently underway to develop unobtrusive, in-vehicle, real-time drowsy driver detection and fatigue monitoring/alerting systems and evaluate them against a set of proposed design guidelines and user interface/acceptance criteria.

Dr. Barr received his bachelor’s degree in mechanical engineering from the University of California at Davis, a master’s degree in aerospace engineering from the Pennsylvania State University, and a doctoral degree in systems engineering from the University of New Hampshire.

Dr. Martin Golz

university oF applied sCienCes, sChmalKalden, germany

Prof. Dr. Martin Golz graduated with a degree in Electrical Engineering from the Institute of Microelectronics from Technical University of Ilmenau (Germany). He received his doctoral degree in applied physics from the Physical Institute, Technical University Ilme-nau. After graduation, Prof. Golz worked for four years as research assistant at Central Research Hospital of Neurology and Psychiatry “Wilhelm Griesinger”, Berlin (Germany) in the Laboratory of Evoked Potentials (1988-1990) and in the Sleep Polygraphy Lab (1990-1992). Here he worked in biosignal analysis as well as in specialized hardware development for performance test technologies.

Afterwards he took a position as a Professor at the Department of Computer Science at the University of Applied Sciences Schmalkalden (Germany). From 1992 to 2003 he was Professor for Physics and Measurement Engineering, and since 2004 he has been holding the full time Professorship for Signal Processing and Neuroinformatics at the same De-partment.

Prof. Golz pursues research on automatic detection of microsleeps and of evaluation of drowsiness from electrophysiological recordings utilizing numerous data fusion algorithms from the field of Soft Computing, especially Fuzzy, Neural, Neuro-Fuzzy and Evolutionary Technology.

He supervised several Master and PhD thesis as well as several research projects focusing on occulography, posturography and on classification of sleep composition. In 2001, he built a driving simulation laboratory to further his fatigue research. Since then, he has conducted more than ten research studies on fatigue, microsleep and driver performance. Recently, Prof. Golz established a gold standard for fatigue prediction and detection based on a database of about 20,000 examples of microsleep events. His presentation on this topic was awarded at the Sensation International Conference “Monitoring Sleep and Sleepiness - From




Physiology to New Sensors” in Basel May 2006.

Prof. Golz is the organizer of the special invited session “Signal Processing Techniques for Knowledge Extraction and Information Fusion” at the 10th International Conference on Knowledge-Based & Intelligent Information & Engineering Systems, Bournemouth U.K. which will be held in October 2006.

Dr. Rich Hanowski

virginia teCh transportation institute, blaCKsburg virginia

Richard Hanowski is the Leader of the Truck and Bus Safety Group at the Virginia Tech Transportation Institute (VTTI). Dr. Hanowski has been conducting transportation safety research since 1992. Previously a Research Scientist at the Battelle Memorial Institute, Dr. Hanowski joined the Safety & Human Factors Engineering Group at VTTI in 1996. In 2003, after completing several successful projects and building a sustainable program in truck and bus safety, a new group at VTTI was formed with Dr. Hanowski as leader.

Dr. Hanowski has formal training in human factors, system design, safety, cognitive psychology, human-computer interaction, training & evaluation, experimental design & methods, and statistics. His experience includes transportation human factors with both light & heavy vehicles, laboratory & field-testing, focus groups, real-time automobile & heavy vehicle simulation, human factors design guideline development, older driver investigation, collision warning, and Intelligent Transportation Systems. Dr. Hanowski specializes in human factors engineering, advanced product design/test/evaluation, and human performance evaluation. He is skilled in all phases of research including conceptual framing, research design, data collection, data synthesis & analysis, assessment of results, and presentation of findings.

Dr. Hanowski is the author of over 70 scientific articles and technical reports. He is an active member of the Intelligent Transportation Society of America, and served as Chairman of the ITS America Safety & Human Factors

Committee (2000-2002). Dr. Hanowski is also active in the Human Factors and Ergonomics Society, and serves as a technical paper reviewer for various transportation-related organizations and journals.

Todd Ruff

national institute oF oCCupational saFety and health, spoKane researCh Center

Todd Ruff obtained a Bachelor of Science in Electrical Engineering from Gonzaga University in 1988 and a Master degree in Electrical Engineering from Gonzaga in 1993. He currently works for the National Institute for Occupational Safety and Health, Spokane Research Laboratory, in Spokane, WA as an electrical engineer and research project manager.

The Spokane Research Laboratory (SRL) serves as the second focal point for mine health and safety research. While research programs touch most mining sectors, the major program focus is on metal and nonmetal mining. To prevent injuries and fatalities in both underground and surface mines, SRL: (1) identifies and classifies risk factors in mining; (2) evaluates recommendations for strategies to prevent injuries and disease through the use of effective control technologies; (3) studies the design of mining equipment to assess the potential risks involved in using it; and, (4) designs, builds, and tests equipment that incorporates innovative control technologies.

Mr. Ruff has been the technical team leader for the development and evaluation of technologies to improve the safety of mining and construction equipment, particularly in the area of collision warning systems and operator fatigue detection. He is registered as a Professional Engineer in Washington State.

Dr. Mario Sandoval

FulCrum engineering, partner and direCtor

Dr. Sandoval has 17 years of experience on mining and work at high altitude. He is a partner and director of Fulcrum Engineering; a company specialized on help




business in the areas of project engineering, human resources (shift work) and strategic planning. One of his main projects in Fulcrum has been as a director of a project to develop software programs to evaluate and design shift work systems for mining operations. Besides holding a MD degree, Dr. Sandoval holds a Master in Environmental Science (University of Chile) and a Master in Ergonomics (Universidad Politécnica, Catalonia, Spain). He also obtained a Diploma on Evaluation and Preparation of Health Projects (University of Chile).

Before joining Fulcrum, Dr. Sandoval has been Director of the Department of Work in Altitude (Workers Hospital, depending of the Chilean Safety Association), Director of the Center of Ergonomics of Work in Altitude (1997-2002), and Director of the R&D Department of the Aerospace Medical Center in the Chilean Air Force (1999-2002). Since 2002, he has been Medical Advisor for the Chilean Safety Association.

In addition to his consulting and research work, Dr. Sandoval also participates as a instructor in the Master on Public Health and Risk Prevention (Institute of Public Health, University of Santiago, Chile) and Master on Science of Exercise (University Andrés Bello, Chile), where he is in charge of the courses on physiology of extreme environments.



appenDix 3: prODuct SummarieS

contact Info/website: http://www.arrb.com.au

primary technology: Stimulus-reaction device.

primary fatigue measure: Decision response reaction time task. The device measures operator’s reaction time to visual and audible stimulus (based on the individual’s baseline).

fatigue countermeasures: Extremely slow reactions trigger an audible warning sound in the machine. In the case of slow responses, an alert is also sent to supervisor, who can then contact the operator.

trialed in the mining application: Yes

timing for readiness: Research units available. No immediate plans for commercial release.

validation for fatigue: Lab and field studies (mining).

Invasiveness/acceptance: Invasiveness: Introduces a secondary task to the operator. Acceptance: At one mine trial, ¾ of participants carried through the whole testing period. At a second mine, there was a mixed response. Some participants chose not to use the device for fear of being monitored for performance, one person found it annoying, other people had no problem using it, one operator stated that it was helpful in maintaining their alertness while working.

environmental complications: Each operator is issued with their own personal ‘touch key’. Each time the operator uses the machine they must touch their key onto the receptacle on top of the reaction box. May allow tampering by entering false data.

cost: $9,000 per machine on a yearly lease (data from 2002). Purchase data unknown. Not presently a commercially available device.

Industries where used: Trials in mining, commercial vehicles used for moving cattle and two-up driving trucks across Australia.

summary: ARRB device measures operator alertness levels through the operator’s reaction to a visual and audible stimulus. It does this in real-time and throughout the whole working shift. It has the capability to acknowledge reductions in alertness levels and allows an intervention strategy to counter the effects of driving fatigued. Under normal circumstances the device will present a light and audio stimulus every 7 to 10 minutes. Slow reactions trigger more frequent testing. Extremely slow reactions trigger alarm in the cab and an alert is also sent to the dispatcher. In either a reverse movement or when the machine is not in motion (stopped, loading or tipping), the fatigue monitoring device will be disabled so that operators are not tested as they may be looking elsewhere.

ARRB had good operator feedback regarding capability to detect fatigue, but many quickly became annoyed with the constant testing.

Pro-Active Fatigue Management System

arrb




contact Info/website: http://www.pernix.co.uk

primary technology: Predictive model of fatigue based on input of operator sleep history combined with tracking of steering wheel movements.

primary fatigue measure: Uses both predicted fatigue levels using mathematical fatigue models and fatigue-related steering behavior.

fatigue countermeasures: Audible and visual alarm, dispatch.


timing for readiness: Commercially available.

validation for fatigue: Correlated with PVT and subjective measures of fatigue.

Invasiveness/acceptance: Non-invasive. System can request operator to input their sleep history (which is optional) at start of system use.

environmental complications: Device is ruggedized and has stood up well during mining trials. System is not tamperproof. Operators can “unplug” the device.

cost: Approximately $4,000 USD per unit installed. New models for Summer of 2006 expected to be around $6,000.

Industries where used: Mining and currently being trialed in on-highway trucks.

summary: The ASTiD™ device has received positive feedback from trials in BHP-Billiton, Barrick and AngloCoal mines that were interviewed. The system utilizes a sleep/fatigue software model to predict levels of fatigue based on time of day, CIRCADIAN™ clock, and the amount and quality of sleep of the operator, and also tracking of steering wheel movements. The system can be set to require user input (quality of prior sleep) or can be set to assume varying levels of default sleep quality. Where installed, feedback from operators has been fairly positive. Operators felt that for the most part, the system was in line with how they were feeling. Although some operators experienced alarms when they were not feeling sleepy at all. This was seen largely when operators were not inputting their personal sleep history information.

ASTiD™

pernix ltd.




contact Info/website: http://attentiontechnology.com/index.html

primary technology: Infrared eye tracking camera

primary fatigue measure: Percentage eye closure (PERCLOS)

fatigue countermeasures: Audible and visual alarm


timing for readiness: Commercially available, but would require minor modifications for mining applications.

validation for fatigue: Correlated with PVT and subjective measures of fatigue.

Invasiveness/acceptance: The device has been trialed in the mining industry. Operators commented the rhythmic light emitted from the IR camera was hypnotic and made them drowsier. Operators complained about numerous false alarms due to the camera system being misaligned with the operators’ head. The system also was found to falsely detect fatigue during intentional head turns. Overall the device has not been well accepted in either the transportation or mining industry.

environmental complications: Does not work well in bright light and in rough environments.

cost: $1500

Industries where used: Primarily trucking. Limited trials in railroad and mining.

summary: The DFM was the first commercially available device to automatically calculate PERCLOS, making it a popular device among transportation researchers. Field trials in the trucking, rail, and mining industry have universally shown this technology to be overly sensitive to operator head movement and lighting conditions. Driver and operator feedback have been mostly negative due to high rates of false alarms. Other technologies have perfected automated PERCLOS detection and do not suffer from the same limitations.

Driver Fatigue Monitor

attention teChnologies



contact Info/website: http://www.delphi.com/search/?search=driver+state

(Keep in mind car manufacturers are very secret about their research, you will not find much information here.) For more information, see the following video link: Video Demonstration

primary technology: IR camera with real time assessment of driver fatigue and visual distraction using facial feature recognition algorithms.

primary fatigue measure: AVECLOS is a binary eye closure status measure and is defined as: AVECLOS = (Time Eyes Closed/ Time Total).

fatigue countermeasures: System uses adaptive audible and visual alarms commensurate with level of alertness (fatigue and distraction).

trialed in the mining application: None.

timing for readiness: In the short-term for field-testing. Potential use more for middle-term (1-2 years) and long-term.

validation for fatigue: Correlated with PVT and PERCLOS

Invasiveness/acceptance: Remote system, no direct contact with the driver. No data available on driver acceptance.

environmental complications: From lab testing, the system is robust to lighting conditions. Tolerant of traditional eye glasses as well as sun glasses. Automatically calibrates itself and easily re-establishes eye and head detection following significant body movement or head turns. The effect of vibration and extremely bumpy roads on the system is unknown.

cost: $500-3000 based on volume.

Industries were used: Intended for automotive and trucking industry

summary: The system uses an IR camera to assess driver fatigue and visual distraction using facial feature recognition algorithms, and has adaptive audible and visual alarms. This system is currently under development. It improves upon previous PERCLOS systems with functionality in all lighting conditions as well as tolerating head movement. The system has the added benefit, absent from all other technologies, of detecting both fatigue and driver distraction.

Driver State Monitor

delphi




contact Info/website:

http://neurocom.webzone.ru/eng/product/tskbm.htm

primary technology: The system measures electrodermal activity (skin conductance).

primary fatigue measure: The device is using a derivative of electrodermal activity, to predict times when a driver begins to loose concentration and alertness.

fatigue countermeasures: The device presents an audible and visual alarm to the user a few tens of seconds before the level of alertness decreases to a critical value.

trialed in the mining application: No

timing for readiness: Commercially available

validation for fatigue: The primary measure Electrodermal activity is validated against EEG and EOG.

Invasiveness/acceptance: The degree of invasiveness for the device is low to medium. Some people don’t like to wear wristwatches. Currently, the driver is required to ensure that the electrodes of the wrist watch make good contact with the skin. The driver must ensure the

radio signal receiver is illuminated. The device has to be tested and calibrated before each shift. Manipulations by the operator are possible at this stage.

environmental complications: Use in Russia in the railroad and truck operations assured the robustness of the device against rough environmental conditions. Device is not robust against the potential of manipulations by the operator.

cost: Approximately: 7000-8000 US$

Industries where used: Railway, Road transport; the device has been in use for 5 years on the Russian Railways and has been used in road transport for long-haul trips and transporting hazardous goods. The Ministry of Transport of Russia has approved the device.

In Australia, a modified version of the system called the DVTCS-L system is currently undergoing an evaluation testing by QR human factor specialists and rolling stock engineers in Brisbane. Worldwide, some 8000 Train Drivers are using the device.

summary: The device evaluates electrodermal activity to predict decreased alertness and concentration using a wristwatch like monitor, and triggers audible and visual alarm a few tens of seconds before the level of alertness decreases to a critical value. It is a tested and approved device, that has proved to be robust to environment conditions, and that is ready to be evaluated in the mining industry. The calibration process and the possibility of operator manipulation could be a disadvantage.

Engine Driver/Driver Vigilance Telemetric Control System

edvtCs/dvtCs




contact Info/website:

http://www.mcjeyecheck.com/www.aatl.net

primary technology: The device measures pupil reaction to light flashes (rate of pupil constriction, final minimum pupil diameter). Infrared optics.

primary fatigue measure: Pupil dilation/constriction fitness for duty device.

fatigue countermeasures: No



validation for fatigue: Yes, studies performing correlation with Blood Alcohol Content and consecutive number of hours of wakefulness. Also studies evaluating fatigue in different shift work patterns.

Invasiveness/acceptance: Not invasive, operator only has to look into the device. Good acceptance (test last less than 1 minute). May present privacy concerns (detects eye problems, prescription and recreational drug use).

environmental complications: None

cost: $8,000 (lower for large orders)

Industries where used: Police Departments, studies in aviation, trial in mining.

summary: It is a fitness-for-duty test. Would require repeated measurements during shift to evaluate fatigue. Based on medical principles and practical use. Easy to use, minimal training required. Analysis programs supplied.

First, the eye is allowed to adapt to dark (30 sec), and initial pupil diameter is measured. After a brief flash of light, the device measures reflex amplitude, rate of constriction, final minimum pupil diameter and time to minimum diameter

For drug-related impairment, a study found a sensitivity of 86% and specificity of 79%. Validated for fatigue. Minimal maintenance, minimal training for operator.

Manufacturer notes that it would not be difficult to install in truck cabin.

EyeCheck

mCj inC.edvtCs/dvtCs




contact Info/website: http://www.Seeing Machines.com.au

primary technology: Fully automatic, non-contact, remote IR video cameras. A GPS/GPRS tracking system connectivity for fleet management is intended for the future.

primary fatigue measure: Tracks driver’s head, face and eye features, including PERCLOS as fatigue measure. Latest system implements fatigue management based on detection of micro-sleep events.

fatigue countermeasures: Flexible-warning modalities (especially with OEM partnership): audible, visual, tactile and dispatch communications are in development.

trialed in the mining application: No, but demonstrated in CAT machine.

timing for readiness: Embedded fatigue warning system under development. Middle term to long-term.

validation for fatigue: PERCLOS is a scientifically validated fatigue measure. Field test in the US are under way at the moment. Results are still confidential.

Invasiveness/acceptance: Fully automatic, no driver interaction or calibration required. There is no direct contact with the driver.

environmental complications: The system is robust for different light conditions. Robustness to dust, vibration and extremely bumpy roads can only be evaluated by field test in the mining environment.

cost: Unknown

Industries where used: Intended for transportation industry in general.

summary: FaceLab is an automatic, non-contact, remote video camera based system to track driver visual behavior. The system is well established in a lab-environment. Field trials have recently commenced in US. The company is looking for Australian partners. Field trials under mining conditions are needed before recommending such technology. System could be on option for midterm or long-term technology solution.

FaceLab

seeing maChines


DSS CAMERA MOUNTED IN DEMONSTRATOR STATUS VIEW OF THE DSS DEMONSTRATOR



contact Info/website: http://www.acumine.com.au/hc/index.html

primary technology: HaulCheck is using a combination of two sensor technologies. The GPS system is used to track global position of the vehicle, and a laser scanning technology for local lane keeping requiring PVC post on the roadside.

primary fatigue measure: HaulCheck tracks the machine position relative to lane markers.

fatigue countermeasures: HaulCheck is using audible and visual warnings for roadway departure and collision avoidance with varying strength and alarm patterns.


timing for readiness: The system is currently installed in two mines (ALCOA Willowdale and Huntly sites). During the process the system will be further improved.

validation for fatigue: The HaulCheck system is using different approaches to prevent accidents due to operator fatigue problems. It saves all the events generated by the operator such as number of high and low alarms and the number of times the alarms have been cancelled manually or automatically. By looking at this data and comparing with truck trajectories and additional information operator fatigue condition can be predicted.

Invasiveness/acceptance: The HaulCheck system is completely non-invasive and operators cannot manipulate the system.

environmental complications: The system is built for mining operations and should be able to cope with all occurring conditions.

cost: There are costs for the basic equipment (GPS, laser scanner) and costs for infrastructure (setting up and maintaining PVC-posts)

Industries where used: The system was developed for mining operations in cooperation with Komatsu. It is currently used at Alcoa’s Willowdale and Huntly mines in Western Australia.

summary: HaulCheck uses a combination of GPS and a laser scanning technology to track the machine position relative to lane markers. It provides

audible and visual warnings for roadway departure and collision avoidance.

AcuMine the vendor for HaulCheck is working jointly with Komatsu on further improvements of the system. The system has advantages because the developers were familiar with the mining environment. In addition, centralized real time tracking of truck positions, driver status and alarms events should benefit the mining operation beyond the fatigue detection task. The major downside is the requirement for infrastructure (PVC-posts) to be installed and maintained.

HaulCheck

aCCumine


HAUL TRUCK EqUIPPED wITH FLEET MONITORING CAPABILITIES



contact Info/website: http://www.iteris.com/av/avindex.html

primary technology: AutoVue is a Lane Departure Warning System (LDWS) that uses video camera and machine vision to recognize road markers, detect and prevent lane departure of the vehicle.

primary fatigue measure: Uses variation in lane deviation as fatigue measure.

fatigue countermeasures: When a lane departure occurs, the unit automatically emits the commonly known rumble strip sound, alerting the driver to make a correction.


timing for readiness: The system is ready now and widely used in commercial truck fleets. The adaptation to mining operation will take time and money and is feasible only long-term.

validation for fatigue: Was validated in the lab from against other fatigue measure such as EOG, and PERCLOS. A combination of the LDWS with eye tracking technology led to reduction of false alarms by 70%. No field evaluation results known at the present time.

environmental complications: System shows sensitivity to weather and road conditions, such as light reflections, and doesn’t work on snow-covered roads.

cost: Unknown - Information will be provided when available.

Industries where used: The LDW system is used in the commercial truck fleets in the US.

summary: The system uses video camera and machine vision to prevent lane deviation and triggers an audio alarm (rumble strip sound) when lane departure is detected. Widely accepted system in the trucking industry. However, the system is very sensitive to environmental conditions. System needs a major modification to become useful in mining environment as tool for accident prevention.

AutoVue

iteris




contact Info/website: http://www.mobileye.com

primary technology: Video camera and imaging processing sensor technologies integrated in one chip for multiple car safety and accident avoidance functions. (1) Pattern recognition is one of the core technologies developed by Mobileye. It is used for obstacle detection and classification. (2) The image processing techniques are used for lane detection and road geometry prediction. (3) An optic flow analysis allows detecting interesting regions in the scene even before the entire object is fully visible. (4) The ego-motion system recovers the motion parameters of the car. The estimated parameters include Pitch, Yaw, Roll and Forward Translation. (5) With vision range estimation a measure of distance to obstacles, and relative velocity can be obtained.

primary fatigue measure: System could use variation of lane deviation as well as vehicle motion as fatigue measure. But the company is focused primary on car safety and accident prevention.

fatigue countermeasures: Audible and visual alarm is based on leaving the lane or on collision avoidance.


timing for readiness: For general car safety application the system is ready now. For safety applications, accident prevention, fatigue detection in the mining industry the system has to be adapted and has only long-term potential.

validation for fatigue: No validation for fatigue is available. System is focusing on safety.

Invasiveness/acceptance: System is vehicle based and non-invasive.

environmental complications: Could be problems with rough conditions in mining environments such as dust, vibrations, heat, etc.

cost: Unknown. Complete system can provide multiple functions of driver support and vehicle monitoring to reasonable price because it is integrated in one chip.

Industries Where Used: Automotive industry.

summary: The system use cameras with imaging processing technology to detect road features and driving behavior. The system provides an audible and visual alarm to avoid lane departure and collisions. The system has everything that is needed for accident prevention, but considerable time and money would be required to modify such system for mining operations. Company expressed no interest in working in mining industry.

MobilEye




contact Info/website: http://www.rct.net.au


primary fatigue measure: Reaction time task. The device measures operator’s reaction time to visual stimulus.

fatigue countermeasures: Audible alarm.



validation for fatigue: No data provided, however slowed mental reaction time is a known correlate with fatigue.

Invasiveness/acceptance: The device is fairly invasive, as it requires secondary task performance of the operator. Usually well accepted, especially when company had recent fatigue-related incidents. Manufacturer also noted cultural differences affecting

operator acceptance.

environmental complications: None. Resistant to dust, vibration, etc.

cost: $1,300 (Australian)

Industries where used: Mining, transportation, agriculture, construction, industrial

summary: The system checks operator alertness using a visual signal. Two modes of operation: a) random signal, b) stimulus gets more frequent with slower reaction time. If operator does not respond (by resetting switch), an audible alarm will sound until operator resets the system. Does not produce stimulus when vehicle is parked. Requires minimal maintenance, about 5-6 hours for installation in truck, about 30 minutes training for operators. The system has been usually well accepted by operators. Mine users noted they think it has helped avoid accidents.

Fatigue Warning System

muirhead




contact Info/website: http://www.safetyproductsunlimited.com/nap_zapper.html

primary technology: Mercury switch.

primary fatigue measure: Head nodding

fatigue countermeasures: When operator inclines his head beyond a certain angle, an audible alarm sounds.



validation for fatigue: No data provided.

Invasiveness/acceptance: Operator has to wear the device around his ear (very light, less than 1 oz.). If person wears glasses it sometimes hinders proper “hanging” of NapZapper.

environmental complications: No

cost: US $10 (could be lower for large orders).

Industries where used: Automotive, mining

summary: Small device, worn around the ear. When reaching a certain angle the mercury closes the switch and starts a beeper. The device has an ON/OFF switch that has to be turned ON for the device to be functional (possible that operator wears the device, but the device is not active). Vendor (Safety Products Unlimited) imported the device from China, noted that has started working with a different device (Driver Alert), due to often receiving devices that did not work. Interview with mining company that has been using it for over one year, said very satisfied, use of device is mandatory, accepted by operators. Mine stated that there were no fatigue-related accidents since implementation. Interview with another mining company indicated that the device is delicate and should be used/carried with care; mechanism should be tested regularly to be sure that is working (breaks easily). Shape of ear plays a role for proper “hanging” of device.

NapZapper




contact Info/website: http://www.atlas-arl.com

primary technology: EMG

primary fatigue measure: Muscle tone.

fatigue countermeasures: Vibration and audible alarm



validation for fatigue: Yes, extensive lab tests, EEG validation.

Invasiveness/acceptance: Operator must wear a device similar to a wristwatch. Not tried in workplace, no data on operator acceptance.

environmental complications: No.

cost: Inexpensive < $1000 USD

Industries where used: Not used in workplace yet.

summary: When muscle activity decreases below a set baseline, the device administers a of dual pulse vibratory stimulation on the wrist. The user should respond to the second pulse, increasing grip of the monitored hand. The vibration is repeated every 10 seconds if the user does not respond correctly. If the user is inattentive and extremely fatigued, the alert-o-meter will also sound an alarm buzzer to suggest that the driver stop immediately. Device needs to be calibrated for each individual before each use (calibration takes less than 5 minutes). Possible to transmit driver status to remote data collection facility. Manufacturer is currently developing smart steering-wheel cover (Actigrip), and ultimately goal is to embed algorithms into steering wheel.

NOVAlert

atlas ltd.




contact Info/website: http://www.Optalert™.com

primary technology: IR eye blink analysis

primary fatigue measure: PERCLOS, Gaze, blink rate, AVECLOS, various other eye characteristics.

fatigue countermeasures: Audible and voice alarms.


timing for readiness: Commercially available for trial.

validation for fatigue: Validated in the lab with EEG, PERCLOS, PVT, and commercial vehicle field trials.

Invasiveness/acceptance: Requires operators to wear special glasses with a wire attached. Operators have complained about discomfort from the eyeglasses.

environmental complications: Sensitive to dust and vibration. Newer generation have proven to be more robust.

cost: $16,000 AU per unit: includes 1 pairs of glasses (Volume pricing possible).

Industries where used: Intended for trucking and perhaps mining.

summary: Optalert™ takes some of the basic principles of other fatigue detection technologies and has innovatively incorporated them into a pair of eyeglasses. Optalert™ has taken advantage of the extensive sleep physiology expertise of the company’s founder, Dr. Murray John. Using proprietary fatigue algorithms, Optalert™ detects and monitors several measures of alertness and microsleeps. The company also claims their system is capable of detecting loss of attention. Optalert™ has spent considerable time evaluating their technology in the mining environment and is working to address the system’s sensitivity to vibration and eyeglass slippage. With continued upgrades this system could become a key part of a comprehensive alertness program.

Optalert™

sleep diagnostiCs ltd.




contact Info/website: http://www.ospat.com/

primary technology: Stimulus reaction time.

primary fatigue measure: Hand/eye coordination

fatigue countermeasures: Pass/Fail as a fit for duty test



validation for fatigue: Correlated with PVT.

Invasiveness/acceptance: Operator must interact with a screen and mouse for 90 seconds.

environmental complications: Some issues with first generation equipment and downtime. Newer models have been more reliable.

cost: Unknown at time of printing

Industries where used: Mining, shift work, transportation

summary: The OSPAT is a fitness for duty test that evaluates eye-hand coordination test using video screen and a mouse or other computer interface. The system has shown very good correlations to other accepted measures of alertness testing (such as PVT). The system is particularly good for pre-shift testing but not on-shift monitoring. Would make a good part of a comprehensive alertness program. The OSPAT does not discriminate between fatigue-related and other kinds of impairment (alcohol or other drugs).

OSPAT




contact Info/website: http://internet.cybermesa.com/~roger_jones/press_dd.htm http://www.headtrak.com/

primary technology: An electronic sensor array detects the highly conductive human body. A change in capacitance correlates to proximity of the body, which is changing with motion.

primary fatigue measure: Real time tracking of head movement, micro-nods over time correlate to drowsiness, strong head nodding can indicate microsleeps.

fatigue countermeasures: Visual notification is initiated when the system output is above its threshold for a specified percentage of time. If this percentage increases, visual notification is combined with a few seconds of audible notification. If the percentage of time above threshold continues to increase, the audible notification cycle time lengthens and repeats. A reset switch is provided to momentarily disable the notification devices. Besides the described options of audio and visual alarms a dispatcher warning can be triggered.


timing for readiness: Not in the short-term. System must be integrated by car manufacturer. Therefore, has potential only for long-term applications.

validation for fatigue: The system was evaluated in lab setting. The system has shown a medium ability for fatigue prediction, but good ability for microsleep detection.

Invasiveness/acceptance: Device has to be integrated in the truck. Therefore, it is completely non-invasive. No possibility by the operator to manipulate or damage the product.

environmental

complications: System is very robust, as it is not affected by dust, noise, or lighting conditions. There could be some issues in an environment with large vibrations and extreme up and down movements of the operator.

cost: Unknown, but it is an already proven car technology, if it is integrated in the truck the costs should be reasonable.

Industries where used: System is used in the automotive industry as sensor to trigger airbag. In the future, there are intentions to introducing systems for long-haul trucking.

summary: System tracks head movements in real time, and triggers an alarm when output is above threshold. Car manufacturer or other OEM must integrate system. It cannot operate as drowsiness detection system alone. Covers some features of drowsiness that no other system can cover. Key is a sophisticated data analysis method. If integrated directly in the mining truck, the system would be very stable to the mining environment and could help to prevent accidents.

PASS




contact Info/website: http://www.assistware.com/

primary technology: Digital video, road feature detection. SafeTRAC is an in-vehicle safety system that uses a forward-looking video camera to monitor the road ahead. It tracks road features to determine a vehicle’s position and trajectory and generates a warning if a vehicle begins to drift out of its lane. SafeTRAC also detects drowsy or distracted driving by sensing weaving or erratic lane keeping. The ultimate goal of these vehicle-based technologies is accident prevention.

primary fatigue measure: Lane deviation warning system. Tracks lane maintenance over time.

fatigue countermeasures: Audible, visual and vibration alarm


timing for readiness: Commercially available for trucking industry, technology is not transferable to mining environment. Requires uniform lane markings.

validation for fatigue: Lane deviation, erratic steering behavior has been shown to correlate with fatigue benchmarks.

Invasiveness/acceptance: No data available

environmental complications: Works in most lighting and road conditions, but has some sensitivity to rain. As all camera systems there may be issues with dust and vibration, but not specific data are available.

cost: Unknown at time of printing

Industries where used: Trucking, entering into automotive

summary: SafeTrac uses imaging processing methods to track and analyze lane-keeping behavior. The system warns for roadway departure, unsignaled lane change and reduced alertness. Driver lane-keeping accuracy over time gives an alertness measure. The system has been very well accepted in the trucking industry, however this is not a technology capable of being transitioned to the mining industry.

SafeTrac

assistware




contact Info/website: http://www.ericar.cl

primary technology: Head inclination detection

primary fatigue measure: Head nodding.

fatigue countermeasures: Audible and visual (internal and external) alarm



validation for fatigue: Manufacturer stated that head nodding angle that triggers alarm was determined by trial and error testing. Several field trials conducted in mining operations.

Invasiveness/acceptance: Operator must wear the helmet. Manufacturer indicated acceptance problems at the beginning of trials, but very good acceptance later on. Mining users did not indicate acceptance problems.


cost: Approximately $1,000 (kit in cabin plus helmet device)

Industries where used: Mining

summary: The system consists of a device in the helmet that triggers an audible and visual alarm when head nodding is detected. There are three levels of sensitivity that can be adjusted by operator. Alarm is triggered in both the fatigued operators’ cab and also in other trucks within 50 yards. Other operators can also trigger an alarm if they think that a colleague is having fatigue problems. Works with 2 batteries, which could be removed by operator to inactivate the system. There is no ON/OFF switch meaning the system is always active. Mining trials have been mostly positive with some complaints of the systems sensitivity to non-fatigue related head movement. Data storage and dispatcher feedback could be implemented.

Sleep Control Helmet




contact Info/website: http://www.smarteye.se/products.html

primary technology: The Smart Eye is a single or multiple camera based system that uses an innovative active IR lighting method.

primary fatigue measure: Head position, head orientation, gaze direction, eyelid opening.

fatigue countermeasures: No fatigue countermeasure implemented.

trialed in the mining application: No, but demonstrated in a CAT machine

timing for readiness: As research tool it is ready now. Has potential as driver monitoring system under general operation conditions in the near future. The system is only appropriate for mining operations in the long-term and with major modifications.

validation for fatigue: System has all possible features to be a benchmark for the evaluation of other fatigue monitoring technologies.

Invasiveness/acceptance: Non-intrusive gaze, eyelid and head tracking system and it is difficult to manipulate by user especially when multiple cameras are used.

environmental complications: Special efforts have been put on robustness to different types of eyeglasses, in particular situations where the frames occlude parts of the eye. Reflections in the eyeglasses are efficiently eliminated using a patented illumination technique. System is sensitive to dust and vibrations.

cost: Research system approximately between $10,000-$30,000 USD

Industries where used: Automotive safety research

summary: Camera based system that uses an innovative IR lighting method to track head position and orientation and eye features. No fatigue countermeasure.

System is at the present time a very comprehensive tool to test and evaluate other driver fatigue monitoring technologies. Not considered a stand-alone fatigue detection device.

AntiSleep

smarteye




contact Info/website: http://www.smi.de/

primary technology: Digital video, IR eye tracking

primary fatigue measure: Blink rate, eye closure, head movements, eye features, gaze, PERCLOS

fatigue countermeasures: None at present

trialed in the mining application: No, but a demonstration is planned for a Caterpillar machine this summer.

timing for readiness: Demonstration system available for field trials.

validation for fatigue: PERCLOS is a validated fatigue measure.

Invasiveness/acceptance: Non invasive.

environmental complications: In its current state, vibration is likely to be an issue in the mining industry.

cost: Not a commercial product

Industries where used: Automotive

summary: This should not be considered a fatigue detection device, as it is currently only used for experimental eye tracking. It has been used in lab studies to measure fatigue but only through an automated PERCLOS calculation. It has the potential to track much more including microsleeps with the proper software algorithms. So far, most of its use has been in the passenger automobile industry.

InSight

smi




contact Info/website: http://www.internationalmining.com.au


primary fatigue measure: Reaction time task. The device measures operator’s reaction time to audible stimulus.

fatigue countermeasures: Audible and voice alarm.



validation for fatigue: Slowed mental reaction time is a known correlate with fatigue.

Invasiveness/acceptance: Invasiveness: Addition of a secondary task for the operator. Acceptance: Mining user did not report major acceptance problems (they noted that their system deactivated the am/fm radio when activated, which annoyed the operator).


cost: Unknown at time of printing.

Industries where used: Mining

summary: The system has two options, simple reaction time or multiple choice reaction (simply push a button, or to push the button with a particular shape). The frequency rate and volume of commands can be programmed (e.g., commands more frequent at times of day when alertness is lower such as at 3 am, commands more frequent at the end of the shift), and the system can also be programmed to ask specific questions and respond to individual situations or response habits. When slow responses occur, an alarm is triggered in the cabin, and an alarm also sent to control room and may be sent to all trucks in the fleet. Data sent by radio to control room in real time. Can record and store data for long periods of time. Mining users indicated simple, robust design, but need to appoint a person to be in charge of system maintenance. There is the possibility that the operator may subconsciously reset the alarm while remaining fatigued. System is not activated when truck in reverse gear or neutral, heavy-duty foot operated reset button installed, improved flexibility with the alarm timing depending on the hauling circuit travel time.

Voice Commander

international mining teChnologies
