problems of attention detection and prediction for subjects interacting with transportation systems...

Problems of Attention Detection and Problems of Attention Detection and Prediction for Subjects Interacting with Prediction for Subjects Interacting with

Transportation SystemsTransportation Systems

Prof. Ing. Mirko Novák, DrSc.

Czech Technical University, PragueFaculty of Transportation Sciences,

Department of Control Engineering and TelematicsLaboratory of System ReliabilityKonviktská 20, Prague 1, 110 00

e-mail: [email protected]

AbstractAbstract:

In this contribution the main problems are discussed, which one has to face when wish to detect, analyze and predict the decrease of human subject attention in the course of his/her interaction (control, use) of artificial (namely transportation) system, having in the mind the need for the development of practical and function reliable on-board applicable operator and/or his/her supervisor warning system.

Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems

Content:

1. The Problem and its Significance

2. Possibilities of Solution

3. Results and Applications

4. Expectance


1. The Problem and its SignificancePremises:

- Till now, there is no artificial system at disposal, which can operate or be used absolutely without any interaction with human subject.- Human subject can deal with artificial system without faults only fir limited time, after that his/her attention decrease and the danger of accidents caused by wrong interaction increases. - Statistics tell us that e.g. on roads 10 – 15% of all accidents is caused by decrease of drivers attention. In the Czech Republic this leads to losses of about 1 billion CK per year.

Problem: Insufficient reliability of system operators interaction causes extremly high losses on health and money – especially as concerns the transportation. The problem consists in finding the way how to minimize or even prevent this.


The goal:

To find methods and tools for detection,

analysis and prediction of transport

system operators attention decrease

(even micro-sleeps) and to develop

reliable and practically applicable

warning systems.


Requirements:

• Non-invasive for operator

• Speed of operation

• Operation reliability

• Safety

• Easy use

• User friedly

• Satisfactory long life-time

• Acceptable investment and operation price


2. Possibilities of Solution:

- the use of so called secondary significant markers of attention (eye movement frequency, temperature, skin el. resistance, face character etc.)

advantage: easier practical use

disadvantage: insufficient specificity,

high time delay

- the use of primary markers of attention (EEG, EMG):

advantage: minimal time-delay,

high specificity

disadvantage: complicated practical use.


3. Results and applications:

After abot 5 years of research (team LSR FTS CTU, ICS CAS, MA Brno, Neurological clinic 1.MF UK etc) is:

- at disposal and laboratory tested methodology for in-time detection, analysis and prediction of attention decrease of artificial systems operators in the course of their service based on EEG analysis

- formulated the necessary background of specific data-base of results of coordinated measurements of EEG signals and probands reaction (Micro Sleep Base – MSB)

- developed the concept of the warning system

Problems of Attention Detection and Prediction for Subjects Interacting with Transportation SystemsOpen questions:

- the creation of satisfactory wide data-base for generalization of the obtained results,- the development of satisfactory reliable and low power consuming transmission tool between the head detectors and on-board equipment - development of EEG detection set minimizing the discomfort of operator- verification of long time applicability of the specific individual tuned classificators and predictors of attention decrease and micro-sleeps and possibilities of their eventual typization- finding of optimal ways for operator warning with respect to specific operation conditions - solution of specific ethic and legal problems connected with exploation of operators and probands personnal data


4. Expectance

a) In the national (CNNN – Czech National Node for Neuroinformatics) and international cooperation (Global Science Forum OECD) the Micro Sleep Base – MSB will be filled out by satisfactory amount of relevant data

b) The methodology and tools for economic, ethic and legal acceptable use of long time applicable individual oriented prediction and warning against decrease of interaction reliability of operator dealing with artificial system will be developed

c) The practical aplicable and economic acceptable on-board warning systems will be developed and set into wide scale use.

1. Introduction

No of these many artificial systems, with which human society has daily to deal is able to operate quite independent – all of them up to now have to interact with human subject, i.e. to be controlled, supervised or at least used by man.

The quality and reliability of such interaction is of top importance for all of us, without respect to the level of the particular artificial system intelligence.

Unfortunately the very long experience of human dealing with artificial systems lead to observation, that very often the human ability is the weakest point in such interaction.

The reason is quite natural and easy to understand:

• On contrary to artificial systems, human cannot operate too long without brake - he needs to relax, rest and to sleep. The limitation and in the course of his/her operation service or usage of the respective system decreasing human vigilance and attention was and still is the most frequent reason of many system failures and accidents.

• Another source of system failures has to be seen in the possibility, that the human operator (or user) of particular artificial system can react late and that his/her decision and choice of kind of reaction must not be correct.

• Human behavior is not fully deterministic; it varies from subject to subject.

All these factors combine with the result:

the reliability of

human subject – artificial system interaction

is limited,

dominantly from the human side.

The price, which all we pay for the not satisfactory reliable human – artificial system interaction (non-satisfactory before all from the human side) is tremendous.

More over:

it increases by time

and

by level of artificial system sophistication.

We urgently need to do something against it.

There are various approaches how we can try to minimize the hazard of system operation

failures.

In general there are the following main ways of improvement the system reliability:

1. - One of the oldest one consists in the tendency to make the system as much robust, preferring the use of maximal possible reliable (and therefore large, massive and also expensive) components for its construction.

• - Another approach is based on the duplication and often even the multiplication of the most important parts of the system under consideration or even of the duplication or multiplication of the whole systems, which eventually can operate as “hot” reserve. Of course, like the first mentioned approach, also this can require very large effort.

• - More sophisticated approach tries to modify the system structure so, that the sensitivity of its system functions to the parameter changes will be minimized.

- In very sophisticated systems the approach of the predictive diagnostics can be applied, by which one can analyze, how much and in what direction the values of some system parameters can deviate from their nominal values so that the system as a whole is still of acceptable properties and to predict with which probability and when these limits of acceptable system behavior will be broken.

Actually, no of these four main approaches is universal, in praxis they are usually combined.

As concerns the reliability of interaction among human subjects and artificial systems, only the approach of predictive diagnostics seems to be prospective for to be candidate of really effective minimizing the frequency of system operation failures caused by human errors and miss-functions,

though the first mentioned approach, represented in the area of human subject the tendency of selection the best qualified people from all candidates for operators and in the improvement of their training and testing is also very important.

In this talk, devoted to the problems related to the technical aspects of the reliability of interactions between human subject - either in the role of operator (driver, pilot, dispatcher), supervisor or user - and the artificial, namely technical (and before all transportation) system, the final attention is given dominantly to the possibilities how to create a practically applicable tool for diminishing the losses, which are up to now daily caused by decrease and limitation of such interaction reliability.

The problem of interaction reliability among artificial systems and human beings (operators, users), though extremely important, is still not solved satisfactorily even the control of dominant part of complex and complicated systems is now realized by computers or at least by computer assistance.

The human operator, who has to interact with powerful, complicated and often also efficient artificial system (such as transport system, aircrafts, express trains, large trucks, extensive power stations, security and defense systems etc.) is imposed to requirements on fast and correct reactions on very variable actual situations and he/she is exposed to this for considerably long time of his/her service. The resulting high load of such operator brain and nervous system results necessarily to decrease of his/her vigilance and to degradation of his/her attention.

The complex demands on human operator interacting with complicated and powerful artificial system are more over often jointed with influence of some internal and external factors (such as long length of service, psychical isolation in the course of service, operators actual mental and physical state, climatic conditions, quality of environment in the particular cock-pit or control room, monotony of the scene or image, which operator has to observe etc).

Also some psychical load of the respective system operator and the presence of pre-service or in the course of the service acting stress factors can play very important role.

In this respect we have to take into account also various other circumstances, typical for humans, namely:

- The extremely high variability of human subjects, especially as concerns their intelligence, education, level of operation training and skill, ability to concentrate, tendency to no rational and panic reactions on unexpected and suddenly appearing situations, general temper, tendency to be aggressive etc.

- The human subjects dealing with artificial systems also differ very much in age, race and sex.

- In contrary to artificial systems, the reaction of human subjects cannot be exactly repeated – the human subject learns from experience and modifies his/her behavior to minimal expense of physical and psychical energy, which he/she has to use for system control or use.

All these factor complicate seriously the problem – we cannot use the standard methods of control and system engineering. Nevertheless we need to transfer the methodology from standard psychologically oriented approaches used for decades in system operator education and training to more contemporary ones, dealing with much more knowledge of the field of system reliability theory, mathematical analysis, neurology, artificial neural networks,numerical methods etc.

2. The role of interaction with human subject in artificial system operation

Artificial system can interact with human being in following principal ways:

- - Interaction on the basis of human control of system operation,

- - Interaction on the base of human supervision of system operation,

- - Interaction on the base of human use of system operation,

- - Interaction of human society on system operation.

It is evident, that in all these cases the interaction failures can cause fatal situations, or at least serious economic losses.

However, also in the case, when the interaction among some artificial system and various members of human society forming its environment is considered, the interaction reliability and predictability could be of very high importance. This concerns especially such situations, when the artificial system changes suddenly its behavior and when it interacts with considerably large and heterogeneous part of human society. The acceptable reliable estimation of most probable kind of such environmental reaction needs deep understanding not only of individual behavior of human subject exposed to interaction with varying properties of particular artificial system, but also of the possible social factors, which exist or which could be activated in the respective part of human society. Such studies are evidently of top importance for general safety, however are very difficult and laborious and require the access to satisfactory large special databases in which the results of many measurements of human subject interaction reliability markers are stored. The later on mentioned Micro Sleep Base represents an example of one of them.

Recently, the technological progress of the reliability of artificial systems has greatly improved.

Consequently the probability of technical faults in well-designed and well -manufactured artificial systems is now usually very limited (though unfortunately still not excluded).

However, the probability of failure caused by misuse and faults in system operator activities increases rapidly.

The main reason for this not acceptable situation can be seen:

• in increasing complexity of the operated and used systems,

• in increasing requirements on the operator’s ability to deal with them,

• in the increasing requirements on his/her level of continuous and long- time attention and

• in the increasing requirements on the speed of his/her reactions.

Naturally, the losses caused by artificial system operation faults are proportional to their power, significance and value.

In the case of many modern transportation systems (large planes, fast trains, large ships, trucks), large power plants, important financial systems, security and defense systems, and also important medical care systems, the losses caused by their malfunction could be extreme high or also of catastrophe character.

Therefore, besides the continuing interest in diminishing the probability of technical failures in any artificial system as much as possible (with respect to economically acceptable expenses), considerable interest has also been shown in recent years in the reliability of system operator activity.

Many statistics demonstrate that the amount of human error represents a still larger proportion of all the expenses, which are required for the compensation of artificial system malfunctions.

The requirements on a human operator of an artificial system can be concentrated in the following main categories:

a) Requirements on attention level and continuity,

b) Requirements on the speed of operator reaction,

c) Requirements on the correctness of operator decisions.

Suppose that all three above mentioned kinds of requirements can be expressed as the limits, outside which cannot be accepted the real number xi of some significant parameter xi, i =

1,2,3 by which we express the level of attention, the speed of reaction and the correctness of operator decision.

The acceptable values of the vector X = {x1,

x2, x3} fill out in the space X some region RA, called

the region of acceptability. The concept of regions of acceptability is well known in the theory of tolerances of system parameters. In the theoretical case, that the parameters xi are independent, the

region RA has rectangular form. However, in

practice, among x1, x2 and x3 certain correlations

exist. This leads to much complicated form of RA.

Let us suppose, that we deal with so called well operated systems only, i.e. with such of operator - system interactions, which work well at least of the beginning of their use or of their observation.

One can formulate therefore the following theorem:

Each real interaction between human operator and artificial system, when exposed to the influence of a set of independent variables P (before all of the time P = pt), can be represented by

some position of the vector X in the parameter space, moving along certain interaction life-curve . For t , the life-curve breaks at least once the boundaries of RA.

t t

• Fig.1-The idealized run of the interaction life-curve of operator – system interaction inside and outside the respective region of acceptability.

In accordance with the general theory of system reliability, we can define

The reliability of some operator - system interaction is represented as as the probability H that for certain time interval (or some interval of other independent variable influencing the system under consideration) the respective interaction life-curve (t) will remain inside the region of acceptability RA .

Similarly:

Safety of the operator - system interaction is to be considered as the probability that it will be resistant against the eventual disturbing influences.

A straightforward correlation exists especially between attention level and speed of reaction.

Operators commanding at high level of attention usually also possess very fast reactions. On the other hand cases can appear, when fast, almost impulsive reaction may not be accompanied by very high level of the operator’s concentration and attention. Some people can react fast also, when their attention is dispersed on very different objects (they have very fast but unreliable reflexes).

A drop in the attention level of a particular human operator can be caused by various external or internal reasons; some of them have a general character; the intensity of others depends significantly on the operator’s individuality.

Among the general conditions causing the decrease in attention is: • - Extreme length of a particular operator’s service without breaks,• - Operator’s physical and mental exhaustion,• - Monotonous scene, which the operator has to observe for a long time,• - Extreme temperature in which the operator has to serve (too high or

too low),• - Extreme humidity in which the operator has to serve (too high or too

low),• - Extreme air pressure,• - Air smell, dust density etc.

In practice, the shape of RA and life-curve ψ(t) is much more complicated, as it is shown in the schematic Fig. 1.

Before all, the operator attention can improve in the course of his/her interaction with the system even if he/she fall already in the stage of real micro-sleep. In such case the respective interaction life-curve ψ(t) returns inside RA and after some time the procedure of attention decrease starts again. Such episodes, schematically expressed in Fig. 2 for two repetitions, can repeat several-time, until successful end of particular operator service in fortunate case, or until accident in negative case.

Fig. 2: The repetition of braking RA boundaries by (t) in

two temporary and one final micro-sleep episodes.

The number of possible repetitions of such braking RA boundaries in micro-sleep

episodes depends on many factors, including the individuality of particular operator,

his actual mental and physical state,

the environmental and technical conditions of system operation and control etc.

Though this represents a very important problem, the contemporary knowledge in this respect is far from to be called satisfactory and much more research in this area needs to be done.

Like in many cases when the regions of acceptability of technical systems operation are investigated, also here very often the boundaries of the regions of acceptable interaction can be of fuzzy character.

3. Human vigilance and attention and ways to their decrease classification and prediction

Micro-sleep is usually characterized as such a state of an organism, in which the eyes are closed and vigilance approaches zero.

On the contrary, one can understand by micro-sleep also such a state of the organism, when its vigilance decreases below a certain limit, without respect to the activity of visual tract. There are also several other conceptions of micro-sleep between these two limits.

Evidently, for the purpose of dealing with man-system interaction reliability, the aspect of decrease of vigilance is much more significant than closing of eyes, but the break of visual information input longer than certain limit (about 1 second for car drivers e.g.) can be also dangerous.

The following statement has been considered by the opportunity as a reasonable basis for the conception of micro-sleep, as it will be used in further discussion:

Let suppose that the level of a human operator’s attention can be measured by some real figure of merit LAT.

As it is schematically shown in Fig. 3, the level of attention LAT decreases with the time, in which the human subject mental activity in the course of his/her interaction with the artificial system is observed.

Here we can distinguish the following four basic stages:

a) Stage of the vigilance (full attention), in which the respective subject is completely competent to control (or use) the system under consideration,

b) Stage of relaxation, in which the respective subject is still competent to deal with the system under consideration, however where his/her attention decreases subsequently. This stage can last for considerably long time.

c) Stage of somnolence, in which the competence of the respective subject to interact with the system under consideration becomes to be restricted. This stage can last also considerably long, however in contrary to the previous one, here the real danger of the respective subject control faults exist.

d) Stage of hypnagogium, in which the respective subject fall into micro-sleep, at first with open eyes, however with very limited ability to control the system under consideration, later on into the micro-sleep with closed eyes, in which the respective subject competence to control the system is almost zero (some very skilled drivers - namely the professional truck drivers - can also in this stage hold the vehicle in straight move, however they cannot adequate react

to any road curvature or barrier on the road).

Fig. 3: The decrease of attention in the course of human subject activity

In general, one can define the micro-sleep as follows:

Micro-sleep is such a state of the human organism, in which the mental vigilance and attention of the human operator dealing with some artificial system (operating,controlling or using it) decreases for below a certain limit.

Micro-sleeps episodes can be of various lengths. For each kind of human subject interaction with artificial system some limits of maximum acceptable decrease of human subject attention and of micro-sleep lengths exist, over which the particular micro-sleep must be considered as dangerous.

This discussion must be completed by at least the following considerations:

a) The minimal acceptable level LATmin of the

human organism’s mental attention LAT

depends significantly on the requirements which are necessary for a certain application of a human operator - artificial system interaction.

b) The micro-sleep episodes can be classified according to their general length in t and according to the depths of the decrease in LAT

level (depth of micro-sleep).

The stages belonging to the class of micro-sleep with open eyes are usually as precursor of the micro-sleep with closed eyes. At such a state of organism certain level of its vigilance still exists, but his/her attention is considerably lowered and his/her reaction time RT is significantly prolonged. Also the probability of a correct and fast decision how to react to presented stimuli can decrease significantly.

After some time the light micro-sleep modifies usually into real micro-sleep with closed eyes.

This second class of micro-sleep is usually partially similar to the regular REM phase of a real night sleep nevertheless it lasts for a much shorter time.

The operator sleeping in micro-sleep with closed eyes cannot respond to any change of the system parameters, which he/she has to control.

We can suppose, that driver in the stage of micro-sleep with open eyes can react on some input signals of acoustic and visual character, though much slower and with higher possibility of wrong reaction, while when he falls in the mentioned light form of micro-sleep with closed eyes, he eventually can react on some basic input signals of mechanical character, like vibrations, acceleration and deceleration and position (stability), while in very deep micro-sleep with closed eyes he/she cannot react at all.

The micro-sleep with open eyes can last a considerable time and though the respective operator’s attention in such a situation is still near the limit of the acceptable level (his actual LATmin < LAT < LLATmin). The operator sleeping in such a form of micro-sleep has quite changed (lowered) other significant parameters (markers) of his attention and therefore he/she is practically unable to control in full quality any artificial system. Therefore, this can be also quite dangerous. The human operator’s attention is represented in Fig. 3 by a scalar figure of merit LAT. In maximal simplification, this can be considered as the inverse of reaction time RT.

When the human subject falls subsequently in the course of his/her artificial system operation in the stages of relaxation, somnolence and hypnagogium, his/her reaction times prolong significantly. While in full vigilance, the typical values of RT are about 200 ms, for the stage of somnolence RT in the range from 400ms to 500 ms appear and in the stage of hypnagogium the values of RT can exceed 800 ms. If the human subject load is prolonged more, it usually falls in real micro-sleep with closed eyes and his/her RT increases up to the time of awaking (this can be also several minutes). The values of RT in full vigilance below 200 ms are found exceptionally only.

The significance of danger caused by such prolongation of RT varies according the kind of human interaction with particular artificial system.

Typically, for car drivers, the values of RT above 400 ms represent the distance about 15 m in speed of 100 km/s, which the vehicle runs without any specific control (braking, turn etc.) corresponding to the stimulus (signal) received by the drivers sensors (of course, to this distance one has to add the distance, caused by technical reasons, like braking time etc.).

The endeavor of our investigations is at present the diagnosis of attention decrease and of the related reduction of the reaction speed to unexpected emerging situation. Such attention degradation can also result in real micro-sleep. Both these critical types of states of operator brain can be extremely dangerous and can result not only in huge material and financial damages, but also to losses of human life.

For to be able to detect the attention decrease one has to select a set of significant parameters which can be used for identification the attention decrease and onset of micro-sleep.

Among such parameters belong: the electro-magnetic activity of brain, frequency of breath, frequency of hearth beats, eye movements, skin resistance, face grimaces etc. All these parameters have their specific

advantages and also disadvantages.

We have chosen the EEG activity as the dominant significant parameter, because this is probably the only one, from which the almost immediate and reliable information about the brain function can be analyzed (similar information can be of course obtained from brain activity magnetic measurements, however their technical realization is much more difficult).

The analysis of EEG signals is in the focus of interest of many researchers, not only from the area of neurology, but also from electrical, control, signal engineering and mathematics.

The EEG measurements can be realized in laboratory quite well and also we got considerably good results. The preliminary results with EEG measurements in the moving vehicle are also quite promising. There is a good hope that in not far future such EEG measurements in moving objects could be realized in real praxis.

After information mining from many EEG time-series recorded on several tenths of probands in our laboratory we can analyze the procedure of the respective operator vigilance decrease and attention degradation almost immediately and with quite acceptable reliability.

As basis for such information mining we have used certain relations, which can be found among some components of EEG signal.

Here one has of course to take into account that the time-series representing the sampled EEG signal are in principle of the quasi-periodic and quasi-stationary character.

Therefore, the classical concept of spectrum for such time-series does not exist.

The standard methods for spectral analysis, based on Fourier decomposition into sum of periodical functions do not give accurate, representative and replicable results.

Nevertheless, they are widely used for such purpose up to now, though their results are of limited use only.

To avoid the eventual misunderstanding, we shall denote such results further on as pseudo-spectra.

Careful analyses based on long-time experience of skilled human expert allow mine from the sets of data, obtained by such pseudo-spectral analyses a lot of useful information and knowledge.

However, there is a good hope that another non-spectral approaches to quasi-periodical and quasi stationary time-series will lead to development of more accurate and sharp analytical tool than, which can be reached by pseudo-spectral methods.

Nevertheless, on the base of results derived up to now from up to now made analyses of above mentioned pseudo-spectral measurements

compared with

the in-time made measurements of the particular proband reaction-time RT and the correctness of his/her response on presented stimulus one can find the dependences like that shown schematically in Fig. 4.

Fig. 4: Example of a typical relation among pseudo-spectral components, and reaction time RT

However, the presented relation has still to be taken as example only, nevertheless it was found as result of carefully measurement. The reason for this is in the very high individuality of the EEG signal structure and distribution. Up to now we have at disposal the set of such complete measurements realized on about 30 human subjects. This seems to be enough for formulation of a hypothesis that from knowledge of the pseudo-spectral components and from the prediction of their values for satisfactory prediction horizon the estimation of actual and expected attention level of particular human subject, interacting with artificial system will be possible.

A proposed block structure of such micro-sleep warning system is presented in the next Fig.5

For reliable function of such warning systém we need to store a huge amount of data concerning the attention and micro-sleep parameters (marker) of very many human subjests. This is extected to be realized by creation of the Micro-Sleep Base (MSB) as a part of the international Global Neuroinformatic Net, organized by the Global Science Forum OECD.

The basic structure of MSB is shown in Fig.6.

Fig. 6. The basic structure of MSB operation flow for the phase of system MSB content exploitation for other research purposes.

problems of attention detection and prediction for subjects interacting with transportation systems...

Documents

hisher attention

drivers attention

transportation systemsresults

transportation systemsprof

transportation systemscontent

artificial system

transportation systemsthe

main problems