hemant sonawane report

2

ABSTRACT: -

This project focuses on the research of human robot interaction through tele-operation with

the help of brain-computer interfaces (BCIs). To accomplish that, a working system has

been created based on off-the-shelf components. The experimental prototype uses the basic

movement operations and obstacle detection of a Lego Mindstorms NXT Robot. There are

two versions of this prototype, taking readings from the users' brain electrical activity in

real-time performance. The first version is Mindset, and is based on the attention levels of

the user as the robot accelerates or decelerates. The second version is Emotiv Epoc headset

which we actually used in our project. Its taking readings from 14 sensors, being able to

control fully the robot.

3

INTRODUCTION: -

Robotics is an increasingly developed field with elements from different scientific principles

including electronics, mechatronics, software engineering, physiology, human-machine

interaction and psychology. On the other hand, Brain-Computer Interface (BCI) technology

is a rapidly growing field of research with various applications in modern computer games,

prosthetics and control systems through to medical diagnostics. Although research on BCI

started during the 1970’s only the last few years we have been able to introduce brain-

computer interfacing to simple users mostly through serious games with commercial

products. In the past, a number of research groups have investigated various ways in

controlling robotic platforms mainly electrical wheelchairs or robotic arms for people

suffering from a diverse range of impairments. A very good example that combines both

platforms is the 9-degree-of-freedom (DoF) Wheelchair-Mounted Robotic Arm System

from the University of South Florida, Tampa. This kind of research had a profound impact

on the person’s independence; social-activity; and self-esteem. This technology launched for

commercial use with headsets that use simplified version of an electroencephalograph.

Examples include the mind controlled electric carts using NeuroSky’s technology with the

drivers wearing the Mindset headsets demonstrated on Courtesy of “The Doctors” show on

CBS. Another relevant project originated from the BCI research group of The University of

Essex and demonstrated the NXT robot as the main control object on Channel 4 TV in 2007.

Another example of a nonmedical EEG for controlling a remote device is the Emotiv Epoc

headset moving a robotic arm. Using a series of 16 electrodes, the EPOC can measure levels

of attention and facial expressions. Emotiv Epoc was launched in the same period as the

Neurosky’s Headset. The aim of this research is to introduce the field of human-robot

interaction using BCIs to tele-operate a robotic unit. In particular, this research focuses on

how a robotic unit operated through brainwaves can overcome the kinetic constraints of the

user. The prototypes use the basic movement operations and obstacle detection of a Lego

Mindstroms NXT Robot. The robot is controlled through a Neurosky Mindset for the first

prototype and Emotiv Epoc for the second taking readings from the users' brain electrical

activity in realtime performance. This activity produced by firing neurons of the brain is

recorded through electroencephalography (EEG) and it can be used as the basis for the

development of serious applications ranging from computer games to computer simulations.

4

SYSTEM ARCHITECTURE: -

For the development of the experimental prototype, a Desktop PC running Windows 8 was

used as the main development system and a Netbook PC running Windows 10 for

demonstration purposes. The basic hardware components are a Lego Mindstorms NXT

robot, the Neurosky Headset and the Emotiv Epoc headset as illustrated in Figure 1.

Figure: 1 System Architecture

The raw EEG has usually been described in terms of frequency bands: GAMMA greater

than 30 (Hz) BETA (13-30 Hz), ALPHA (8-12 Hz), THETA (4-8 Hz), and DELTA (less

than 4 Hz) [12]. The headset is calculating through a chip the Raw EEG to produce the

eSense Meters (Attention and Meditation) and output it to the PC [5]. On the other hand, the

Emotiv EPOC Headset is a neuro-signal acquisition and processing wireless neuroheadset

with 14 saline sensors being able not only to detect brain signals but also user’s facial

expressions. As for the robot, the Lego Mindstorms NXT kit has been used being a cross

platform programmable robotics unit released by Lego in 2006. The version that was used

includes a 32-bit AT91SAM7S256 (ARM7TDMI) main microprocessor at 48 MHz (256 KB

flash memory, 64 KB RAM). It includes 4 different types of sensors such as: servo motors,

touch, light, sound and ultrasonic.

Three identical servo motors that have built-in reduction gear assemblies with internal

optical rotary encoders that sense their rotations within one degree of accuracy. The touch

sensor detects whether it is currently pressed, has been bumped, or released. This has been

implemented in the prototype to allow for collision detection. The light sensor detects the

light level in one direction, and also includes an LED for illuminating an object. The light

sensor can sense reflected light values (using the built-in red LED), or ambient light on a

scale of ‘0’ to ‘100’, ‘100’ being very bright and ‘0’ being dark. If calibrated, the sensor can

also be used as a distance sensor. The light sensor is used as a height detector on this model

programming language that was used is JAVA with the use of Eclipse IDE. LeJOS was

installed to the NXT robot as the main OS so it can be programmed in JAVA. LeJOS is a

tiny Java Virtual Machine ported to the NXT brick in 2006. The main advantage of LeJOS

NXJ is that it provides the extensibility and adaptability of JAVA [13]. Furthermore, the

pccomm and blue cove libraries were used for the Bluetooth communication between the

computer and the robot. Finally, for the computer headset communication set the think gear

library has been used from the Mindset development tools and for the Emotiv Epoc headset

the Emotiv Development Kit.

5

BRAIN COMPUTER INTERFACE: -

Human-machine interface is one of the growing areas of research and development in recent

years. Most of all efforts through innovative interface has been paid to the design user-

friendly and ergonomic systems, such as voice recognition or virtual reality. Brain-

computer interface (BCI), sometimes called a direct neural interface or interfaces Machine

Brain, is a direct communication path between the human or animal brain (or brain cell

culture) and an external device. In one BCI device, either computers accept commands from

the brain or send signals to it. Both variants are also possible. Two communications are

permitted only in BCI brain and an external device to exchange information, but so far only

been successfully implanted in animals or humans. In the previous definition of the brain

means the brain or nervous system forms an organic life rather than the mind. Computer

refers to any processing or computing devices, from simple circuits after silicon chips

(including hypothetical future technologies such as quantum computing)

Figure: 2 BCI System

6

USED SOFTWARE AND EQUIPMENTS: -

1) Emotive EPOC: -

Emotiv EPOC is a wireless EEG system for research enabling entertainment, market

research and usability testing. Emotiv is used in more than 100 countries in which published

work in different areas of research on this topic. This device is intended for practical

research applications, providing access to high quality raw EEG data using software and

Enhanced Software development kit (SDK). Is it possible to conduct research Application

Programming Interface (API) and detection libraries that recognize facial expressions,

mental commands and performance metrics.

Figure: 3 Emotiv EPOC Headset

It is possible to connect to a PC, Tablet PC and smartphones through a protected 2.4 GHz

wireless network. HW key is compatible with USB and requires no custom drivers. It is

powered lithium battery that provides 12 hours of continuous operation. Emotiv EPOC has

14 channel EEG + 2 reference, which offer optimum location for accurate spatial resolution.

Channel names are based on the international 10-20 system electrode placement: AF3, F7,

F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4, with reference electrodes in the P3 / P4.

This device uses a sequential method sampling and one ADC. It works at a resolution of 14

bits per channel frequency response 0.16 - 43 Hz

Figure: 4 Location of Electrode

7

2) EmoComposer: -

The equipment includes Emotiv Epoc is EmoComposer version 1.0.0.5-PREMIUM which

serves as a simulator Emotiv Epoc. This program has been used in programming application

because it can send the same signals as the Emotiv Epoc.

Figure: 5 Location of Electrode

3) Emotive Control Panel: -

Emotiv Control Panel version 1.0.0.5-PREMIUM (build 116) is also part of Emotiv

Epoc. Unlike EmoComposeru that transmits signals, the program broadcast signals direct

and detects them. The program can communicate with the Emotiv Epoc and is

EmoComposerem

8

There are instructions on how to properly deploy Emotiv Epoc. This program also

differentiates strength Signal individual sensors by color: -

Black - no signal

Red - a very weak signal,

orange - weak signal

Yellow - decent signal,

Green - good signal

Figure: 6 Emotive Control Panel

4) LEGO Mindstorms: -

LEGO MINDSTORMS EV3 is a combination of universal LEGO building system and the

most advanced technologies. With this kit can build robots that can walk or travel, speak and

perform everything just can invade its builder. The kit includes 3D tutorials, step by step

how to build TRACK3R, R3PTAR, SPIK3R, EV3RSTORM and GRIPP3R. These robots

can be revived through a simple, intuitive, icon-based programming interface. The robot can

be controlled via remote control or through free application on your smartphone or tablet.

9

Figure: 7 Controller EV3

5) LEGO Digital Designer:-

Figure: 8 LEGO Digital Designer

10

SELECT THE APPROPRIATE SIGNAL: -

Emotiv Epoc can recognize these three types of signals:

Expressive,

affective,

cognitive.

1) Expressive signals

Expressive signals are signals emitted by the brain based on eye movement and expression

face. EmoComposer capable of sending and Emotiv Control Panel to detect these expressive

signals:

blink with both eyes (B link)

blink his left eye (Left Wink)

blink his right eye (Right Wink)

look to the left (Look Left)

look right (Look Right)

raising eyebrows (Raise Brow)

frown (Furrow Brow)

Smile (Smile)

grin (clench)

smirk right corner with the top (Right smirk)

grin to the left corner on the mountain (Left smirk)

laugh (Laugh).

From this menu were selected into all options to sneer at the right or

left corner up. The expression signals are suitable to control, because it is enough to perform

individual action on his face. A person does not need to concentrate on a specific image in

your mind.

Figure: 9 Expressive Signals

11

2) Affective Signals

Figure: 10 Affective Signals

Affective signals are signals emitted by the brain based on emotion or meditation.

EmoComposer capable of sending and Emotiv Control Panel to detect these affective

signals:

The abolition or excitement (E Xcitement)

of engagement / boring (Engagement / Boredom)

Meditation (Meditation)

frustration (Frustration).

3) Cognitive signals

In this case, the cognitive brain sends signals based on imagination. The user in your mind

will introduce cube, which is moving in a certain direction. Based on the direction of

movement EmoComposer capable of sending and Emotiv Control Panel to detect these

cognitive signals:

Push

Pull

Lift

Drop

Left

Right

Rotate Left

Rotate Right

Rotate Clockwise

Rotate Counter -clockwise

Rotate Forward

Rotate Reverse.

12

Figure: 11 Cognitive Signals

MODEL FOR TESTING: -

Figure: 12 Physical Test Model

13



14

Flow Chart: -

Figure: 14 Flow Chart

15

CODE: -

1 private void Pripoj_Emotiv_Click(object sender, EventArgs e)

2 {

3 if (this.Pripoj_Emotiv.Text == "Připojit")

4 {

5 this.Pripoj_Emotiv.Text = "Odpojit";

6 //engine.RemoteConnect("127.0.0.1", 1726);

7 engine.Connect();

8 if (Stav_pripojeni_Lego.Text == "Připojeno")

9 {

10 Uloz.Enabled = true;

11 }

12 Emotiv_obrazek.Visible = true;

13 }

14 else

15 {

16 this.Pripoj_Emotiv.Text = "Připojit";

17 engine.Disconnect();

18 if (this.Pripoj_Lego.Text == "Odpojit")

19 {

20 Motory_Off();

21 }

22 Uloz.Enabled = false;

23 Emotiv_obrazek.Visible = false;

24 }

25 Vlakno = new Thread(new ThreadStart(this.UpdateEmotiv));

26 Vlakno.Start();

27 }

Source Code 1: Connecting and Disconnecting EPOC

1 Brick<IRSensor, Sensor, Sensor, IRSensor> Lego; // senzory Lega

2 EmoState Stavy; // expresivní stavy vysílané z headsetu

3 EmoState StavyKognitiv; // kognitivní stavy vysílané z headsetu

Source Code 2: Sensors Lega and save current expressive and cognitive states

16

1 try

2 {

3 Lego = new Brick

4 <IRSensor,Sensor,Sensor,IRSensor>(CisloComu.Text);

5 //Pro připojení přes Bluetooth

6 Lego.Connection.Open();

7 Lego.Beep(1, 1000);

8 this.Stav_pripojeni_Lego.Text = "Připojeno";

9 this.Pripoj_Lego.Text = "Odpojit";

10 if (Stav_pripojeni.Text == "Připojeno")

11 {

12 Uloz.Enabled = true;

13 }

14 Lego_obrazek.Visible = true;

15 }

16 catch

17 {

18 this.Stav_pripojeni_Lego.Text = "Odpojeno";

19 Uloz.Enabled = false;

20 }

Source Code 3: Connecting the Lego Mindstorms EV3

1 private void VyberAkceRobota(int a)

2 {

3 switch (a)

4 {

5 case 1:

6 MotorB_Rychlost = Vpred_rychlost.Value;

7 MotorC_Rychlost = Vpred_rychlost.Value;

8 Vpred_aktivita.Value = Vpred_rychlost.Value;

9 timerRobot.Start();

10 break;

11

12 case 2:

13 MotorB_Rychlost = -Vzad_rychlost.Value;

14 MotorC_Rychlost = -Vzad_rychlost.Value;

15 Vzad_aktivita.Value = Vzad_rychlost.Value;


17 break;

18

19 case 3:

20 MotorB_Rychlost = ZataceniVpravo_rychlost.Value;

21 if (ZataceniVpravo_rychlost.Value <= 21)

22 {

23 MotorC_Rychlost = 2;

24 }

25 else

26 {

27 MotorC_Rychlost = ZataceniVpravo_rychlost.Value-20;

17

28 }

29 ZataceniVpravo_aktivita.Value =

30 ZataceniVpravo_rychlost.Value;


32 break;

33

34 case 4:

35 if (ZataceniVlevo_rychlost.Value <= 21)

36 {

37 MotorB_Rychlost = 2;

38 }

39 else

40 {

41 MotorB_Rychlost = ZataceniVlevo_rychlost.Value-20;

42 }

43 MotorC_Rychlost = ZataceniVlevo_rychlost.Value;

44 ZataceniVlevo_aktivita.Value =

45 ZataceniVlevo_rychlost.Value;


47 break;

48

49 case 5:

50 MotorB_Rychlost = -ZataceniVpravo_rychlost.Value;

51 if (ZataceniVpravo_rychlost.Value <= 21)

52 {

53 MotorC_Rychlost = -2;

54 }

55 else

56 {

57 MotorC_Rychlost = -(ZataceniVpravo_rychlost.Value-20);

58 }

59 CouvaniVpravo_aktivita.Value =

60 CouvaniVpravo_rychlost.Value;


62 break;

63

64 case 6:

65 if (ZataceniVlevo_rychlost.Value <= 21)

66 {

67 MotorB_Rychlost = -2;

68 }

69 else

70 {

71 MotorB_Rychlost = -(ZataceniVlevo_rychlost.Value-20);

72 }

73 MotorC_Rychlost = -ZataceniVlevo_rychlost.Value;

74 CouvaniVlevo_aktivita.Value =

75 CouvaniVlevo_rychlost.Value;


77 break;

18

78

79 default:

80 break;

81 }

82 }

Source Code 4: Switch Setting and Speed Engines

1 private void Uloz_Click(object sender, EventArgs e)

2 {

3 int[] Emo = new int[7];

4 float[] Num = new float[6];

5 int i, j;

6 bool k = false;

7

8 if (!SpousteniMotoru)

9 {

10 Emo[0] = NabidkaEmotiv1.SelectedIndex;

11


13 Num[0] = (float)FloatEmotiv2.Value;

14



17



20



23



26



29

30 if (Emo[0] == -1)

31 {

32 k = true;

33 MessageBox.Show("Není vybrána položka Zastavit!");

34 }

35

36 for (i = 0; i < 6; i++)

37 {

38 if (Emo[i] != -1)

39 {

40 for (j = i + 1; j < 7; j++)

41 {

42 if (Emo[i] == Emo[j])

43 {

19

44 k = true;

45 MessageBox.Show("Dva stejné ovládací

46 parametry!");

47 }

48 }

49 }

50 }

51

52 for (i = 0; i < 5; i++)

53 {

54 if (Num[i] < 0 || Num[i] > 1)

55 {

56 k = true;

57 MessageBox.Show("Síla není v rozmezí 0 - 1!");

58 }

59 }

60

61 if (!k)

62 {

63 timer.Start();

64 }

65 }

66 }

Source Code 5: Check selected items and error messages

1 switch (NabidkaEmotiv1.SelectedIndex)

2 {

3 case 0:

4 Zastaveni = Stavy.ExpressivIsBlink();

5 break;

6

7 case 1:

8 Zastaveni = Stavy.ExpressivIsLeftWink();

9 break;

10

11 case 2:

12 Zastaveni = Stavy.ExpressivIsRightWink();

13 break;

14

15 case 3:

16 Zastaveni = Stavy.ExpressivIsLookingLeft();

17 break;

18

19 case 4:

20 Zastaveni = Stavy.ExpressivIsLookingRight();

21 break;

22 } Source Code 6: Switch emotive states to stop

20

APPLICATIONS IN EDUCATION FIELD: -

Superior problems are able to develop pupils' interest in science and research in that area

and is important in the interest of students and professional activities. The output of this

work is described accurately and in school practice applicable procedure as EEG signals to

control external devices. Said the source code can be used whole or it can be tweak to suit

the needs of future users.

21

CONCLUSIONS AND FUTURE WORK

It was an interesting international experience to be part of this project and Tomas Bata

University (TBU).

Control external devices using ideas (and the facial muscles) looks like science fiction, but

progress in this area it is already so far that it is possible. It is good that researchers

interested about this topic because it can help people both handicap and move on control

various devices.

In this Project we describe the design management solution using an external robotic system

Brain signals scanned using an EEG device. The system is based on processing EEG signals

and convert the output to an external device understandable instructions.

This project presented an affordable human-robot interaction system based on tele-operation

with the help of brain-computer interfaces. The experimental prototype allows a robot to be

controlled successfully through users' brain electrical activity in real-time performance,

allowing for the generation of serious applications. The aim of this research focuses in the

ways that a robotic platform is controlled for future applications and studies the way the

human brain works (in terms of attention and medidation levels). This kind of research can

transform the way we live today enhancing human potential. Future work will include the

combination of different readings with the use of more sophisticated device equipped with

more electrodes. Moreover, a large sample of users will be tested to qualitatively measure

the effectiveness of the system. Furthermore, a third prototype will be developed with the

robot performing more functionality with the help of custom BCI devices [16]. Finally,

other physiological data will be extracted including body posture and facial expressions so

that it will be possible to determine user’s visual attention.

22

References: -

[1] PALANIAPPAN, Ramaswamy. Biological Signal Analysis [online]. Ramaswamy

Palaniappan & bookboon.com, 2010 [cit. 2016-01-11]. ISBN 978-87-7681-594-3

http://bookboon.com/cs/introduction-to-biological-signal-analysisebook

[2] Recording electrodes. In: Indiana.edu [online]. [cit. 2016-04-07]

http://cognitrn.psych.indiana.edu/busey/eegseminar/pdfs/EEGPrimerCh2.pdf

[3] Interpreting EEG Functional Brain Activity. In: FIU [online]. 2004 [cit. 2016-04-

https://cate.fiu.edu/sites/default/files/Publications/01266932.pdf

[4] KUMAR, Nitish. BRAIN COMPUTER INTERFACE. In: Digital Library

CUSAT [online]. 2008 [cit. 2016-01-13].

http://dspace.cusat.ac.in/jspui/bitstream/123456789/2621/1/Brain%20computer%20

interface.pdf

[5] Emotiv EPOC / EPOC+. In: EMOTIV [online]. 2014 [cit. 2016-02-01].

https://emotiv.com/epoc.php

[6] 31313 MINDSTORMS EV3. In: Lego Mindstorms [online]. 2016 [cit. 2016-02-01].

http://www.lego.com/cs-cz/mindstorms/products/31313-mindstormsev3

http://bookboon.com/cs/introduction-to-biological-signal-analysisebook

http://cognitrn.psych.indiana.edu/busey/eegseminar/pdfs/EEGPrimerCh2.pdf

https://cate.fiu.edu/sites/default/files/Publications/01266932.pdf

http://dspace.cusat.ac.in/jspui/bitstream/123456789/2621/1/Brain%20computer

http://dspace.cusat.ac.in/jspui/bitstream/123456789/2621/1/Brain%20computer

https://emotiv.com/epoc.php

http://www.lego.com/cs-cz/mindstorms/products/31313-mindstormsev3

hemant sonawane report

Documents