team iralar breanna heidenburg -- michael lenisa -- daniel wentzel advisor: dr. malinowski
TRANSCRIPT
![Page 1: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/1.jpg)
Team IRALARBreanna Heidenburg -- Michael Lenisa -- Daniel Wentzel
Advisor: Dr. Malinowski
![Page 2: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/2.jpg)
The Project◦ Why is it important
The Goals◦ System breakdown
Image recognition Point transformation User Interface
The Results
![Page 3: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/3.jpg)
What is our project?
![Page 4: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/4.jpg)
Track a user’s eye and use the information to control a computer cursor
![Page 5: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/5.jpg)
Enhances Human Computer Interaction
◦Speed of use◦Hands-free use
![Page 6: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/6.jpg)
3 Part System◦ Image Processing Application◦ Calibration and Mapping system◦ GUI designed for gaze-based interaction
Systems developed concurrently and independently
Separate Applications at run-time
![Page 7: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/7.jpg)
Hardware and Image Processing Application
![Page 8: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/8.jpg)
Hardware◦ Camera
QuickCam Pro for Notebooks Visible Spectrum Camera
◦ Polarizer Tiffen 25mm polarizing filter Removes glare from eye reflections
◦ Lighting Diffuse LEDs Slightly distracting to the user, but necessary to provide
light for the camera
![Page 9: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/9.jpg)
LitEye LE-500◦ High resolution (SVGA)◦ Color Display◦ Translucent or opaque operation◦ Stationary relative to user’s eye
![Page 10: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/10.jpg)
Real time pupil tracking system◦ Developed in C using OpenCV image processing libraries◦ Traditional image processing and blob tracking
Capabilities◦ Locate and determine center of pupil in image◦ Low light and high reflection environments◦ All eye colors◦ Data logging and static test modes◦ Packaged into self contained Windows installer for easy
deployment onto any computer
![Page 11: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/11.jpg)
The Process◦ Capture Image◦ Extract Red Channel◦ Smooth image◦ Apply binary threshold◦ Locate blobs◦ Reject false positives◦ Determine center of pupil blob◦ Adapt threshold◦ Repeat
![Page 12: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/12.jpg)
Summary◦The Good
Dynamically adapts to changing lighting conditions and eye types
Maintains performance in low-light and specularly noisy conditions
◦The Bad Still relies on Logitech camera drivers Extreme reflections still cause problems
![Page 13: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/13.jpg)
Raw Capture
Pre-Processed Image
Completed Recognition
![Page 14: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/14.jpg)
Examples of performance in poor conditions
Low Light Difficult False Positive
![Page 15: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/15.jpg)
Calibration and point mapping
![Page 16: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/16.jpg)
System for mapping the location of the center of the pupil to a pixel on a computer screen
Reasons◦ Geometry
The eye is not flat but a screen is not
◦ User Customization All eyes are different Everyone wears the HMD differently
◦ User Training Calibration system also acts as a quick tutorial
![Page 17: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/17.jpg)
3 dimensional best fit plane◦ Currently using a 4th degree best fit
Xpix = A1 + Xeye*B1 + Yeye*C1
Ypix = A2 + Xeye*B2 + Yeye*C2
Calibration sub-system determines these coefficients
![Page 18: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/18.jpg)
How do we solve the problem?
◦ Multiple Variable Linear Regression – Least Squares
Y = B0 + B1x1 + … + Bkxk
◦ Uses matrix algebra to obtain a coefficient matrixB[] = (X’X)-1X’Y
![Page 19: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/19.jpg)
Results◦ pixelX = 224.9 + 4.8*eyeX – 2.9*eyeY◦ pixelY = 1612.5 + 2.0*eyeX – 5.6*eyeY
B[] B[] == ((XX’’XX))-1-1XX’’YY
![Page 20: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/20.jpg)
How do we click?◦ Monitor eye movements◦ Identify pauses
◦ When eye position is within a small percentage for a certain amount of time
◦ Currently set at 5 frames (~200 mS)◦ Generally, it takes 230 mS for a hand to click a
mouse.
![Page 21: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/21.jpg)
Communications and Custom GUI interface
![Page 22: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/22.jpg)
Why is it important?◦ Allows multiple processes to communicate◦ Allows relay of time sensitive information
UDP vs. TCP◦ UDP -> ‘Connectionless’◦ TCP -> ‘Connection oriented’
Multi-threading◦ Necessary for running multiple pieces of code in a
single process
![Page 23: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/23.jpg)
Multi-threading issues present
![Page 24: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/24.jpg)
Multi-threading◦ Issues with public variable usage◦ Solutions we are using:
Critical Section Raises thread priority (does not allow for interrupts)
Receive request for
data (over UDP)
Raise Thread priority
Lower Thread Priority
Reply to
request
Read Variable
![Page 25: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/25.jpg)
Custom GUI for Gaze Tracking Applications
Why?◦ Gaze tracking accuracy limited by inherent
properties of human vision system◦ Traditional GUI too small and intrusive for use
with transparent HMD◦ Demonstrate applications of gaze tracking
![Page 26: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/26.jpg)
Achieved via a combination of Hardware and Software
Software:WPF & C#◦ XAML
(eXtensible Application Markup Language)◦ Very similar to XML◦ Uses ‘code-behind’ in a similar style to javascript
‘code-behind’ is in C#
![Page 27: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/27.jpg)
Multiple applications have been created within the interface
Multiple screens for functionality testing◦ even games
Additional ability to minimize interface and interact with external applications is present
![Page 28: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/28.jpg)
Calibration Screen Main Screen
![Page 29: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/29.jpg)
What did our test results show?
![Page 30: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/30.jpg)
![Page 31: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/31.jpg)
![Page 32: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/32.jpg)
![Page 33: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/33.jpg)
![Page 34: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/34.jpg)
Who helped us out?
![Page 35: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/35.jpg)
Northrop Grumman LitEye
Dr. Malinowski and the EE faculty Mr. Mattus & Mr. Schmidt
![Page 36: Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski](https://reader035.vdocument.in/reader035/viewer/2022062217/5697bfc11a28abf838ca4707/html5/thumbnails/36.jpg)