enabling closed-sourcepeople.duke.edu/~djzielin/slides/searis_2014_intercept_slides.pdf · hand-off...
TRANSCRIPT
![Page 1: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/1.jpg)
Enabling Closed-Source Applications for Virtual Reality via
OpenGL Intercept Techniques
David J. Zielinski (Duke University) Ryan P. McMahan (The University of Texas at Dallas)
Solaiman Shokur and Edgard Morya (International Institute for Neuroscience of Natal)Regis Kopper (Duke University)
SEARIS 2014 Workshop. March 30th.Co-located with IEEE VR 2014
1
![Page 2: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/2.jpg)
Motivation - Part 1
● VR experiences often utilize special hardware.● So we need to use software that can utilize the
special hardware.
2
![Page 3: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/3.jpg)
Motivation - Part 2● To utilize VR, a user often needs to learn new
language/application● Barrier for adoption of VR.● Could we VR-enable the desktop application
the user is already familiar with? ○ even if it’s commercial/closed source?
Answer: OpenGL intercept-based techniques.3
![Page 4: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/4.jpg)
What is OpenGL?
MATLAB OpenGL Screen
fill3([0 0 0],[1 0 0],[1 0 1], glBegin(GL_TRIANGLES) glVertex3f(0,0,0); glVertex3f(1,0,0); glVertex3f(1,0,1);glEnd();
"OpenGL is an API (Application Progamming Interface) for rendering 2D and 3D computer graphics. The API is typically used to interact with a GPU, to achieve hardware-accelerated rendering. OpenGL was developed by Silicon Graphics Inc. in 1992." -Wikipedia
vertex = corner
4
![Page 5: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/5.jpg)
Intercept Case
Standard Case
Intercept Technique Example
Drawing Command (e.g. “draw a triangle”)
Graphics Driver (opengl32.dll)
Graphics Driver (opengl32.dll)
Intercept Driver (new opengl32.dll) Drawing Command (e.g.
“draw a triangle”)
5[place new dll in directory of executable]
![Page 6: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/6.jpg)
//inside our custom opengl32.dll
glColor3f(float r, float g, float b){
//can do some modification to color here
r=0.0;g=0.0;b=1.0; // make all colors blue!
real_glColor3f(r,g,b); //call real glColor3f}
Intercepted Function Example
6
![Page 7: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/7.jpg)
Technique #1: In-And-Out
Desktop Application
customopengl32.dll
graphics out
Plugin or Built-in Scripting input
device data in
input device data sender
real opengl32.dll
7
![Page 8: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/8.jpg)
Technique #2: Intercept Tags
--- scene ---draw_cube
<intercept_tag> draw_sphere<intercept_tag>
● Tags tell renderer to do extended functionality● Tags are actually specific geometry calls● Interpreted, not rendered
--- scene ---draw_cube
load special transform draw_sphereunload transform
--- scene ---draw_cube
draw_triangle(0,0..) draw_spheredraw_triangle(0,0..)
Conceptually Host Renders VR Application Processes
8
![Page 9: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/9.jpg)
Hand-Off Techniques
Hand-Off Manipulation Hand-Off Slice Plane
Visual Enhancements
Interpolated Animations Shader Insertion
Display Techniques
Level Of Detail Display Lists
<dl_list>objects
<dl_list>
Intercept Techniques Uses
9
![Page 10: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/10.jpg)
Technique #3: Driver Mediated Head Tracking
Desktop Application
customopengl32.dll
graphics
input device data
input device data sender
real opengl32.dll
10
![Page 11: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/11.jpg)
Case Study #1: MATLAB Technique: In-And-Out
"ML2VR: Providing MATLAB Users an Easy Transition to Virtual Reality and Immersive Interactivity". David J. Zielinski, Ryan P. McMahan, Wenjie Lu, Silvia Ferrari. Presented as a poster at IEEE VR 2013. Candidate for best poster award. Pages 83-84. Published in the conference proceedings. 2013. 11
![Page 12: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/12.jpg)
● Minimal additions to access data from VR devices
vr = vr_interface(ServerIP,ServerPort);
while 1 [event,btn,pos]=vr.get_button_event(); if event==1 && btn==0 % modify surface data here end clf; % clear screen surf(dataX,dataY,dataZ); % render surface drawnow; % flush end
MATLAB+ML2VR API Example
12
![Page 13: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/13.jpg)
Results:
● Artificial potential function used to plan the motion of the robot [1]● Potential function recalculates the best path to the target while adapting
to the dynamic addition and removal of obstacles.● We added in ML2VR functions to make the simulation interactive.● We can now explore how the algorithm reacts in immersive VR!
Key:
RobotObstacleGoal
Original Desktop Simulation Now in VR !
[1] G. Zhang, and S. Ferrari, “An adaptive artificial potential function approach for geometric sensing,” in IEEE Decision and Control, 2009, pp. 7903-7910. 13
![Page 14: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/14.jpg)
Case Study #2: MATLAB + Intercept TagsHand-off Dragging
intercept driverDesktop Application VR Application
draw_sphere
intercept driver<intercept tag> draw_sphere<intercept tag>
user presses and holds button
intercept driver<intercept tag> draw_sphere<intercept tag>
intercept driverdraw_sphere
user releases button
1.
2.
3.
4.
Desktop App adds Tags
Desktop App removes Tags
VR Application/Renderer see the tags and allows Virtual Hand interaction at high FPS
14
![Page 15: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/15.jpg)
Hand-Off Manipulation User Study● place a solid cube completely inside
a wireframe cube● varying sizes of wireframe cube ● within subjects design. hand-off
manipulation vs original virtual hand● 14 subjects
Results: Handoff faster, less clutches.Self reported less sickness, more presence, higher usability.
"Intercept Tags: Enhancing Intercept-based Systems". David J. Zielinski, Regis Kopper, Ryan P. McMahan, Wenjie Lu, Silvia Ferrari. VRST 2013. Nanyang Technological University, Singapore. October 6th-8th. Published in the conference proceedings. 2013.
15
![Page 16: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/16.jpg)
Case Study #3:● Using MotionBuilder for BCI Experiments.● Can we VR-Enable MotionBuilder for the
Oculus Rift?
?
16
![Page 17: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/17.jpg)
Background: Brain Computer Interface (BCI)
● Real-time decoding of brain activity● Invasive or Non-Invasive
We believe that VR can be used to train subjects with reduced mobility to utilize a BCI to control physical devices (e.g., wheel chairs and exoskeletons) in a safe environment .
17
![Page 18: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/18.jpg)
Background: MotionBuilder● Commercial motion capture and
animation software by Autodesk.● Normally used as part of larger
workflow.
● Has high quality real-time inverse kinematics, to modify character animations.
● However, can create interactive experiences via its constraints relations system, python scripting, and a C++ software development kit (SDK).
18
![Page 19: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/19.jpg)
MotionBuilder + Oculus: In-And-Out
● Existing MotionBuilder view mode Freeview (Parallel), aka side by side is close to what we need.
● In profiler disable frustrum culling.
● MotionBuilder user must add in camera control.
19
![Page 20: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/20.jpg)
Issues Overriding Projection Matrix
X
Y
A B
● When anti-aliasing is enabled in MB, need to extract the x and y shift, and put them into our new projection matrix.
● We can also extract the existing clip planes by looking at values A and B○ near clip = B/(A-1)○ far clip = B/(A+1)
20
![Page 21: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/21.jpg)
MB + Oculus: Driver Mediated Head Tracking● No extra plugins or
constraints needed for MotionBuilder.
● Latency seemed less. How can we test this?
21
![Page 22: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/22.jpg)
Issues Overriding Model Matrix
● The MotionBuilder “Live Shadow”, no longer works.
● Solution: override glMultMatrixd○ examine matrix being multiplied○ does it match the original (non-modified) model
view matrix?○ if yes - override it.
22
![Page 23: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/23.jpg)
New: Position Tracking
Precision Position Tracker (PPT) - IR
PPT Studio
VRPN Server
VRPN to UDP23
![Page 24: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/24.jpg)
Results: Success!● MotionBuilder ● Oculus Rift● Orientation from Rift● Position from PPT
24
![Page 25: Enabling Closed-Sourcepeople.duke.edu/~djzielin/slides/SEARIS_2014_Intercept_Slides.pdf · Hand-Off Manipulation User Study place a solid cube completely inside a wireframe cube varying](https://reader036.vdocument.in/reader036/viewer/2022081614/5fc3144d66b7485f18692e76/html5/thumbnails/25.jpg)
Thank You!Questions?http://virtualreality.duke.eduhttp://utdallas.edu/~rpm120130/fivelab/
25