Download - Enhancing the 3D Experience - Khronos
![Page 1: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/1.jpg)
© Copyright Khronos Group, 2011 - Page 1
Enhancing the 3D Experience
Erik Noreke Chair of OpenSL ES Working Group
Chair of OpenMAX AL Working Group
![Page 2: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/2.jpg)
© Copyright Khronos Group, 2011 - Page 2
The Total 3D Experience
• Consists of Visual and Auditory input
- Stimulate different senses
- Use different parts of the brain
• The more senses are stimulated, the
more real an experience feels
• The better the 3D experience, the
better the game
![Page 3: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/3.jpg)
© Copyright Khronos Group, 2011 - Page 3
Audio is Underrated
• Audio is probably the most under-rated
element in a game
• There are companies that value audio but,
- Generally speaking it does not get the respect it
deserves
• Audio is often considered just before you ship
• So, are the machines running?
![Page 4: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/4.jpg)
© Copyright Khronos Group, 2011 - Page 4
Ear Is Hard To Trick
• Audio Is Psychological
- It is hard to measure
• If you have bad audio, people notice
- They might not realize what it is - but what they
experience feels wrong
• Audio is less forgiving than video
- Skips, pauses, missing frames
• Audio synch needs to be within 100ms of
graphics
![Page 5: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/5.jpg)
© Copyright Khronos Group, 2011 - Page 5
3D Audio
• 3D audio is experiencing real world sound
- Location – Direction
- Size – Width and Height
- Speed – Doppler
- Power – Amplitude
![Page 6: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/6.jpg)
© Copyright Khronos Group, 2011 - Page 6
– The Easy Way to 3D Audio
• Create theater-quality audio experience
- In a mobile device!
• Profiles reduce application customization
- Applications can query available profiles
- Develop to a specific profile or profile
combination
• Full 3D audio functionality enhances any
gaming experience
- Perfect companion to OpenGL ES
• Designed for implementation by either a
hardware or software solution
- Unlike any other advanced audio API
![Page 7: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/7.jpg)
© Copyright Khronos Group, 2011 - Page 7
Designed with Audio Application Developers in Mind
• Full range of effects and controls – including advanced 3D
effects such as Doppler and virtualization
- Experience rich, enhanced sound from locations other than the
handset, even moving, for the ultimate multimedia experience
• Advanced MIDI
- Use the output of the MIDI engine as a 3D sound source, making
the ring tone appear to be coming from a different direction than
the music
• 3D Audio makes OpenSL ES the natural choice for any audio
application
- Better gaming experience; 3D audio for conferencing calls; more
vivid music experience
Doppler
MIDI messages
Preset Reverb
EQ
Pitch
Buffer Queues
Rate
Metadata extraction
Environmental Reverb
Virtualization
Stereo widening
LED & Vibra
3D position
![Page 8: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/8.jpg)
© Copyright Khronos Group, 2011 - Page 8
• Smartphones are rapidly approaching 25% of
all handsets sold worldwide
- Steadily increasing market share
- Expected to exceed 50% by 2012
• Entertainment is a major selling feature of
smartphones
- Games, Multimedia
• Success in the smartphone market is
dependent on user experience
• Audio & Multimedia performance is quickly
becoming an important factor
- Market needs differentiating factor
- Form factor limits screen growth
- Important for operator services
A Standard for Tomorrow’s Smartphone User
![Page 9: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/9.jpg)
© Copyright Khronos Group, 2011 - Page 9
OpenSL ES Adoption – Standardizing Mobile Audio
IP Vendors
Platform Providers
Driving forces behind enabling high-performance audio in mobile devices
Handset Manufacturers Offers a full
Phone-Music-Game OpenSL ES solution
OpenSL ES now supported in Android 2.3 NDK
First adopter of OpenSL ES
![Page 10: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/10.jpg)
© Copyright Khronos Group, 2011 - Page 10
OpenSL ES Profiles
Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record
functionality, simple 2D audio games
Game-centric mobile devices Advanced MIDI functionality, sophisticated audio
capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc.
Music-centric mobile devices High quality audio, ability to support
multiple music audio codecs, audio streaming support
![Page 11: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/11.jpg)
© Copyright Khronos Group, 2011 - Page 11
– Object-Oriented Audio • OpenSL ES has an object-oriented programming model
- Simplifies common use cases – but also extensible
• Engine Objects are central to any OpenSL ES session
- Objects created using methods on the Engine Object interfaces
• OpenSL ES Objects enable PLAY and RECORD of audio
- Perform some operation on an input and emit the result as output
- Can handle almost any audio use case
• Objects have control interfaces
- For application
![Page 12: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/12.jpg)
© Copyright Khronos Group, 2011 - Page 12
The 1.1 Advantage
• Object-based audio for architectural
simplicity
- Reduces development time
• Native access for speed
- Necessary for high-performance audio
applications
• Advanced functionality for full
application control
- Everything from simple playback to 3D
audio
• Profiles reduce fragmentation
- Phone, Music, Game
• Advanced simplicity
- Reduces development time
- Facilitates porting
![Page 13: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/13.jpg)
© Copyright Khronos Group, 2011 - Page 13
• Same API regardless of underlying solution
- hardware accelerated
- software implementations
• Simplifies porting between platforms
- Spend application development time on application development, not porting
• Meets the demand for high-performance audio
- Opens up the market for advanced audio applications
- Provides mobile consumers with a home-theater-like immersive audio experience
• OpenSL ES does for audio what OpenGL ES does for graphics
- It brings your 3D application alive!
![Page 14: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/14.jpg)
© Copyright Khronos Group, 2011 - Page 14
Tips and Tricks
// Check for the Game profile
(*engine)->getInterface(engine, SL_IID_ENGINECAPABILITIES, &engineCaps);
(*engineCaps)->QuerySupportedProfiles(engineCaps, &profiles);
if(profiles && SL_PROFILES_GAME) GameProfile = TRUE;
![Page 15: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/15.jpg)
© Copyright Khronos Group, 2011 - Page 15
Matching the Audio to the Video
• Don’t use generic audio tracks
- Too recognizable by the user
- Modify or supplement to fit the specific
situation
• Matching the audio to the scene
enhances the 3D perception
- Stimulate more senses for a better
experience
• Sometimes no audio is best!
- Can be very effective
Fix layout
![Page 16: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/16.jpg)
© Copyright Khronos Group, 2011 - Page 16
Vary the Audio
• Don’t use the same track over and over
- Vary with a few extras
- Use double tracks of different lengths
• When using background ’Drone’ sound
add small point effects for detail
- Break the monotony
• Use different audio tracks for right and
left
- Certain background sounds are Right and
Left ear specific
Fix layout
![Page 17: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/17.jpg)
© Copyright Khronos Group, 2011 - Page 17
Equalizer of Death
• Lower the 3D effect and apply a low pass
filter - cut the treble - when the player’s
health is low
- This will lower the clarity and disorient the player
- Like blurring the screen does in the graphics
world
Output Mix Headset
Output
Device
3DGroup
Player
EqualizerItf
URI
3DSourceItf
3DDopplerItf
3DLocationItf
3DGroupItf
EffectSendItf
![Page 18: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/18.jpg)
© Copyright Khronos Group, 2011 - Page 18
Cinematic Stereo Widening
• Widen the sound during cinematic points
• Example: The player walks down a small
hallway into a big room
- Add suspense to a dramatic scene
• Start off with a narrow sound stage and then
widen it as the room opens up
Output Mix Headset
Output
Device
3DGroup
Player
EqualizerItf
VolumeItf
URI
PlayItf
EnvironmentalReverbItf
![Page 19: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/19.jpg)
© Copyright Khronos Group, 2011 - Page 19
Bigger Explosions
• Apply a bass boost effect to enhance the
explosion's boom
- Note that bass boost is different than just
increasing the bass
• Afterwards apply a low pass filter for a short
time to shock the player audibly
Output Mix Headset
Output
Device
3DGroup
Player
VolumeItf
URI
PlayItf
BassBoostItf
EqualizerItf
EnvironmentalReverbItf
![Page 20: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/20.jpg)
© Copyright Khronos Group, 2011 - Page 20
Engulfing Sounds
• Apply stereo widening to sounds that
encompass the player
- Crowd noise
- Fire
- Rain, Wind, Weather
- Traffic
Output Mix Headset
Output
Device
3DGroup
Player
VirtualizerItf
VolumeItf
URI
PlayItf
EqualizerItf
![Page 21: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/21.jpg)
© Copyright Khronos Group, 2011 - Page 21
Point sounds
• Create a single point sound moving in 3D
space to distract the player
- Fly buzzing around the player’s head
• Continually update position
• Vary the distance
Output Mix Headset
Output
Device
3DGroup
Player
EqualizerItf
VolumeItf
URI
PlayItf
3DSourceItf
3DDopplerItf
3DLocationItf
3DGroupItf
EffectSendItf
![Page 22: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/22.jpg)
© Copyright Khronos Group, 2011 - Page 22
EffectSendItf
The Classic
• Good Ol’ Fashion Shotgun
• The shotgun is everyone's favorite weapon
- Why? Because it goes BOOM!
Output Mix Headset
Output
Device Player
EnvironmentalReverbItf
VolumeItf
URI
PlayItf
![Page 23: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/23.jpg)
© Copyright Khronos Group, 2011 - Page 23
![Page 24: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/24.jpg)
© Copyright Khronos Group, 2011 - Page 24
![Page 25: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/25.jpg)
© Copyright Khronos Group, 2011 - Page 25
![Page 26: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/26.jpg)
© Copyright Khronos Group, 2011 - Page 26
![Page 27: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/27.jpg)
© Copyright Khronos Group, 2011 - Page 27
![Page 28: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/28.jpg)
© Copyright Khronos Group, 2011 - Page 28
![Page 29: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/29.jpg)
© Copyright Khronos Group, 2011 - Page 29
![Page 30: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/30.jpg)
© Copyright Khronos Group, 2011 - Page 30
![Page 31: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/31.jpg)
© Copyright Khronos Group, 2011 - Page 31
![Page 32: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/32.jpg)
© Copyright Khronos Group, 2011 - Page 32
![Page 33: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/33.jpg)
© Copyright Khronos Group, 2011 - Page 33
![Page 34: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/34.jpg)
© Copyright Khronos Group, 2011 - Page 34
![Page 35: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/35.jpg)
© Copyright Khronos Group, 2011 - Page 35
To Create the Best 3D Experience
• Plan the audio when creating the video
• Don’t overuse common audio effects
• Stimulate as much of the brain as possible
• Combine sounds & experiment
• Listen to real world examples
![Page 36: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/36.jpg)
© Copyright Khronos Group, 2011 - Page 36
Any Questions?
![Page 37: Enhancing the 3D Experience - Khronos](https://reader030.vdocument.in/reader030/viewer/2022012804/61bd2fd161276e740b10310c/html5/thumbnails/37.jpg)
© Copyright Khronos Group, 2011 - Page 37
Thank You