touch and gesture platform: windows runtime
TRANSCRIPT
![Page 1: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/1.jpg)
www.buildwindows.com
Build advanced touch apps in Windows 8
Reed TownsendProgram ManagerMicrosoft Corporation
APP-186T
![Page 2: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/2.jpg)
www.buildwindows.com
Agenda
• Windows Runtime Platform for touch, gestures, and devices
• Win32 platform for touch, gestures, devices, and more
• Q&A
You’ll leave with examples of how to• Add power and flexibility! Go beyond the touch
support in the HTML and XAML frameworks.• Get down to the metal! Build directly on
ICoreWindow if you choose to.
![Page 3: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/3.jpg)
www.buildwindows.com
Windows 8 is a touch-first OS, and as such,
we had to reimagine the touch platform and the input stack that supports it
![Page 4: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/4.jpg)
www.buildwindows.com
Central Touch Platform Concepts
• Unify touch, mouse, and pen into a single pointer input API
• Express the touch interaction language in the platform
• “Code for touch, get mouse and pen for free!”
• Performance, performance, performance!
![Page 5: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/5.jpg)
Touch and Gesture Platform:Windows Runtime
![Page 6: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/6.jpg)
www.buildwindows.com
Windows Runtime touch APIs let you go beyond what the HTML and XAML
frameworks provide by default
![Page 7: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/7.jpg)
www.buildwindows.com
Touch and Gesture Support in FrameworksAPI surface Metro style app
using HTMLMetro style app using XAML
ICoreWindow Windows Runtime
Pointer events Pointer events Pointer events with PointerPoint
Pointer events with PointerPoint
PointerPoint
Gesture events Gesture events Gesture events GestureRecognizer
Device APIs PointerDevice & Capabilities
Targeting Baked into framework and controls
Baked into framework and controls
TouchHitTesting event
* Plus some Win32 touch APIs in the Metro SDK
![Page 8: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/8.jpg)
www.buildwindows.com
GestureRecognizer Sample
Built using WinRT and ICoreWindowC++ projection
demo
![Page 9: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/9.jpg)
www.buildwindows.com
PointerPoint
• How to get touch data? Use PointerPoint!• Provides raw touch, mouse, and pen data and is the
input for GestureRecognizer
• Pointer unifies all “pointing” devices into a single, consistent set of interfaces and events
• Makes it easy to “code for touch and get mouse and pen for free”
• If you want input-specific UX, there are properties in pointer to support that too
![Page 10: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/10.jpg)
Handling Pointer Eventsvoid GestureRecognizerSample::OnPointerPressed( ... ){ // Hit testing: find the object under the pointer ...
if (iObject != -1) { // Assign the pointer to the object found in hit testing _gestureRecognizers[iObject]->ProcessDownEvent(pointerPoint); } else { // No objects found in hit testing. Assign this pointer to background _gestureRecognizers[_objects->Length]->ProcessDownEvent(pointerPoint); }}
![Page 11: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/11.jpg)
www.buildwindows.com
Code for touch, get mouse and pen for free
demo
![Page 12: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/12.jpg)
Handling Pointer Eventsvoid GestureRecognizerSample::OnPointerPressed( ... ){ // Hit testing: find the object under the pointer ...
if (iObject != -1) { // Assign the pointer to the object found in hit testing _gestureRecognizers[iObject]->ProcessDownEvent(pointerPoint); } else { // No objects found in hit testing. Assign this pointer to background _gestureRecognizers[_objects->Length]->ProcessDownEvent(pointerPoint); }}
![Page 13: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/13.jpg)
www.buildwindows.com
PointerPoint
• Tiered approach to properties• Basic, common data is on provided as properties on the
object directly• Extended, input-specific data hangs off the object as a
property bag• Many properties have smart defaults to make
coding against features that may or may not be there easier
• Can get/set raw HID usages as well
![Page 14: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/14.jpg)
PointerPoint – Raw HID Usagesvoid Sample::OnPointerPressed( _In_ Windows::UI::Core::CoreWindow^, _In_ Windows::UI::Core::PointerEventArgs^ args){ Windows::UI::Input::PointerPoint^ pointerPoint = args->CurrentPoint; bool fHasPressure; INT32 vendorValue;
// Does the packet have pressure info? pointerPoint->HasUsage(0x0D, 0x30, &fHasPressure);
// Get vendor defined usage from HID packet pointerPoint->GetUsageValue(0x06, 0x01, &vendorValue);}
![Page 15: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/15.jpg)
www.buildwindows.com
Gesture Recognition
demo
![Page 16: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/16.jpg)
www.buildwindows.com
GestureRecognizer
• Componentized gesture detection based off PointerPoints
• Allows opting in/out of gestures as well as configuring gestures themselves
• Allows multiple simultaneous gestures
• Going beyond frameworks:• Specific gestures and configurations• Using multiple GestureRecognizers together• Detecting multiple gestures within an element, e.g.
<canvas/>• Building on top of ICoreWindow directly
![Page 17: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/17.jpg)
GestureRecognizer: Setup and Configurationvoid DrawingObject::Attach(_In_ Windows::UI::Input::GestureRecognizer^ gestureRecognizer){ // Configure gesture recognizer gestureRecognizer->GestureSettings = Windows::UI::Input::GestureSettings::Tap | Windows::UI::Input::GestureSettings::Hold | Windows::UI::Input::GestureSettings::RightTap | Windows::UI::Input::GestureSettings::ManipulationTranslateX | Windows::UI::Input::GestureSettings::ManipulationTranslateY | Windows::UI::Input::GestureSettings::ManipulationTranslateInertia; ... // Register all the gesture event handlers _tokenTapped = gestureRecognizer->Tapped::add( ref new Windows::Foundation::TypedEventHandler< Windows::UI::Input::GestureRecognizer^, Windows::UI::Input::TappedEventArgs^>( this, &DrawingObject::OnTapped)); ...}
![Page 18: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/18.jpg)
GestureRecognizer: Feeding Pointer Datavoid GestureRecognizerSample::OnPointerMoved( _In_ Windows::UI::Core::CoreWindow^, _In_ Windows::UI::Core::PointerEventArgs^ args){ Windows::Foundation::Collections::IVector<Windows::UI::Input::PointerPoint^>^ pointerPoints = args->GetIntermediatePoints();
for (unsigned int i=0; i<_gestureRecognizers->Length; ++i) { _gestureRecognizers[i]->ProcessMoveEvents(pointerPoints); }
// Render the latest position OnRender();}
![Page 19: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/19.jpg)
GestureRecognizer: Handling an Eventvoid DrawingObject::OnTapped( _In_ Windows::UI::Input::GestureRecognizer^, _In_ Windows::UI::Input::TappedEventArgs^){ ++_color; if (_color >= Color::MaxCount) { _color = Color::First; }
_parent->RequestRedraw(false);}
![Page 20: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/20.jpg)
www.buildwindows.com
GestureRecognizer
• “Code for touch, get mouse and pen for free” is supported in gesture recognition too
• RightTap gesture is the set of all interactions that are intended to be “right taps”• Mouse right click, touch press and hold, pen press and
hold, and pen barrel button tap
![Page 21: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/21.jpg)
GestureRecognizer: RightTap Gesturevoid DrawingObject::Attach( ... ){ // Configure gesture recognizer gestureRecognizer->GestureSettings = Windows::UI::Input::GestureSettings::Tap | Windows::UI::Input::GestureSettings::Hold | Windows::UI::Input::GestureSettings::RightTap | ...}
void DrawingObject::OnRightTapped( _In_ Windows::UI::Input::GestureRecognizer^, _In_ Windows::UI::Input::RightTappedEventArgs^){ Initialize(_initColor, _initX, _initY, _initDX, _initDY); _parent->RequestRedraw(false);}
![Page 22: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/22.jpg)
www.buildwindows.com
Device Support in the SampleGesture Sample
demo
![Page 23: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/23.jpg)
www.buildwindows.com
PointerDevice
• Query the set of pointer devices on the system and get their properties:• Type, max contacts, device rects, etc
• Get supported HID usages for the device
![Page 24: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/24.jpg)
PointerDevice: Device Propertiesvoid BackgroundObject::GetDeviceDescription(){ Windows::Devices::Input::PointerDeviceType type = _pointerDevice->PointerDeviceType; boolean external = !_pointerDevice->IsIntegrated; UINT32 maxContacts = _pointerDevice->MaxContacts; Windows::Foundation::Rect rcPhysical = _pointerDevice->PhysicalDeviceRect; Windows::Foundation::Rect rcScreen = _pointerDevice->ScreenRect;
// Build device description string ...}
![Page 25: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/25.jpg)
PointerDevice: List of HID usagesvoid Sample::GetDeviceSupportedUsages(UINT 32 pid){ Windows::Devices::Input::PointerDevice device; IVectorView<PointerDeviceUsage> usages;
Windows::Devices::Input::PointerDevice::GetPointerDevice(pid, &device); usages = device->SupportedUsages; ...
for (int i = 0; i < usages.length; i++) { UINT32 HIDUsagePage = usages[i].UsagePage; UINT32 HIDUsage = usages[i].Usage; ... }}
![Page 26: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/26.jpg)
www.buildwindows.com
Other Device Goodies
• Touch/Pen/MouseCapabilities – Get quick access to the system’s input properties
![Page 27: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/27.jpg)
Hardware Capabilities Helpersbool Sample::IsMultiplayerSupported(){ PenCapabilities ^pPenCapabilities = ref new PenCapabilities(); bool PenPresent = pPenCapabilities->PenPresent;
TouchCapabilities ^pTouchCapabilities = ref new TouchCapabilities(); UINT32 TotalContacts = pTouchCapabilities->Contacts; return PenPresent && (TotalContacts >= 8);}
![Page 28: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/28.jpg)
www.buildwindows.com
Targeting
• Targeting uses touch contact geometry to make smart decisions about what you intended to tap
• Using ICoreWindow’s TouchHitTesting event, you can control how targeting behaves
![Page 29: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/29.jpg)
Targeting: Handling OnTouchHitTestingvoid CProgram::OnTouchHitTesting(_In_ Windows::UI::Core::CoreWindow^ /*sender*/, _In_ Windows::UI::Core::TouchHitTestingEventArgs^ touchHitTestingArgs){ // Current and best touch hit testing results Windows::UI::Core::CoreProximityEvaluation bestResult; Windows::UI::Core::CoreProximityEvaluation currentResult;
// Initialize best result and best touch hit testing target from input arguments bestResult = touchHitTestingArgs->ProximityEvaluation; bestResult.AdjustedPoint = touchHitTestingArgs->Point;
// Use appropriate logic to find best target – see ShapesPuzzle sample ...
// Set best target touchHitTestingArgs->ProximityEvaluation = bestResult;
// Event is handled touchHitTestingArgs->Handled = true;}
![Page 30: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/30.jpg)
Touch and Gesture Platform:Win32
![Page 31: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/31.jpg)
www.buildwindows.com
For desktop developers, Win32 support mirrors Windows Runtime Platform
support and lets you build great touch apps
![Page 32: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/32.jpg)
www.buildwindows.com
WM_POINTER, InteractionContexts, and Devices• WM_POINTER• Win32 representation of the pointer concept• Can be used in either a per-pointer or frame-based
manner• InteractionContext• Mirrors GestureRecognizer and provides componentized
gesture recognition• Pointer device APIs• Query the set of pointer devices on the system and get
their properties, supported HID usages, etc• Register for device arrival/departure notifications*
* Available in the Metro SDK as well for Metro style apps
![Page 33: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/33.jpg)
www.buildwindows.com
Other Win32 Goodies
• Touch targeting• Framework or custom control developers can influence
touch targeting• Touch visual feedback• Configure touch contact and gesture feedback
• Touch injection• Lets you inject touch input similarly to SendInput()
• Input source identification• Helps identify source devices (e.g. touch) when relying
on legacy mouse messages
![Page 34: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/34.jpg)
Q&A
![Page 35: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/35.jpg)
www.buildwindows.com
Recap
• Start building Metro style apps with HTML and XAML!
• When you need richer, more flexible control of touch, mix in the Windows Runtime touch and gesture APIs
• If you want to get down to the metal, use ICoreWindow and the Windows Runtime touch and gesture APIs
![Page 36: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/36.jpg)
www.buildwindows.com
Related sessions
• [APP-185T] Make great Metro style apps that are touch-optimized using HTML5
• [APP-503T] Make great touch apps using XAML
• [APP-162T] Building high performance Metro style apps using HTML5
• [PLAT-754T] From touch to gamepads: master player input in your Metro style game
• [PLAT-874T] Lap around the Windows Runtime
• [APP-391T] Designing Metro style apps that are touch-optimized
![Page 37: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/37.jpg)
www.buildwindows.com
Further reading and documentation
• SDK samples: • GestureRecognizer, GestureRecognizerSample,
ShapesPuzzle• Docs:• http://dev.windows.com
• Contact info – [email protected]
![Page 38: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/38.jpg)
www.buildwindows.com
• Feedback and questions http://forums.dev.windows.com
• Session feedbackhttp://bldw.in/SessionFeedback
thank you
![Page 39: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/39.jpg)
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
![Page 40: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/40.jpg)
www.buildwindows.com
Windows 8
Windows Core OS Services
JavaScript(Chakra)
CC++
C#VB
Metro style Apps
Communication
& Data
Application Model
Devices & Printing
WinRT APIsGraphics &
Media
XAML HTML / CSS
HTMLJavaScri
pt
CC++
C#VB
Desktop Apps
Win32
.NET / SL
Internet Explore
r
Syst
em
Serv
ices
Vie w
Mod
el
Con
trolle
r
Cor
e
![Page 41: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/41.jpg)
www.buildwindows.com
The Windows Runtime
• The new platform for building Windows Metro style apps
• Bring forward the full power of Windows• A family of integrated APIs with great tools
• Build upon that a new foundation for the future• Clean & simple• Flexible markup (HTML or XAML)• Flexible languages (JavaScript, C++, C#/VB, …)
![Page 42: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/42.jpg)
www.buildwindows.com
WM_POINTER vs WM_TOUCHAPI behavior WM_TOUCH WM_POINTER
Exclusivity of gestures and raw data WM_TOUCH and WM_GESTURE are exclusive
WM_POINTER can be used in conjunction with TIE
Input delays WM_TOUCH has palm rejection delays by default
WM_POINTER has no input delays
Multiple window support WM_TOUCH can only be delivered to one window. Contacts in other windows are dropped after the initial target window is determined.
WM_POINTER messages can be delivered to multiple windows simultaneously. Capture APIs provide additional functionality here as well.
Frame vs non-frame WM_TOUCH is frame-based WM_POINTER is non-frame-based by default, but developers can consume frames if they choose.
Rich data WM_TOUCH contains basic contact geometry (height and width only)
WM_POINTER supports more detailed contact geometry data as well as other richer information in the associated structs such as TOUCH_INFO
Tying touch to mouse data WM_TOUCH always promotes the primary contact to mouse
WM_POINTER does no mouse promotion by default. WM_POINTER has a separate method for controlling activation.
Capture support WM_TOUCH supports only it’s implicit capture model on a single window
WM_POINTER provides a capture API for controlling how capture works for contacts and windows
Activation support WM_TOUCH relies on the mouse messages it promotes to control activation
WM_POINTER provides an activation API to control how activation works for a window
Pen and other pointing device input WM_TOUCH does not include pen input WM_POINTER can carry any pointing device’s input
![Page 43: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/43.jpg)
www.buildwindows.com
WM_POINTER
• Win32 representation of the pointer concept• Unifies pointing inputs into a single API• Provides raw pointer data and is the basis for the
platform
• Can be used in either a per-pointer or frame-based manner
• Retrieve pointer data with GetPointerInfo() and related functions
• Part of the massive touch input stack work for Windows 8
![Page 44: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/44.jpg)
www.buildwindows.com
InteractionContext
• Mirrors WinRT GestureRecognizer• Componentized gesture detection based off pointer data• Allows opting in/out of gestures as well as configuring
gestures themselves
• Create as many ICs as you want, configure them, feed them data, and handle event callbacks
![Page 45: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/45.jpg)
www.buildwindows.com
Pointer Device APIs
• Query the set of pointer devices on the system and get their properties:• Max contacts, device rects, etc
• Get supported HID usages for the device• Register for device arrival/departure notifications*
* Available in the Metro SDK as well for Metro style apps
![Page 46: Touch and Gesture Platform: Windows Runtime](https://reader030.vdocument.in/reader030/viewer/2022032612/56649f0b5503460f94c1eb83/html5/thumbnails/46.jpg)