tablet pc platform advanced topics software design engineer/test lead toddland@microsoft.com

Tablet PC Platform Advanced Topics

Tablet PC Platform Advanced Topics

Software Design Engineer/Test LeadSoftware Design Engineer/Test LeadtoddLand@microsoft.comtoddLand@microsoft.com

TopicsTopics

1.1. Tablet Hardware API Tablet Hardware API

2.2. Overview of the Tablet SDKOverview of the Tablet SDKa.a. Ink, Stroke and related objectsInk, Stroke and related objects

b.b. RendererRenderer

3.3. Clipboard SupportClipboard Support

4.4. Ink Recognition Ink Recognition

5.5. Gesture SupportGesture Support

Tablet PC Platform Hardware APITablet PC Platform Hardware API

Examining Tablet HardwareExamining Tablet Hardware

Electromagnetic digitizersElectromagnetic digitizersHigh resolution, High throughputHigh resolution, High throughput

There are two types of digitizersThere are two types of digitizersIntegrated and Non-IntegratedIntegrated and Non-Integrated

Multiple Digitizers are supportedMultiple Digitizers are supportedThe mouse is also represented as a digitizerThe mouse is also represented as a digitizer

Digitizers support various capabilitiesDigitizers support various capabilities# of stylus buttons, pressure, angle or rotation # of stylus buttons, pressure, angle or rotation attributesattributes

Ink Stroke PacketData varies between digitizersInk Stroke PacketData varies between digitizers

Tablet Hardware APITablet Hardware API

TabletTabletObject representation of a digitizerObject representation of a digitizer

Query for digitizer’s supported properties Query for digitizer’s supported properties

Get property metricsGet property metricsE.g.: How many levels of pressure are E.g.: How many levels of pressure are supported?supported?

TabletsTabletsReturns collection of all connected Returns collection of all connected digitizersdigitizers

Including the mouseIncluding the mouse

Tablet Hardware API Cont.Tablet Hardware API Cont.

CursorCursorRepresents a stylusRepresents a stylus

Allows for access to stylus propertiesAllows for access to stylus propertiesInverted, Button state, NameInverted, Button state, Name

Available in all InkCollector/Overlay Available in all InkCollector/Overlay eventsevents

CursorInRange / CursorOut0fRangeCursorInRange / CursorOut0fRange

CursorDown / CursorUpCursorDown / CursorUp

CursorsCursorsCollection of Cursor objectsCollection of Cursor objects

Ink Overlay & Ink CollectorInk Overlay & Ink Collector

Main function is to collect inkMain function is to collect ink

Ink Collector provides core Ink Collector provides core functionalityfunctionality

Ink Overlay is an Ink Collector with the Ink Overlay is an Ink Collector with the addition of easy to use Editing Modes addition of easy to use Editing Modes (Ink, Select, Erase)(Ink, Select, Erase)

Contains an Ink objectContains an Ink object

Full set of event notificationsFull set of event notifications

InkCollectorInkCollectorTop level object for collecting ink Top level object for collecting ink

Attaches to any HWNDAttaches to any HWND

Handles input from all attached Tablets Handles input from all attached Tablets (digitizers) and Cursors (pens)(digitizers) and Cursors (pens)

Intercepts cursor movementsIntercepts cursor movements

CollectionMode propertyCollectionMode propertyInk OnlyInk Only

Gesture OnlyGesture Only

Ink and GestureInk and Gesture

InkOverlayInkOverlay

Superset of InkCollectorSuperset of InkCollector

Adds Editing ModesAdds Editing ModesInk – Real-time inking modeInk – Real-time inking mode

Select – Selection modeSelect – Selection mode

Delete – Eraser mode (both point and stroke)Delete – Eraser mode (both point and stroke)

Ink Collection EventsInk Collection EventsCursorInRange/CursorOutOfRangeCursorInRange/CursorOutOfRange

NewInAirPacketsNewInAirPackets

CursorDownCursorDown

NewPacketsNewPackets

StrokeStroke

GestureGesture

SystemGestureSystemGesture

Mouse EventsMouse Events

Ink ObjectInk Object

Primary Data Object in the Tablet Primary Data Object in the Tablet PlatformPlatform

Container for:Container for:Stroke dataStroke data

MetadataMetadata

Controls all means of persistenceControls all means of persistence

Extensible via ExtendedProperties Extensible via ExtendedProperties collectioncollection

Stroke ObjectStroke ObjectRepresents a single strokeRepresents a single stroke

Single pen down, pen move, pen up Single pen down, pen move, pen up sequencesequence

ExtendedPropertiesExtendedPropertiesAllows applications to access and store Allows applications to access and store custom datacustom data

DrawingAttributesDrawingAttributesControls the rendering of the strokeControls the rendering of the stroke

Rich API Rich API PacketData, BezierPoints, PacketData, BezierPoints,

PolylineCusps, SelfIntersectionsPolylineCusps, SelfIntersections

Strokes CollectionsStrokes CollectionsWhat are Stroke collections?What are Stroke collections?

Collections of references to strokesCollections of references to strokes

Not the stroke data itselfNot the stroke data itself

Important so you can pass a subset to the Important so you can pass a subset to the RecognizersRecognizers

Stroke Collections can share strokesStroke Collections can share strokesBoth Stroke collections include Stroke 3Both Stroke collections include Stroke 3

Strokes

Stroke (1)

Strokes

Stroke (2)Stroke (3)Stroke (4)Stroke (5)

DrawingAttributes ObjectDrawingAttributes Object

Controls the appearance of InkControls the appearance of InkColor, width, transparencyColor, width, transparency

Smoothing, anti-aliasingSmoothing, anti-aliasing

Height, widthHeight, width

Pen Tip: Ball or Rectangle Pen Tip: Ball or Rectangle

DefaultDrawingAttributes property DefaultDrawingAttributes property Specifies the DrawingAttributes for all CursorsSpecifies the DrawingAttributes for all Cursors

Can be applied to Can be applied to StrokeStroke

Strokes collectionStrokes collection

Cursor (Pen)Cursor (Pen)

Renderer ObjectRenderer Object

Two coordinate systems on a Tablet PCTwo coordinate systems on a Tablet PCDevice coordinatesDevice coordinates

Ink coordinates (HIMETRIC units)Ink coordinates (HIMETRIC units)

Renderer objectRenderer objectMap between coordinate systemsMap between coordinate systems

Draw methods for renderingDraw methods for rendering

Manipulation methodsManipulation methodsObject and View transforms supportedObject and View transforms supported

Helper methods: Scale, Move, RotateHelper methods: Scale, Move, Rotate

Clipboard SupportClipboard Support

Clipboard SupportClipboard SupportUse the Ink API to copy and pasteUse the Ink API to copy and paste

ClipboardCopyClipboardCopy

ClipboardPaste ClipboardPaste

Multiple formats supportedMultiple formats supportedISF – Ink Serialized FormatISF – Ink Serialized Format

HTMLHTML

Bitmap, Metafile Bitmap, Metafile

Applications interacting with the Applications interacting with the clipboard can choose the formatclipboard can choose the format

Ink RecognitionInk RecognitionInterpret pen movements and/or strokes asInterpret pen movements and/or strokes as

Text (Handwriting)Text (Handwriting)

GesturesGestures

Shapes and symbolsShapes and symbols

Language SupportLanguage Support

Synchronous and Asynchronous Synchronous and Asynchronous recognition supportedrecognition supported

Provides a full set of recognition resultsProvides a full set of recognition results

Extensibility architectureExtensibility architecture

Recognizer ContextRecognizer ContextObject for accessing handwriting Object for accessing handwriting recognizersrecognizers

Binds ink data to a particular language for Binds ink data to a particular language for recognitionrecognition

Performs synchronous or asynchronous Performs synchronous or asynchronous handwriting recognitionhandwriting recognition

Recognition results and alternates Recognition results and alternates returnedreturned

Attributes used to bias recognition resultsAttributes used to bias recognition results

RecognitionResultRecognitionResultRecognizers contain a set of Recognizers contain a set of structuresstructures

Each structure called a “lattice”Each structure called a “lattice”

TopString propertyTopString property

TopConfidence propertyTopConfidence property

TopAlternate propertyTopAlternate property

RecognitionAlternates objectRecognitionAlternates objectGetAlternatesFromSelection methodGetAlternatesFromSelection method

Access to alternate paths through the Access to alternate paths through the latticelattice

Gesture SupportGesture Support

GesturesGestures

Collected through InkCollector, Collected through InkCollector, InkOverlay, or RealTimeStylusInkOverlay, or RealTimeStylus

Two Types: System or ApplicationTwo Types: System or Application

System GesturesSystem GesturesMimic mouse eventsMimic mouse events

Cannot be disabledCannot be disabled

Application GesturesApplication GesturesCan be single or multi-strokeCan be single or multi-stroke

Can be turned on/off individuallyCan be turned on/off individually

System GesturesSystem Gestures

Available GesturesAvailable GesturesTap Tap

DoubleTapDoubleTap

RightTapRightTap

DragDrag

RightDragRightDrag

HoldEnterHoldEnter

HoverEnterHoverEnter

HoverLeaveHoverLeave

Coordinates are Window CoordinatesCoordinates are Window Coordinates

Can’t be cancelled Can’t be cancelled

Application GesturesApplication Gestures

CollectionMode determines supportCollectionMode determines supportInkOnlyInkOnly

InkAndGestureInkAndGesture

GestureOnlyGestureOnly

Single vs. Multi-Stroke GesturesSingle vs. Multi-Stroke Gestures

Inform your users about gesture Inform your users about gesture supportsupport

Good UI feedbackGood UI feedback

Consistent use (i.e. Scratchout)Consistent use (i.e. Scratchout)

GesturesGesturesFocus:Focus:•Scenario driven: don’t over Scenario driven: don’t over useuse

The Tablet and Mobile PC The Tablet and Mobile PC Developer CenterDeveloper Center

http://msdn.microsoft.com/mobilepc

Tablet PC Developer CenterTablet PC Developer Center

The latest technical articlesThe latest technical articles

DownloadsDownloads

Developer resourcesDeveloper resources

Regular columnsRegular columns

..

NewsgroupsNewsgroups

Featured partnersFeatured partners

and more……and more……

Attend a free chat or web castAttend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspxhttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asphttp://www.microsoft.com/usa/webcasts/default.asp

List of newsgroupsList of newsgroupshttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspxhttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspx

MS Community SitesMS Community Siteshttp://www.microsoft.com/communities/default.mspxhttp://www.microsoft.com/communities/default.mspx

Locate Local User Groups and Community SitesLocate Local User Groups and Community Siteshttp://www.microsoft.com/communities/usergroups/default.mspxhttp://www.microsoft.com/communities/usergroups/default.mspx http://www.microsoft.com/communities/related/default.mspxhttp://www.microsoft.com/communities/related/default.mspx

© 2005 Microsoft Corporation. All rights reserved.© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

AppendixAppendix

RealTimeStylus Plug-in InterfacesRealTimeStylus Plug-in Interfaces

RealTimeStylusRealTimeStylus

1.1. PerformancePerformanceDe-couples rendering and collection of inkDe-couples rendering and collection of ink

2.2. FlexibilityFlexibilityYou can control exactly how and when the You can control exactly how and when the ink is collected and renderedink is collected and rendered

3.3. ExtensibleExtensibleYou can create your own plug-ins to render You can create your own plug-ins to render and collect inkand collect ink

RealTimeStylusRealTimeStylus

1.1. PerformancePerformanceGives you the opportunity to ensure users Gives you the opportunity to ensure users see no lag when creating inksee no lag when creating ink

Collection of stylus input is moved off the Collection of stylus input is moved off the UI thread and is collected asynchronouslyUI thread and is collected asynchronously

Provides direct access to stylus input Provides direct access to stylus input eveneven before it is renderedbefore it is rendered

Queued events allow for separation of Real Queued events allow for separation of Real Time Collection thread and UI threadTime Collection thread and UI thread

Minimizes thread usage Minimizes thread usage (From 5 per instance to 1 (From 5 per instance to 1 per process)per process)

RealTimeStylusRealTimeStylusFocus:Focus:•No lag renderingNo lag rendering

RealTimeStylusRealTimeStylus

2.2. FlexibilityFlexibilityPlug-in based architecturePlug-in based architecture

You determine how you render and collect You determine how you render and collect ink based on which plug-ins you chooseink based on which plug-ins you choose

Plug-ins can be dynamically added, Plug-ins can be dynamically added, removed or re-orderedremoved or re-ordered

RealTimeStylusRealTimeStylus

3.3. ExtensibleExtensibleAllows for the creation of new Real Time Allows for the creation of new Real Time componentscomponents

Custom dynamic rendering, gesture Custom dynamic rendering, gesture recognition, disabled regions, etc.recognition, disabled regions, etc.

Customize your ink collection and Customize your ink collection and rendering to suit your needrendering to suit your need

Architecture (Overview)Architecture (Overview)

Pen Service

RealTimePlugIn(IStylusSyncPlugin)

RealTimeStylus

queue

RealTimePlugIn(IStylusSyncPlugin)

“Inking” Thread UI Thread

Ink CollectingObject

(IStylusAsyncPlugin)

storage

InkCollector

RealTimeStylus Plug-insRealTimeStylus Plug-ins

Object that implements a synchronous or Object that implements a synchronous or asynchronous interfaceasynchronous interface

SyncPlugin: fast custom renderingSyncPlugin: fast custom renderingAsyncPlugin: ink data collectionAsyncPlugin: ink data collection

Can manipulate packet dataCan manipulate packet dataRead, Modify, Add, or IgnoreRead, Modify, Add, or Ignore

Tablet OS includes two plug-insTablet OS includes two plug-insDynamic RenderingDynamic RenderingGesture Recognition Gesture Recognition

SDK includes two sample plug-insSDK includes two sample plug-insCustom RenderingCustom RenderingConstrained RegionConstrained Region

RealTimeStylus PluginRealTimeStylus PluginFocus:Focus:•Flexibility in rendering ink Flexibility in rendering ink •Flexibility in collecting stroke Flexibility in collecting stroke datadata

Journal Note ReaderJournal Note Reader

Journal Note ReaderJournal Note Reader

Provides Read-only access to Journal Provides Read-only access to Journal NotesNotes

Exposes all attributes in Journal filesExposes all attributes in Journal filesInk, Images, Title, Stationary, etc.Ink, Images, Title, Stationary, etc.

Enables your application to consume common Enables your application to consume common note filesnote files

JNT file is presented as an XML JNT file is presented as an XML documentdocument

Native and Managed versionsNative and Managed versions

DDownloadownload as an SDK update as an SDK update

Journal Reader with Ink Journal Reader with Ink DividerDividerFocus:Focus:•Easy to consume Journal filesEasy to consume Journal files