michael bernstein principal software design engineer microsoft corporation
TRANSCRIPT
![Page 1: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/1.jpg)
The New COM API for Accessibility and Automation in Windows 7
Michael BernsteinPrincipal Software Design EngineerMicrosoft Corporation
![Page 2: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/2.jpg)
Programmatic UI Access: A Key Element of Accessibility
Enables developers to write code that:
Navigates between UI elements
Gathers information about the UI
Interacts with UI elements
Receives notifications when the UI changes
Accessibility
Programmatic Access
2
![Page 3: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/3.jpg)
UI Automation (UIA):Programmatic Access in Windows
Successor to Microsoft Active Accessibility (MSAA) Creates a universal object model for accessing UI Designed based on experience with MSAA usage Supported on Windows Vista, XP and Server 2003 UIA COM Client API introduced in Windows 7
3
UI Automation Core MSAA
UIA Managed Client API UIA Provider API UIA COM Client
API
![Page 4: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/4.jpg)
What’s new in Windows 7?
UIA Clients can now use COM instead of managed code
New Accessibility properties are available Native clients can register custom ‘proxies’
to make existing UI more accessible Developers can define custom Accessibility
properties and events beyond the official UIA list
![Page 5: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/5.jpg)
Inspecting UIA Properties
demo
![Page 6: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/6.jpg)
Context: Clients and Providers
Client-side API (C# or C++)
Client Provider
UIA Infrastructure
UIA
Provider-side interface definitions (C# or C++)
Test Automation and Assistive Technology Manufacturers
Application and Control Developers
![Page 7: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/7.jpg)
TAKING ADVANTAGE OF THE NEW COM CLIENT API
Client ProviderUIA
![Page 8: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/8.jpg)
SAMPLE: Querying a window’s password statusCComPtr<IUIAutomation> spAutomation;CComPtr<IUIAutomationElement> spElement;HRESULT hr;hr = spAutomation.CoCreateInstance(CLSID_CUIAutomation);if (SUCCEEDED(hr)){
hr = spAutomation->GetFocusedElement(&spElement);if (SUCCEEDED(hr)){
BOOL isPassword;spElement-
>get_CurrentIsPassword(&isPassword);}
}
![Page 9: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/9.jpg)
Automation Elements
Just as in the managed API, an automation element represents a piece of UI, regardless of underlying implementation First, create an instance of
CUIAutomation, the factory for the object model
Next, get an element: From an HWND: ElementFromHandle From a Point: ElementFromPoint From current focus: GetFocusedElement
Each method returns an IUIAutomationElement
![Page 10: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/10.jpg)
Basic Properties
IUIAutomationElement exposes properties that make sense for elements in general
Properties are exposed in COM manner and are consistent across all underlying UI
UIA Control Type
UI Framework Framework Specific
Property
UI Automati
onProperty
Button Windows Presentation Foundation
Content NameProperty
Button Win32 Caption NameProperty
Image Trident/HTML ALT NameProperty
![Page 11: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/11.jpg)
Three Key Properties
1. NameProperty String that a user would use to explain which control
was being referred to (Customer: Accessibility Tools)
2. ControlTypeProperty Primary identifier for what type of control the
automation element represents. (Customers: Both)
3. AutomationIdProperty String that uniquely identifies an element among its
siblings in the tree (Customer: Test Automation)
![Page 12: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/12.jpg)
Control Patterns
Control Patterns allow access to a control’s particular abilities InvokePattern:
‘Push Me’ TransformPattern:
‘Move Me’ ScrollPattern:
‘Scroll Me’ SelectionItemPattern:
‘Select Me’ Patterns can contain
properties, methods and events
ComboBox
Simple
ExpandCollaps
e
Value
![Page 13: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/13.jpg)
Find Requests
Find requests allow you to locate elements based on conditions Conditions are based on property matching They can include Boolean operators as well
Requests can be restricted by scope Find can return the first match or all
matches Faster alternative to manual searching
![Page 14: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/14.jpg)
Visiting the Relatives: TreeWalker
All UIA elements are organized into a tree
TreeWalkers allow navigation around the tree GetParentElement,
GetFirstChildElement,GetNextSiblingElement, etc.
Can filter by condition
List View
ListItem1 ListItem2 Scroll Bar
Up button
Down button
A sample control tree:
![Page 15: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/15.jpg)
Consuming Events
Enable clients to be notified when changes occur in the UI Client implements a listener interface Register your listener with CUIAutomation
Clients can scope what parts of automation tree to listen for events on
Applicable events are always associated with a reference to automation element where change occurred
http://msdn.microsoft.com/en-us/library/aa359583(VS.85).aspx
15
![Page 16: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/16.jpg)
DumpUIATree
sample
![Page 17: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/17.jpg)
Accelerate your access: CacheRequests
Define •Create a request and add entries for properties and patterns you want cached•Define filter and scope for caching
Query •Call the element’s BuildUpdatedCache method or “…BuildCache” methods•A new copy with cached data is returned in one chunk
Use •Use ‘Cached’ flavor of property accessors to read cached data or patterns•Children can be cached as well
![Page 18: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/18.jpg)
Revisiting DumpUIATree
sample
![Page 19: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/19.jpg)
NEW PROPERTIES AND PATTERNS
Client ProviderUIA
![Page 20: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/20.jpg)
Patterns for Control Virtualization
Allows controls to have more children than are exposed in Automation tree at one time Parent container permits searching for children
by property Children can be de-virtualized upon request
Useful for: Containers with massive numbers of children
![Page 21: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/21.jpg)
Other new properties and patterns
Name PurposeARIA Properties Expose ARIA properties for
browser elements
Provider Description Query for the name of the provider(s) for an element
Synchronized Input Pattern Wait for a UI element to receive mouse/kbd input
Legacy MSAA Pattern Access original MSAA properties for a provider
Menu mode start/end events Find out when an application enters/exits menu mode
![Page 22: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/22.jpg)
BUILDING GREAT UIA PROVIDERS
Client ProviderUIA
![Page 23: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/23.jpg)
When do you need a provider?
You don’t always need to write a provider. Customizing beyond your framework
Changing perceived properties Annotation API may be sufficient
Adding new abilities Full custom controls
Creating new frameworks Silverlight had to implement Accessibility
![Page 24: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/24.jpg)
Implementing the simple interface
Create a COM object that implements IRawElementProviderSimple Retrieve VARIANT properties
Implement the most important ones for your control, especially Name and ControlType
Retrieve a pattern implementation Connect your COM object to the outside
world through WM_GETOBJECT
![Page 25: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/25.jpg)
Implementing patterns
Control patterns are much like interfacesCan reside on same
object or different object from Simple interface
Implement patterns that reflect your control’s abilities
ComboBoxSimpleExpandCollapseInvokeTableTextToggleValueVirtualized
![Page 26: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/26.jpg)
A basic custom-control UIA provider
sample
http://msdn.microsoft.com/en-us/accessibility/cc307849.aspx
![Page 27: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/27.jpg)
Beyond Simple Elements
Representing children Free support for mirroring HWND tree When you have non-window children …
Implement Fragment for all nodes in your tree Implement FragmentRoot for the root of your tree
Firing Events If a user would notice a change in your control,
you need to fire an event You may check for listeners first
![Page 28: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/28.jpg)
Custom patterns and properties
Allows client and provider to exchange data beyond original Accessibility design
Custom work must be done on both sides Properties and events: business as usual Methods: extra marshalling work required
Client ProviderUIACustom dataCustom data
![Page 29: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/29.jpg)
Custom ‘proxy’ support
Enables a client to register a substitute provider for a UI that lacks a real provider Proxy implements UIA interfaces on behalf of
UI Useful for legacy code that cannot be
modified but must be made accessible
Client ProviderUIA
Custom Proxy Non-UIA channel
![Page 30: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/30.jpg)
WRAPPING IT UP
![Page 31: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/31.jpg)
Is this all of Accessibility?
Is your software usable with … Programmatic access? Keyboard-only access? High contrast / high DPI? Slow reaction times?
Accessibility is not a yes/no question: it is usability for a specific audience
http://msdn.microsoft.com/accessibility
![Page 32: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/32.jpg)
What about managed clients?
The new COM API was designed with .NET interoperability in mind Pro: Fastest performance for programmatic access Con: Imported COM interfaces are not identical with
older interfaces Con: Some Win32 controls have slight behavior
differences
UI Automation Core
UIA Managed Client API UIA Provider API UIA COM Client API
Your Client Code;TlbImport Wrapper
![Page 33: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/33.jpg)
Managed client option #2:Use the existing managed library
We continue to support the existing System.Windows.Automation namespace Pro: No change to existing code Con: May be slower when accessing MSAA applications Con: Cannot access new properties and patterns with
current version
UI Automation Core
UIA Managed Client API UIA Provider API UIA COM Client API
Your Client Code
![Page 34: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/34.jpg)
Call to Action
Use the new UI Automation Client API to interact with existing user interfaces
Create UI Automation providers for new custom controls and UI frameworks
Consider the Accessibility needs of your customers when designing applications
![Page 35: Michael Bernstein Principal Software Design Engineer Microsoft Corporation](https://reader033.vdocument.in/reader033/viewer/2022061508/56649e625503460f94b5ea07/html5/thumbnails/35.jpg)
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.