“Avalon”: Using UI Automation to Improve Application Testability

Robert SinclairGroup ManagerMicrosoft Corporationrobsi@microsoft.com

Session Code: CLI307

AgendaAgenda

What is UI Automation? Demo: Windows UI Automation in actionA comparison of approachesOverview of Windows UI AutomationSample CodeSecurity ConsiderationsKey Take-Aways

What is UI Automation? Demo: Windows UI Automation in actionA comparison of approachesOverview of Windows UI AutomationSample CodeSecurity ConsiderationsKey Take-Aways

Tools

Client Application Model

Avalon Windows Forms

Web & Service Application Model

ASP.NET / Indigo Win FSCompact

FrameworkYukon Mobile PC Optimized

Data Systems Application Model

Presentation Data

Mobile PC & Devices Application Model

Communication

Command Line

NT Service

DataSetDataSet

MappingMapping

ObjectSpacesObjectSpaces

ObjectSpaceObjectSpace

QueryQuery

SchemaSchema

ItemItem

RelationshipRelationship

MediaMedia

AudioAudio

VideoVideo

ImagesImages

System.MessagingSystem.Messaging System.DiscoverySystem.Discovery

System.DirectoryServicesSystem.DirectoryServices

System.RemotingSystem.Remoting

System.Runtime.RemotingSystem.Runtime.Remoting

ActiveDirectoryActiveDirectory

UddiUddi

System.Web.ServicesSystem.Web.Services

Web.ServiceWeb.Service

DescriptionDescription

DiscoveryDiscovery

ProtocolsProtocols

System.MessageBusSystem.MessageBus

TransportTransport

PortPort

ChannelChannel

ServiceService

QueueQueue

PubSubPubSub

RouterRouter

Fundamentals

System.ComponentModelSystem.ComponentModel

System.CodeDomSystem.CodeDom

System.ReflectionSystem.Reflection

System.EnterpriseServicesSystem.EnterpriseServices

System.TransactionsSystem.Transactions

Security

System.Windows.TrustManagementSystem.Windows.TrustManagement

System.Web.SecuritySystem.Web.Security

System.MessageBus.SecuritySystem.MessageBus.Security

AuthorizationAuthorization

AccessControlAccessControl

CredentialsCredentials

CryptographyCryptography

System.Web.ConfigurationSystem.Web.Configuration

System.MessageBus.ConfigurationSystem.MessageBus.Configuration

System.ConfigurationSystem.Configuration

System.ResourcesSystem.ResourcesSystem.ManagementSystem.Management

System.DeploymentSystem.Deployment

System.DiagnosticsSystem.Diagnostics

Configuration Deployment/Management

System.WindowsSystem.WindowsSystem.WindowsSystem.WindowsSystem.Windows.Form

sSystem.Windows.Forms

System.ConsoleSystem.Console

System.ServiceProcessSystem.ServiceProcess

System.Windows.FormsSystem.Windows.Forms

System.WebSystem.Web System.StorageSystem.Storage System.Data.SqlServerSystem.Data.SqlServer

AnimationAnimation

ControlsControls

ControlControl

DesignDesign

PanelPanel

ControlsControls

DialogsDialogs

SideBarSideBar

NotificationNotification

System.WindowsSystem.Windows

DocumentsDocuments

Text ElementText Element

ShapesShapes

ShapeShape

InkInk

UI ElementUI Element ExplorerExplorer MediaMedia

System.Windows.FormsSystem.Windows.Forms

FormsForms

ControlControl

Print DialogPrint Dialog

DesignDesign

System.Web.UISystem.Web.UI

PagePage

ControlControl

HtmlControlsHtmlControls

MobileControlsMobileControls

WebControlsWebControls

AdaptorsAdaptors

DesignDesign

System.CollectionsSystem.Collections

GenericGeneric

System.SearchSystem.Search

AnnotationsAnnotations

MonitoringMonitoring

LoggingLogging

RelevanceRelevance

System.DataSystem.Data

SqlClientSqlClient

SqlTypesSqlTypes

SqlXMLSqlXML

OdbcClientOdbcClient

OleDbClientOleDbClient

OracleClientOracleClient

CoreCore

ContactContact

LocationLocation

MessageMessage

DocumentDocument

EventEvent

System.StorageSystem.Storage

System.WebSystem.Web

PersonalizationPersonalization

CachingCaching

SessionStateSessionState

System.XmlSystem.Xml

SchemaSchema

SerializationSerialization

XpathXpath

QueryQuery

PermissionsPermissions

PolicyPolicy

PrincipalPrincipal

TokenToken

System.SecuritySystem.Security

System.CollaborationSystem.Collaboration

RealTimeEndpointRealTimeEndpoint

TransientDataSessionTransientDataSession

SignalingSessionSignalingSession

MediaMedia

ActivitiesActivities

HttpWebRequestHttpWebRequest

FtpWebListenerFtpWebListener

SslClientStreamSslClientStream

WebClientWebClient

System.NetSystem.Net

NetworkInformationNetworkInformation

SocketsSockets

CacheCache

System.WebSystem.Web

AdministrationAdministration

ManagementManagement

NavigationNavigation

Peer GroupPeer Group

PolicyPolicy

RecognitionRecognition

System.SpeechSystem.Speech

SynthesisSynthesis

UI AutomationSystem.Windows.AutomationSystem.Windows.Automation

System.TimersSystem.Timers

System.GlobalizationSystem.Globalization

System.SerializationSystem.Serialization

System.ThreadingSystem.Threading

System.TextSystem.Text

System.DesignSystem.Design

PortsPorts

InteropServicesInteropServices

System.RuntimeSystem.Runtime

System.IOSystem.IO

SerializationSerialization

CompilerServicesCompilerServices

Base & Application Services

System.Windows.AutomationSystem.Windows.Automation

ProviderProvider

InteropProviderInteropProvider

“UI Automation” defined“UI Automation” defined

Gather information about the UIDynamically discover UI structureExtract property information Receive event notifications when UI changesQuery an element for its behavior

Interact with UI elementsClick a button, scroll a list, move a window, etc.Inject keystrokes and mouse input

Gather information about the UIDynamically discover UI structureExtract property information Receive event notifications when UI changesQuery an element for its behavior

Interact with UI elementsClick a button, scroll a list, move a window, etc.Inject keystrokes and mouse input

– code that programmatically drives another application’s UI to yield a desired

result

Windows UI Automation in Action

Goals:Traverse the UI treeQuery a control for its functionalityInteract with the control

Application #2Application #1

Today’s Approach to AutomationToday’s Approach to Automation

Custom 1Custom 1 WinForms SDM Win32Win32 Custom 2Custom 2Text PanelWin32Win32

Automation UtilityAutomation Utility

Win32Messages &

APIs

Win32Messages &

APIs

SDMObjectModel

SDMObjectModel

Custom 1APIs

Custom 1APIs

……

……Text Object

ModelText Object

Model

Custom 2APIs

Custom 2APIs

WinFormsMethods

WinFormsMethods

Interacting with a ControlComparison of ApproachesInteracting with a ControlComparison of Approaches

Today – Invest in each control libraryUse each library’s proprietary APIs to:

Traverse its treeCollect UI informationManipulate the controls

At runtime:Test the control’s classnameCall your library or control-specific routines

Longhorn – Use Windows UI Automation

Traverse UI structure using a single treeBase your app logic on control behaviorsObtain UI Automation security permissions

Today – Invest in each control libraryUse each library’s proprietary APIs to:

Traverse its treeCollect UI informationManipulate the controls

At runtime:Test the control’s classnameCall your library or control-specific routines

Longhorn – Use Windows UI Automation

Traverse UI structure using a single treeBase your app logic on control behaviorsObtain UI Automation security permissions

UI Automation ClientsUI Automation Clients

UI Automation Providers

Windows UI Automation Core

Longhorn Model: UI AutomationLonghorn Model: UI Automation

Assistive TechnologyProducts

ScriptingUtilities

TestingFramework

“Avalon”Implementation

Control Proxy CustomImplementation

…InteropProvider…Provider

MSAvalon.Windows.Automation

Leg

acy C

on

trol

ISV

Ob

ject

Mod

el

Atu

om

ate

d T

est

…InteropProvider

Windows UI AutomationWindows UI AutomationAutomation framework built into Longhorn

Platform-level support for automating all UI elements

Avalon, WinForms, Win32, Visual Basic, etc.

Exposes a consistent object model for all controls

3rd party controls easily integrate into model

Security Model – client must be trustedLocale, machine, and resolution independent

Creates new opportunities for innovation in:

Automated UI TestingAssistive Technology ProductsCommand-and-Control Utilities

Automation framework built into Longhorn

Platform-level support for automating all UI elements

Avalon, WinForms, Win32, Visual Basic, etc.

Exposes a consistent object model for all controls

3rd party controls easily integrate into model

Security Model – client must be trustedLocale, machine, and resolution independent

Creates new opportunities for innovation in:

Automated UI TestingAssistive Technology ProductsCommand-and-Control Utilities

UI Automation OverviewUI Automation OverviewLogical Tree – structure of the UI

Stitches all UI trees into one coherent structureEliminates unnecessary elementsResembles the structure perceived by an end user

Properties – important UI informationName, Bounding Rectangle, Persistent ID, etc.

Events – notification of UI changes Window creation, change in focus or selection, etc

Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc.

Input – simple mouse and keyboard input

Logical Tree – structure of the UI Stitches all UI trees into one coherent structureEliminates unnecessary elementsResembles the structure perceived by an end user

Properties – important UI informationName, Bounding Rectangle, Persistent ID, etc.

Events – notification of UI changes Window creation, change in focus or selection, etc

Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc.

Input – simple mouse and keyboard input

UI Automation Control PatternsUI Automation Control Patterns

Mutually exclusive classes of control behaviorControl developers (providers) expose these patterns for new or existing controlsAutomation developers (clients) use patterns to

Discover what functionality a control offersGather pattern-specific property informationReceive pattern-specific eventsProgrammatically manipulate the control

Examples: Button: InvokeListBox: Scroll, SelectionComboBox: Scroll, Selection, ExpandCollapse

Mutually exclusive classes of control behaviorControl developers (providers) expose these patterns for new or existing controlsAutomation developers (clients) use patterns to

Discover what functionality a control offersGather pattern-specific property informationReceive pattern-specific eventsProgrammatically manipulate the control

Examples: Button: InvokeListBox: Scroll, SelectionComboBox: Scroll, Selection, ExpandCollapse

Expand Collapse Pattern

Goals:Traverse Logical Element treeQuery for supported patternsFind an element that supports ExpandCollapseManipulate the control

Security ModelSecurity Model

No default automation permissions UI Automation functionality is protected according to the following permissions:

Read – Navigate tree, get properties, receive eventsWrite – Call control pattern methodsInput – Call methods in the Input class

Access to Rights Managed requires additional permissions

No default automation permissions UI Automation functionality is protected according to the following permissions:

Read – Navigate tree, get properties, receive eventsWrite – Call control pattern methodsInput – Call methods in the Input class

Access to Rights Managed requires additional permissions

3 Key Take-aways3 Key Take-aways

1. Platform-level support for automating UI

2. Control independence Simplifies your client logicMeans less code to write and maintain

3. New opportunities Integrate additional control librariesExpose custom object models, properties, events, etc.Extend the UI Automation framework

1. Platform-level support for automating UI

2. Control independence Simplifies your client logicMeans less code to write and maintain

3. New opportunities Integrate additional control librariesExpose custom object models, properties, events, etc.Extend the UI Automation framework

Call to ActionCall to Action

Next 30 daysStudy System.Windows.Automation.*Spend a day playing with UI AutomationSubmit feedback on client or provider side

Longhorn Beta 1Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients

Longhorn RTMShip your fully-enabled UI Automation providers and clients!!

Next 30 daysStudy System.Windows.Automation.*Spend a day playing with UI AutomationSubmit feedback on client or provider side

Longhorn Beta 1Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients

Longhorn RTMShip your fully-enabled UI Automation providers and clients!!

Question & AnswerQuestion & Answer

Join me for further discussion in the Client Track LoungePlease fill out an on-line session evaluation on COMMNET

Thank You for Coming!

Join me for further discussion in the Client Track LoungePlease fill out an on-line session evaluation on COMMNET

Thank You for Coming!

© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.