Microsoft UI Automation
Encyclopedia
Microsoft
UI Automation (UIA) is an Application Programming Interface
(API) for User Interface
(UI) accessibility that is designed to help Assistive Technology
(AT) products interact with standard and custom UI elements of an application
(or the operating system
) as well as to access, identify, and manipulate an application's UI elements. AT products work with applications in order to provide better access for individuals who have physical or cognitive difficulties, impairments, or disabilities. Some examples of this are screen readers for users with limited sight, on-screen keyboards for users with limited physical access, or captions for users with limited hearing.
UIA is similar to Microsoft Active Accessibility
(MSAA) in that it provides a means for exposing and collecting information about user interface elements and controls to support user interface accessibility and software test automation. However, UIA is a newer technology that provides a much richer object model than MSAA, and is compatible with both Win32 and the .NET Framework
. UIA is designed so that it can be supported across platforms other than Microsoft Windows
. For controls and frameworks written from scratch, UIA is recommended today. While MSAA is handy for relatively simple controls, the technology doesn't support the complexity of modern user interfaces.
The current and latest specification of UIA is found as part of the Microsoft UI Automation Community Promise Specification
.
) framework.
UIA has been supported by in Windows XP
, Windows Vista
, Windows Server 2003
, and Windows Server 2008. UIA has been released in Windows 7 as part of the new Windows Automation API framework, where UIA is a subset of that framework.
UI Automation API or the native UI Automation API. The native API is prominent between the two.
The managed UI Automation API is a part of the overall .NET Framework. It was included starting in version 3.0 of the .NET Framework.
The native UI Automation API (provider) is included as part of the Windows Vista and Windows Server 2008 SDK
. The native UI Automation API is also distributed with the .NET Framework. The Windows Automation API 3.0, which includes managed and native UI Automation API as well as MSAA 3.0, is included in Windows 7 and available for Windows XP, Windows Vista, and Windows Server 2003 and 2008.
The Microsoft Windows implementation of UIA features COM
-based interfaces and managed interfaces that are introduced with Microsoft .NET Framework. An API reference is available for both managed and unmanaged code.
With UIA, the UIA Core component (UIAutomationCore.dll) is loaded into the processes of both the accessibility
tools and applications. This component manages cross-process communication, and it also provides higher-level services, such as searching for elements by property values.
UIA has four main provider and client components, as shown in the following table.
Automation Element objects expose common properties of the UI elements they represent. One of these properties is the control type, which defines its basic appearance and functionality as a single recognizable entity (e.g., a button or check box).
In addition, elements expose control patterns that provide properties specific to their control types. Control patterns also expose methods that enable clients to get further information about the element and to provide input.
The UIA tree is not a fixed structure and is seldom seen in its totality because it might contain thousands of elements. Parts of the tree are built as they are needed, and the tree can undergo changes as elements are added, moved, or removed.
Having a well-known identifier allows assistive technology (AT) devices to more easily determine what types of controls are available in the user interface (UI) and how to interact with the controls. A human-readable representation of the UIA control type information is available as a LocalizedControlType property, which can be customizable by control or application developers.
UIA uses control patterns to represent common control behaviors. For example, the Invoke control pattern is used for controls that can be invoked (such as buttons) and the Scroll control pattern is used for controls that are scrollable viewport
s (such as list boxes, list views, or combo boxes). Because each control pattern represents a separate functionality, they can be combined to describe the full set of functionality supported by a particular control.
Efficiency is improved by allowing provider applications to raise events selectively, depending on whether any clients are subscribed to those events, or not at all, if no clients are listening for any events.
Exposing the textual content of a control is accomplished through the use of the TextPattern control pattern, which represents the contents of a text container as a text stream. In turn, TextPattern requires the support of the TextPatternRange class to expose format and style attributes. TextPatternRange supports TextPattern by representing a contiguous text span in a text container with the Start and End endpoints. Multiple or disjoint text spans can be represented by more than one TextPatternRange objects. TextPatternRange supports functionality such as clone, selection, comparison, retrieval and traversal.
Programmatic access provides the ability to imitate, through code, any interaction and experience exposed by traditional user interactions. UIA enables programmatic access through five components:
Besides Windows platforms, the Olive project (which is a set of add-on libraries for the Mono core aiming for the .NET Framework support) includes a subset of WPF (PresentationFramework and WindowsBase) and UI Automation.
Novell's Mono Accessibility project is an implementation of the UIA Provider and Client specifications targeted for the Mono framework. Additionally, the project provides a bridge to the Accessibility Toolkit (ATK) for Linux assistive technologies (ATs). Novell is also working on a bridge for UIA-based ATs to interact with applications that implement ATK.
Microsoft
Microsoft Corporation is an American public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through its various product divisions...
UI Automation (UIA) is an Application Programming Interface
Application programming interface
An application programming interface is a source code based specification intended to be used as an interface by software components to communicate with each other...
(API) for User Interface
User interface
The user interface, in the industrial design field of human–machine interaction, is the space where interaction between humans and machines occurs. The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the...
(UI) accessibility that is designed to help Assistive Technology
Assistive technology
Assistive technology or adaptive technology is an umbrella term that includes assistive, adaptive, and rehabilitative devices for people with disabilities and also includes the process used in selecting, locating, and using them...
(AT) products interact with standard and custom UI elements of an application
Application software
Application software, also known as an application or an "app", is computer software designed to help the user to perform specific tasks. Examples include enterprise software, accounting software, office suites, graphics software and media players. Many application programs deal principally with...
(or the operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
) as well as to access, identify, and manipulate an application's UI elements. AT products work with applications in order to provide better access for individuals who have physical or cognitive difficulties, impairments, or disabilities. Some examples of this are screen readers for users with limited sight, on-screen keyboards for users with limited physical access, or captions for users with limited hearing.
UIA is similar to Microsoft Active Accessibility
Microsoft Active Accessibility
Microsoft Active Accessibility is an Application Programming Interface for user interface accessibility. MSAA was introduced as a platform add-on to Microsoft Windows 95 in 1997...
(MSAA) in that it provides a means for exposing and collecting information about user interface elements and controls to support user interface accessibility and software test automation. However, UIA is a newer technology that provides a much richer object model than MSAA, and is compatible with both Win32 and the .NET Framework
.NET Framework
The .NET Framework is a software framework that runs primarily on Microsoft Windows. It includes a large library and supports several programming languages which allows language interoperability...
. UIA is designed so that it can be supported across platforms other than Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
. For controls and frameworks written from scratch, UIA is recommended today. While MSAA is handy for relatively simple controls, the technology doesn't support the complexity of modern user interfaces.
The current and latest specification of UIA is found as part of the Microsoft UI Automation Community Promise Specification
Microsoft Open Specification Promise
The Microsoft Open Specification Promise , is a promise by Microsoft, published in September 2006, to not assert legal rights over certain Microsoft patents on implementations of an included list of technologies....
.
History
In 2005, Microsoft released UIA as a successor to the older Microsoft Active Accessibility (MSAAMicrosoft Active Accessibility
Microsoft Active Accessibility is an Application Programming Interface for user interface accessibility. MSAA was introduced as a platform add-on to Microsoft Windows 95 in 1997...
) framework.
UIA has been supported by in Windows XP
Windows XP
Windows XP is an operating system produced by Microsoft for use on personal computers, including home and business desktops, laptops and media centers. First released to computer manufacturers on August 24, 2001, it is the second most popular version of Windows, based on installed user base...
, Windows Vista
Windows Vista
Windows Vista is an operating system released in several variations developed by Microsoft for use on personal computers, including home and business desktops, laptops, tablet PCs, and media center PCs...
, Windows Server 2003
Windows Server 2003
Windows Server 2003 is a server operating system produced by Microsoft, introduced on 24 April 2003. An updated version, Windows Server 2003 R2, was released to manufacturing on 6 December 2005...
, and Windows Server 2008. UIA has been released in Windows 7 as part of the new Windows Automation API framework, where UIA is a subset of that framework.
Version history
Developers can utilize either the managedManaged code
Managed code is a term coined by Microsoft to identify computer program code that requires and will only execute under the "management" of a Common Language Runtime virtual machine ....
UI Automation API or the native UI Automation API. The native API is prominent between the two.
The managed UI Automation API is a part of the overall .NET Framework. It was included starting in version 3.0 of the .NET Framework.
The native UI Automation API (provider) is included as part of the Windows Vista and Windows Server 2008 SDK
Software development kit
A software development kit is typically a set of software development tools that allows for the creation of applications for a certain software package, software framework, hardware platform, computer system, video game console, operating system, or similar platform.It may be something as simple...
. The native UI Automation API is also distributed with the .NET Framework. The Windows Automation API 3.0, which includes managed and native UI Automation API as well as MSAA 3.0, is included in Windows 7 and available for Windows XP, Windows Vista, and Windows Server 2003 and 2008.
Motivation and goals
In order to improve on Microsoft Active Accessibility (MSAA), UIA aims to address the following group of goals:- Enable efficient client performance without forcing clients to hook into a target application’s process.
- Expose more information about the UI.
- Co-exist with and use MSAA, but do not inherit problems that exist in MSAA.
- Provide an alternative to MSAA that is simple to implement.
The Microsoft Windows implementation of UIA features COM
Component Object Model
Component Object Model is a binary-interface standard for software componentry introduced by Microsoft in 1993. It is used to enable interprocess communication and dynamic object creation in a large range of programming languages...
-based interfaces and managed interfaces that are introduced with Microsoft .NET Framework. An API reference is available for both managed and unmanaged code.
Technical overview
Figure 1 below shows the architecture of UIA. Applications such as word processing programs are called servers in Microsoft Active Accessibility (MSAA) and providers in UIA because they serve or provide information about their user interfaces (UI). Accessibility tools such as screen readers are called clients in both MSAA and UIA because they consume and interact with application UI information.With UIA, the UIA Core component (UIAutomationCore.dll) is loaded into the processes of both the accessibility
Accessibility
Accessibility is a general term used to describe the degree to which a product, device, service, or environment is available to as many people as possible. Accessibility can be viewed as the "ability to access" and benefit from some system or entity...
tools and applications. This component manages cross-process communication, and it also provides higher-level services, such as searching for elements by property values.
UIA has four main provider and client components, as shown in the following table.
| Component | Description |
|---|---|
| UIAutomationCore (UIAutomationCore.dll and dependents) | The underlying code (sometimes called the UIA core) that handles communication between providers and clients. UI Automation Core also offers the provider and client API interfaces for unmanaged applications and clients; unmanaged applications (either clients or providers) do not require the managed assemblies listed below. |
| Managed Provider API (UIAutomationProvider.dll and dependents) | A set of interface Interface (computer science) In the field of computer science, an interface is a tool and concept that refers to a point of interaction between components, and is applicable at the level of both hardware and software... definitions and functions that are implemented by managed UIA provider applications. Providers are objects that provide information about UI elements and respond to programmatic input. |
| Managed Client API (UIAutomationClient.dll and dependents) | A set of interface definitions and functions for managed UIA client applications. |
| UIAutomationClientsideProviders.dll | A set of UIA provider implementations for legacy Win32 controls and MSAA applications. This client-side provider is available to managed client applications by default. |
Elements
UIA exposes every piece of the UI to client applications as an Automation Element. Elements are contained in a tree structure, with the desktop as the root element. Clients can filter the raw view of the tree as a control view or a content view. Applications can also create custom views.Automation Element objects expose common properties of the UI elements they represent. One of these properties is the control type, which defines its basic appearance and functionality as a single recognizable entity (e.g., a button or check box).
In addition, elements expose control patterns that provide properties specific to their control types. Control patterns also expose methods that enable clients to get further information about the element and to provide input.
Tree
Within the UIA tree there is a root element that represents the current desktop and whose child elements represent application windows. Each of these child elements may contain elements representing pieces of UI such as menus, buttons, toolbars, and list boxes. These elements, in turn, can contain other elements, such as list items.The UIA tree is not a fixed structure and is seldom seen in its totality because it might contain thousands of elements. Parts of the tree are built as they are needed, and the tree can undergo changes as elements are added, moved, or removed.
Control types
UIA control types are well-known identifiers that can be used to indicate what kind of control a particular element represents, such as a combo box or a button.Having a well-known identifier allows assistive technology (AT) devices to more easily determine what types of controls are available in the user interface (UI) and how to interact with the controls. A human-readable representation of the UIA control type information is available as a LocalizedControlType property, which can be customizable by control or application developers.
Control patterns
Control patterns provide a way to categorize and expose a control's functionality independent of the control type or the appearance of the control.UIA uses control patterns to represent common control behaviors. For example, the Invoke control pattern is used for controls that can be invoked (such as buttons) and the Scroll control pattern is used for controls that are scrollable viewport
Viewport
A viewport is a rectangular viewing region in computer graphics, or a term used for optical components. It has several definitions in different contexts:- Computing :...
s (such as list boxes, list views, or combo boxes). Because each control pattern represents a separate functionality, they can be combined to describe the full set of functionality supported by a particular control.
Properties
UIA providers expose properties on UIA elements and the control patterns. These properties enable UIA client applications to discover information about pieces of the user interface (UI), especially controls, including both static and dynamic data.Events
UIA event notification is a key feature for assistive technologies (AT) such as screen readers and screen magnifiers. These UIA clients track events that are raised by UIA providers that occur within the UIA, and use the information to notify end users.Efficiency is improved by allowing provider applications to raise events selectively, depending on whether any clients are subscribed to those events, or not at all, if no clients are listening for any events.
TextPattern
UIA exposes the textual content, including format and style attributes, of text controls in UIA-supported platforms. These controls include, but are not limited to, the Microsoft .NET Framework TextBox and RichTextBox as well as their Win32 equivalents.Exposing the textual content of a control is accomplished through the use of the TextPattern control pattern, which represents the contents of a text container as a text stream. In turn, TextPattern requires the support of the TextPatternRange class to expose format and style attributes. TextPatternRange supports TextPattern by representing a contiguous text span in a text container with the Start and End endpoints. Multiple or disjoint text spans can be represented by more than one TextPatternRange objects. TextPatternRange supports functionality such as clone, selection, comparison, retrieval and traversal.
UI Automation for Automated Testing
UIA can also be useful as a framework for programmatic access in automated testing scenarios. In addition to providing more refined solutions for accessibility, it is also specifically designed to provide robust functionality for automated testing.Programmatic access provides the ability to imitate, through code, any interaction and experience exposed by traditional user interactions. UIA enables programmatic access through five components:
- The UIA tree facilitates navigation through the logical structure of the UI for accessibility and automation.
- UI Automation Elements are individual components in the UI.
- UI Automation Properties provide specific information about UI elements or the Control Pattern.
- UI Automation Control Patterns define a particular aspect of a control's functionality or feature; they can consist of property, method, event, and structure information.
- UI Automation Events provide a trigger to respond to changes and notifications in UIA information.
Availability
UIA was initially available on Windows Vista and Windows Server 2008, and it was also made available to Windows XP and Windows Server 2003 as part of .NET Framework 3.0. It has been integrated with all subsequent Windows versions, up to and including Windows 7.Besides Windows platforms, the Olive project (which is a set of add-on libraries for the Mono core aiming for the .NET Framework support) includes a subset of WPF (PresentationFramework and WindowsBase) and UI Automation.
Novell's Mono Accessibility project is an implementation of the UIA Provider and Client specifications targeted for the Mono framework. Additionally, the project provides a bridge to the Accessibility Toolkit (ATK) for Linux assistive technologies (ATs). Novell is also working on a bridge for UIA-based ATs to interact with applications that implement ATK.
Related technology and interoperability
- Microsoft Active AccessibilityMicrosoft Active AccessibilityMicrosoft Active Accessibility is an Application Programming Interface for user interface accessibility. MSAA was introduced as a platform add-on to Microsoft Windows 95 in 1997...
(MSAA): UIA is the successor to MSAA. However, since there are still MSAA based applications in existence, bridges are used to allow communication between UIA and MSAA applications. So information can be shared between the two APIs, an MSAA-to-UIA Proxy and UIA-to-MSAA Bridge were developed. The former is a component that consumes MSAA information and makes it available through the UIA client API. The latter enables client applications using MSAA access applications that implement UIA. - Accessible Rich Internet Applications (ARIA): The UIA AriaRole and AriaProperties properties can provide access to the ARIA attribute values corresponding to an HTML element (which can be exposed as an automation element by web browsers). General mapping from ARIA attributes to UIA is also available.
- Windows Automation API: Starting with Windows 7, Microsoft is packaging its accessibility technologies under a framework called Windows Automation API. Both MSAA and UIA will be part of this framework.
- Mono Accessibility Project: On November 7, 2007, Microsoft and Novell Inc.NovellNovell, Inc. is a multinational software and services company. It is a wholly owned subsidiary of The Attachmate Group. It specializes in network operating systems, such as Novell NetWare; systems management solutions, such as Novell ZENworks; and collaboration solutions, such as Novell Groupwise...
, after completion of a year of their interoperability agreement, announced that they would be extending their agreement to include accessibility. Specifically, it was announced that Novell would develop an open source adapter allowing the UIA framework to work with existing LinuxLinuxLinux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
accessibility projects such as the Linux Accessibility ToolkitAccessibility ToolkitIn computing, Accessibility Toolkit refers in particular to the GNOME ATK.The GNOME ATK, a developer toolkit, allows programmers to use common GNOME accessibility features in their applications. This includes such features as high-contrast visual themes for the visually impaired and keyboard...
(ATK), which ships with SUSESUSE Linux distributionsSUSE Linux is a computer operating system. It is built on top of the open source Linux kernel and is distributed with system and application software from other open source projects. SUSE Linux is of German origin and mainly developed in Europe. The first version appeared in early 1994, making...
Linux Enterprise DesktopSUSE Linux Enterprise DesktopSUSE Linux Enterprise Desktop , formerly introduced as Novell Linux Desktop, is a Linux distribution supplied by Novell and targeted at the business market. It is targeted for desktops. New major versions are released at an interval of 24–36 months, while minor versions are released every 9–12...
, Red HatRed HatRed Hat, Inc. is an S&P 500 company in the free and open source software sector, and a major Linux distribution vendor. Founded in 1993, Red Hat has its corporate headquarters in Raleigh, North Carolina with satellite offices worldwide....
Enterprise LinuxRed Hat Enterprise LinuxRed Hat Enterprise Linux is a Linux-based operating system developed by Red Hat and targeted toward the commercial market. Red Hat Enterprise Linux is released in server versions for x86, x86-64, Itanium, PowerPC and IBM System z, and desktop versions for x86 and x86-64...
and Ubuntu LinuxUbuntu (operating system)Ubuntu is a computer operating system based on the Debian Linux distribution and distributed as free and open source software. It is named after the Southern African philosophy of Ubuntu...
. This would eventually make UIA cross-platform.