Resources for UI Automation and simulating user input

An important part of my job is to be able to interface our product with third party UI’s. Typically, I know virtually nothing about the UI before I begin the project, only that I will need to enter data into different fields.

Over the years, I’ve accumulated a list of my “goto tools” to help me assess each application. Behold:

Tools:

  • UI Spy: This tool used to come with older versions of the Windows SDK.  I personally think this is the most important tool.  It allows you to see properties of every Windows, Pane, and Control in an Win32 application.  I had a real hard time finding this tool a long time ago.  Fortunately I came across Andrew Tokeley’s blog and he has it zipped up and available.  You can download it from there.
    Andrew’s Blog
  • UIA Verify / Inspect.exe / AccScope.exe / AccEvent.exe: These tools are extremely helpful for analyzing user interfaces and will help determine what automation and input functionality is supported.  These are all included with the Windows 8.1 SDK.
    Get the Windows SDKUIA Verify Codeplex Site
  • Snoop (WPF): Snoop is one of my favorite tools for analyzing WPF applications.  It’s open source and can be downloaded from their GitHub page.
    Snoop GitHub
  • Window Detective: A great open source tool similar to Spy++ but it has some unique features and in some ways is more intuitive. I highly recommend checking this one out.
    Window Detective Sourceforge
  • Spy++: Good ol’ Spy++. This tool has been around for nearly a million years, and it’s still distributed with Visual Studio. You can find out a lot of information about a Window, Thread, or Process by monitoring the Win32 Messages and viewing some elements of the control and window hierarchy.
  • XAML Spy: I haven’t done much with automating WPF/Silverlight UIs, but as Keith mentioned (in the comments below), XAML Spy seems to be a nice tool for handling debugging WPF/XAML-based UIs. There is a free Express edition and the the full featured edition which costs money.
    XAML Spy Homepage
  • F12 Dev Tools (IE, Firefox, Chrome): When automating a Web UI through a browser, the Inspector in the F12 Developer Tools can be the difference between a one-week implementation and a one-day implementation.

Libraries:

  • Selenium: I have recently replaced WatiN with Selenium for all my web browser automation needs. It supports all the latest browsers on several operating systems. In order to trigger the automation routine from a Windows application, you should use the Selenium WebDriver libraries. Despite what their supported browser list may say, they are almost always on top of any new browser versions (their documentation guy must not be as on top of things).
    Selenium HQ | Selenium on Google Code
  • WatiN: WatiN (pronounced What-In) is an amazing open source .NET library for automating UI’s in a web browser.  You have full access to the DOM from your WinForms application.  If you ever need to have a .NET project automate the UI for a web form through IE or Firefox you will love the power of this library.  Note, there is some limitations in browser versions, it seems to support IE 6 – 9, and FireFox 2 – 3.  Hopefully a newer version is in the works.
    WatiN Homepage
  • UI Automation Framework: Microsoft .NET includes a set namespaces for helping with UI Automation in WinForms apps (Win32, WPF, .NET WinForms) and it’s really quite powerful.  I believe the first version to include this was .NET 3.0.  You can read more about this at MSDN.  This is great for most native Windows apps, but if you find yourself needing to automate something more complex (such as an application through Terminal Services or Citrix) see the next option below.
    UI Automation Overview
  • BitCollectors.UIAutomationLib: And finally, my own shameless plug.  If you find yourself needing to simulate keystrokes and mouse clicks it really doesn’t get much easier than this.  This is a library I developed and have been maintaining for a couple of years now internally and I just published it on GitHub.  You can pass in an XML file which defines how your keystrokes and mouse clicks will get executed against a specific application.  This will work with just about anything, including applications running through Terminal Services and Citrix.
    Blog Post | GitHub Page | ReadMe

Update (03.24.2013): Added a few new tools to the list

Update (12.01.2014): Minor updates, added Snoop to the tools list

5 thoughts on “Resources for UI Automation and simulating user input”

    1. Thanks Keith, those are some great suggestions. I forgot about XAML Spy – I had to use that a while back for a Silverlight project (I’ll update my post with information on that). As for the other two, I haven’t ever done anything with those. I’ll definitely check out that Coded UI Map in VS2012, that sounds really useful.

  1. UiPath is also a cool tool when it comes to UI Automation. You can use it from code (SDK) or you can create workflows (visual automation) directly from UiPath Studio. It has it’s own tool for exploring the user interface tree (UiExplorer). The good thing is that you can handle complicated scenarios like Citrix (http://pages.uipath.com/tutorials/citrix-automation-data-entry), SAP (http://pages.uipath.com/tutorials/sap-gui-automation-data-entry), web browsers, Java Applications or green screens and any kind of desktops.

Leave a Reply