With Xcode 7, Apple has added integrated support for automated UI testing. I recently tried adding UI test cases to an app and what I thought would be a relatively simple task of recording some clicks and keystrokes and tossing in a few asserts quickly became a multi-day tumble down a rabbit hole of bugs, Xcode quirks, and headaches.

Don't get me wrong, Xcode's UI testing tools are powerful, straightforward, and the integration with Xcode bots is great. However, they are also unsurprisingly rough in some places. To make matters worse, there is a stark lack of online documentation and examples.

This article will introduce some basic UI testing concepts, as well as offer tips for common issues you're likely to encounter (generally applicable to both OS X and iOS testing).

How Xcode's UI Testing Works

Under the hood, UI tests in Xcode are driven by Apple's Accessibility APIs. The accessibility layer acts as a set of eyes and hands for the user, interacting with any on-screen elements that opt in by supplying accessibility metadata to the system. The UI testing frameworks leverage this to exercise your app's UI and test the results. A couple points to note:

Common UI Testing Classes

XCTestCase - Wraps test methods and asserts
XCUIElement - Represents a UI element (window, control, view, etc.)
XCUIElementQuery - Represents a query for one or more elements
XCUIApplication - An XCUIElement subclass which represents the test application

A point worth mentioning: XCUIElement does not actually represent a UI element but rather a query for that element. That is, you can create an XCUIElement which is a valid object but points to UI that does not exist. Likewise, an element on screen that you reference could disappear later, and attempting to interact with it when exists == NO will fail the test.

Leveraging XCUIApplication

XCUIApplication has a minimal API, but a few methods warrant special attention:

UI Recording

With Xcode 7, Apple has added a Record UI Test button which allows you to step through your UI and have those actions recorded for you in whatever current test case method you're editing. While this sounds great in theory, in practice I've found its current implementation to be buggy and unreliable. Here are some tips for using it:

Tokens

When recording UI actions, there are many ways of referencing the same element. Xcode doesn't know which of these is the best way to express that query for your test, so it will tokenize these references to allow you to select other options if you desire.

Unfortunately, tokens are currently quite buggy. Most interactions with them can have unexpected results. Typing with a token selected, for example, explodes it into gibberish. Even the Editor > Flatten Tokens menu item will potentially chop off important token elements unexpectedly.


New in Xcode 7, exploding tokens:

Debugging

While debugging tests you'll find that your console is noisier than normal. As previously mentioned, XCUIElements represent queries, not hard links to UI components. As such, every time they're referenced the Accessibility framework works its way down the entire accessibility hierarchy in your app. This journey is logged to the console. It means that even when your target (and test) are paused, running a simple LLDB command will spit out a surprising amount of text, with your actual debugger output hidden somewhere in the mix.

UI Timeouts

Any action in your UI test cases must be resolved within 2.5 seconds. This is a hardcoded limitation that can't be changed. What does this mean?

Interruption Handlers

You may find that during testing you encounter failures as a result of system alerts on the screen. For example, this can happen when the user is prompted to allow location services, or on OS X if some other process crashes and the Finder displays a floating system alert to notify of the unexpected termination.

These alerts can be surprisingly difficult to deal with. Apple has provided a hook to handle these alerts via the XCTestCase method -addUIInterruptionMonitorWithDescription:handler:. A simple example:


In my experience so far, Apple's interruption handlers are unreliable. While reproducing a crashed application alert on OS X I found that alerts already on screen when the UI tests started were only recognized (and the handler called) about half the time. Alerts that appeared during the test itself fared better but still not 100%, in many cases the handler was never called and the test failed as a result of the system alert floating unexpectedly over the UI being exercised.

It also appears that interruption handlers will not be called if your test is waiting for an expectation via -waitForExpectationsWithTimeout:handler:.

Documentation

A cautionary note: if you plan on writing UI tests be prepared to learn by experimentation and trial-and-error. The documentation for the Accessibility frameworks and related XCTestCase classes is extremely thin in some areas, and online examples can be equally difficult to come by.

In many cases the docs are simply wrong (ex: NSAccessibilityCheckBox documentation doesn't mention that Cocoa checkboxes will return @(2) for -accessibilityValue to indicate a mixed state, despite NSMixedState being defined as -1):


Or they're frustratingly vague:

Accessibility Inspector

This Apple developer app is invaluable when writing and debugging UI tests on OS X (but it's also available for iOS), as it shows you exactly what the Accessibility framework sees. You can lock it while hovering over a particular element to view its reported frame on screen, as well as inspect its properties and its placement in the accessibility hierarchy.

Using it is as simple as launching the app and hovering the mouse over any control or UI element of an accessible application. You can discover a great deal by examining Apple's own apps (the screenshot below shows the Accessibility Inspector locked after hovering over a button in Apple's Keynote application).

It also includes a verifier tool which is available by choosing Window > Accessibility Verifier. This tool will inspect the complete hierarchy of any running application and report missing properties, incorrectly implemented roles, and other accessibility problems. (It's worth mentioning that running this on Apple's own apps will often result in a plethora of warnings & errors, an indicator of how challenging the process of fully supporting accessibility can be.)

Summing Up

UI testing is an awesome way of exercising your custom controls and interface elements, catching bugs and regressions, and ensuring consistent behaviors in your apps. With the new features in Xcode 7 Apple has made it even easier to get started. Though there are some bugs to cope with and pitfalls to avoid, these tools will only get better in time. If you haven't already, check them out!