TweetFollow Us on Twitter

User Interface Scripting

Volume Number: 21 (2005)
Issue Number: 6
Column Tag: Programming

AppleScript Essentials

User Interface Scripting

by Benjamin S. Waldie

As we have seen in the past, AppleScript is a great tool for creating some pretty amazing automated workflows. When implementing an AppleScript-based workflow, you are really only limited by your imagination, and by the AppleScript support that is available in the applications that you want to automate.

Many scriptable applications offer enough AppleScript support for the types of tasks that you would want to automate the most. However, at times, you may find yourself needing to script an application with limited AppleScript support, and that one task you really need to automate just is not accessible through scripting. Or, worse yet, the application you want to automate is not scriptable at all!

How do you handle these limitations? Do you simply give up? I think not. For starters, you might check around to see if there are other scriptable applications that can be substituted to automate the same task.

Another option is to consider trying to automate the application by writing AppleScript code that interacts directly with the application's interface itself. Fortunately, with the release of Mac OS X 10.3, Apple introduced a new AppleScript feature, user interface scripting, or UI Scripting, which can be used to do just that.

What is UI Scripting?

UI Scripting is a feature in Mac OS X that provides a way for AppleScripts to interact directly with interface elements in running applications. For example, you might write a script that clicks a button in a window, selects a menu item in a popup menu, or types text into a text field of a non-scriptable application.

The AppleScript terminology for UI Scripting is not built into AppleScript itself, nor is it included in a scripting addition. Rather, UI Scripting is part of System Events, a background application that provides a central location for interacting with various aspects of Mac OS X itself, including running processes, or applications.

In order to view the AppleScript terminology for UI Scripting, open the dictionary for System Events by double clicking on it in the Script Editor's Library palette.


Figure 1. The Script Editor Library Palette and the System Events Dictionary

If you are using a different script editor, or prefer to open the dictionary by selecting File > Open Dictionary in the Script Editor menu bar, System Events can be found in the System > Library > CoreServices folder on your machine. In the System Events dictionary, UI Scripting-related terminology can be found under the Processes Suite. We will explore the UI Scripting terminology found in this suite shortly.

Enabling UI Scripting

Before writing code that makes use of UI Scripting in Mac OS X, you will first need to manually configure the machine on which the script will run, so that it can respond to UI Scripting commands. To enable UI Scripting, launch the System Preferences application, and click on the Universal Access button. Next, enable the checkbox labeled Enable access for assistive devices. See figure 2.


Figure 2. Universal Access Preference Pane

If the final destination for your script is another user's machine, then you should make sure that you provide that user with instructions for enabling UI Scripting on the machine prior to running the script. In order to ensure that the user actually follows your instructions, you may also want to add some code into your script to verify that UI Scripting has been enabled, and prevent the script from proceeding if it has not been enabled. You can tell if UI Scripting has been enabled by checking the value of the UI elements enabled property of the System Events application. This property will contain a true or false value, indicating whether or not UI Scripting is enabled.

The following example code will check to see if UI Scripting is enabled. If it is not enabled, the script will launch System Preferences, display the Universal Access pane, and notify the user that the Enable access for assistive devices checkbox must be enabled in order to proceed.

tell application "System Events" to set isUIScriptingEnabled to UI elements enabled
if isUIScriptingEnabled = false then
   tell application "System Preferences"
      activate
      set current pane to pane "com.apple.preference.universalaccess"
      display dialog "Your system is not properly configured to run this script.  
         Please select the \"Enable access for assistive devices\" 
         checkbox and trigger the script again to proceed."
      return
   end tell
end if

-- Add your code here, which will be triggered if UI Scripting is enabled

UI Scripting Terminology

As previously mentioned, the AppleScript terminology for UI Scripting is found under the Processes Suite in the System Events dictionary. Here, you will find a variety of classes, or objects, along with a few commands. Let's briefly discuss the commands first. We'll see them in action a little later.

Commands

There are only a handful of commands that you will use for UI Scripting, and they are click, key code, keystroke, perform, and select. In this article, I am going to touch on the click and keystroke commands, as they are what I have found to be particularly useful, and I have chosen to include them in the examples at the end of this article.

As one would expect, the click command is used to simulate the user clicking on an interface element. For example, you might use the click command to click a button on a window.

click button 1 of window 1

The keystroke command is used to enter keystrokes, as if they were typed by a user. The keystroke command also offers the ability, through an optional parameter, to specify modifier keys that should be invoked when the command is executed. For example, you may need to simulate the process of holding down the command key while typing a key.

keystroke "P" using {command down}

Again, these are just very brief introductions to two commands that can be used to interact with interfaces. You should spend some time exploring the other commands on your own, in order to become familiar with them. Now, let's take a look at some of the classes listed under the Processes suite of the System Events dictionary.

The Process Class

When working with UI Scripting, the top-level class you will address is the process class. A process signifies any process, or application, that is currently running on the machine. To get the names of any currently running processes, use the following syntax:

tell application "System Events" to return name of every process

--> {"loginwindow", "Dock", ... "Finder", "Preview", "Script Editor"}

Processes possess a variety of properties, which can be accessed through AppleScript. One such property is the frontmost property. In some cases, you may need to bring a process to the front in order to interact with its interface. This can be done by setting the frontmost property of the process to a value of true. For example, the following code brings the application Preview to the front.

tell application "System Events" to set frontmost of process "Preview" to true

Note in this example that I refer to the Preview application as a process of the System Events application. Be sure to explore the other properties of processes, as well, as they can provide other important information to your script.

I have chosen to use Preview for the examples in this article, since Preview is not AppleScriptable, and since it is installed with Mac OS X.

UI Element Classes

A process with an interface typically contains a variety of interface elements, such as windows, buttons, text fields, and more. These are the elements with which you will primarily want your script to interact.

In the System Events dictionary, a UI Element is, in itself, a class, and possesses a variety of properties common to most interface elements. For example, since many interface elements possess the ability to be enabled or disabled, an enabled property can be found under the more generic class of UI Element. Processes also inherit some properties from this class.

There are a number of different classes of interface elements in the Processes suite. If you have used AppleScript Studio in the past, then you will no doubt be familiar with the names of at least a few of them, although the exact naming may be slightly different from that of AppleScript Studio. Some of the more commonly accessed classes of interface elements include checkboxes, menus, menu items, pop up buttons, radio buttons, text fields, and windows. As mentioned above, since all interface elements possess certain characteristics, each class of interface element inherits properties from the UI Element class.

Be sure to browse through the various classes of interface elements listed in the Processes suite.

Referring to UI Elements

Just like scripting any object in a scriptable application, when writing code to interact with an interface, you must address the interface elements within their object hierarchy. Deciphering an object's hierarchy within a process' interface is usually no small feat. For basic interfaces, it may be as simple as referring to an interface element on a specific window. However, for more complex interfaces, it can quickly become a long set of nested object references.

Let's take a look at an example. The following code will click the Drawer button in the tool bar of an opened document in Preview, toggling the drawer opened or closed.

tell application "System Events"
   tell process "Preview"
      click button "Drawer" of tool bar 1 of window 1
   end tell
end tell

You can see from this example that in order to interact with the Drawer button, you must indicate specifically where the button is located within the interface. In this case, it is on the tool bar of the front window.

There is some good news, though. Apple doesn't leave you high and dry to figure out these object hierarchies on your own. There is a tool available to help. It is called UI Element Inspector, and it can be downloaded from Apple's AppleScript web site <http://www.apple.com/applescript/uiscripting/>.


Figure 3. UI Element Inspector

When launched, UI Element Inspector floats above all interfaces, and displays information about the interface elements located directly below the mouse pointer. Information displayed includes the location of the interface element, within its object hierarchy, along with values from various properties. In figure 2, UI Element Inspector is indicating the exact location of the Drawer button in a Preview window - within a tool bar, which is within a window of the application. As you can see, the tool bar itself does not possess a specific name, which is why I referred to it as tool bar 1 in my code.

Examples

Now, let's take a look at some examples of UI Scripting in action. The following example code assumes that you have a document opened in Preview. When this code is run, it will bring Preview to the front and print the front window.

tell application "System Events"
   tell process "Preview"
      set frontmost to true
      click menu item "Print..." of menu "File" of menu bar 1
      repeat until sheet "Print" of window 1 exists
      end repeat
      keystroke return
   end tell
end tell

The following example code assumes that you have a document opened in Preview. When this code is run, it will bring Preview to the front and export the front window to the desktop in TIFF format.

tell application "System Events"
   tell process "Preview"
      set frontmost to true
      keystroke "E" using {command down, shift down}
      repeat until sheet 1 of window 1 exists
      end repeat
      tell sheet 1 of window 1
         keystroke "Output"
         keystroke "D" using command down
         tell pop up button 1 of group 1
            click
            tell menu 1
               click menu item "TIFF"
            end tell
         end tell
         delay 1
         click button "Save"
      end tell
   end tell
end tell

Limitations

UI Scripting is based on Mac OS X's accessibility framework, which is why it must be enabled from the Universal Access preference pane. Because it is based on the accessibility framework, UI Scripting can only interact with applications that have been updated to support this framework. The accessibility framework was first released with Mac OS X 10.2, so many applications have accessibility support built-in by now. However, some applications do not, and may not respond to UI Scripting terminology. To automate applications that do not support UI Scripting, you may need to turn to other tools, such as QuicKeys <http://www.quickeys.com>.

A Third Party Assistant

While UI Element Inspector provides some much needed input for writing your AppleScript code, a third-party application takes it a step further.

PreFab UI Browser, a commercial application available from PreFab Software, Inc., offers all of the functionality of the UI Element Inspector, and a lot more. PreFab UI Browser provides robust interfaces designed for the serious scripter, making it easy to access information about the complete interface of any accessible running application. Other features of PreFab UI Browser include the ability to trigger hot keys to enable and disable interface element browsing, auto-generation of AppleScript code to interact with interface elements, and much more. Be sure to visit the PreFab Software, Inc. web site <http://www.prefab.com> for more information about this tool, or to download a demo version. While you're there, be sure to check out another product - PreFab UI Actions. This tool will allow you to configure AppleScripts to trigger automatically whenever specific user interface actions occur in an application. For example, you could configure an application to trigger a script whenever a user selects the Save As from the File menu. The possibilities are limitless.

In Closing

As you can see, UI Scripting can be quite a useful tool when trying to automate an application that doesn't provide the AppleScript support that you need. So, before you decide that just because an application isn't scriptable, it can't be automated with AppleScript, be sure to check out its interaction with UI Scripting.

In my experience, there is not much on the Mac that can't be automated... one way or another. Until next time, keep scripting!


Benjamin Waldie is president of Automated Workflows, LLC, a firm specializing in AppleScript and workflow automation consulting. In addition to his role as a consultant, Benjamin is an author for SpiderWorks, LLC and an evangelist of AppleScript. He can often be seen presenting at Macintosh User Groups, Macworld Expos, and more. For additional information about Benjamin, please visit http://www.automatedworkflows.com, or email Benjamin at applescriptguru@mac.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Luminar 3.0.1 - Powerful, adaptive, conf...
Luminar is the new full-featured image editor that adapts to the way you edit photos. Over 300 essential tools to fix, edit, and enhance your photos with comfort. The future of photo editing is here... Read more
iMazing 2.8.5 - Complete iOS device mana...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more
OnyX 3.5.7 - Maintenance and optimizatio...
OnyX is a multifunction utility that you can use to verify the startup disk and the structure of its system files, to run miscellaneous maintenance and cleaning tasks, to configure parameters in the... Read more
TeamViewer 14.1.9025 - Establish remote...
TeamViewer gives you remote control of any computer or Mac over the Internet within seconds or can be used for online meetings. Find out why more than 200 million users trust TeamViewer! Free for non... Read more
Deeper 2.4.0 - Enable hidden features in...
Deeper is a personalization utility for macOS which allows you to enable and disable the hidden functions of the Finder, Dock, QuickTime, Safari, iTunes, login window, Spotlight, and many of Apple's... Read more
A Better Finder Attributes 6.16 - Change...
A Better Finder Attributes 6 allows you to change JPEG & RAW shooting dates, JPEG EXIF meta-data tags, file creation & modification dates, file flags and deal with invisible files. Correct... Read more
Boom 3D 1.2.3 - 3D surround sound and ph...
Boom 3D is a revolutionary app with 3D Surround Sound and phenomenally rich and intense audio that is realistic and works on any headphones. Features 3D surround sound Built-in audio player... Read more
Dash 4.6.1 - Instant search and offline...
Dash is an API documentation browser and code snippet manager. Dash helps you store snippets of code, as well as instantly search and browse documentation for almost any API you might use (for a full... Read more
iClock 4.6.5 - Customizable menu bar clo...
iClock replaces the old Apple's default menu bar clock with more features, customization and increases your productivity. Features: Have your Apple or Google calendar instantly available from the... Read more
Fantastical 2 2.5.7 - Create calendar ev...
Fantastical is the Mac calendar you'll actually enjoy using. Creating an event with Fantastical is quick, easy, and fun: Open Fantastical with a single click or keystroke Type in your event details... Read more

Latest Forum Discussions

See All

PGC London '19: Kaijack Card Attack...
What would happen if giant monsters were indestructible? Wait, let me rephrase that. What would happen if giant monsters were real and indestructible? The answer, according to Kaijack Card Attack, is that they'd play a gamw somewhere between... | Read more »
Pre-register now for epic fantasy RPG Fi...
Final Blade is an epic fantasy RPG that revolves around two warring nations and one extremely powerful sword. Having already found great success in Taiwan and South Korea, the developer is now bringing its acclaimed free-to-play iOS and Android... | Read more »
The top 3 best iPhone and iPad games com...
The end of the week has rolled around. It's been a pretty good seven days for mobile gaming - if you want proof you can click here and check out our five favorite games that came out this week - but now we're going to turn our attention to the... | Read more »
The best games for iPhone and iPad that...
Every week here at 148Apps we like to make sure that you're up to date with all the awesome games that have come out for iPhone and iPad over the past 7 days. We pick the cream of the crop to ensure that you don't have to play anything boring and... | Read more »
The top 3 best idle games for iPhone and...
Sometimes you want to play a game where you don't really have to do anything. Don't worry, we're not here to judge you, we're exactly the same. We love some action when we're in the right mood, but sometimes we want something a little more chill -... | Read more »
The top 5 best games with great stories...
Last week we took a look at the best games for iOS that you could waste hours and hours of your life playing, so now we thought we'd have a look at the top 5 games for iPhone and iPad that have great stories. These are the tales that have stuck... | Read more »
Langrisser Mobile preview - Seminal TRPG...
Langrisser Mobile is set to finally break the revolutionary TRPG franchise's 20 year silence when the it finally makes its way to touchscreens the world over at the end of the month. The original Langrisser game (known as Warsong in the west)... | Read more »
Everything you need to win at Dragalia L...
Want to up your Dragalia Lost game? You've come to the right place. Here at 148Apps, we've been digging deep into Nintendo's latest gacha game to find the best tips, tricks, and strategies to share with all of our future co-op team mates (read: you... | Read more »
How will World of Warcraft, Starcraft, a...
You might have heard that Blizzard is bringing Diablo to mobile with Diablo Ultimate. There's been plenty of controversy surrounding that decision, and it's meant that a really important piece of information has been glossed over a little. Namely... | Read more »
The top 3 best iPhone and iPad games com...
Our mission here at 148Apps is pretty simple - we want to make sure that you're playing the best iPhone and iPad games in the world. Which is why, every Friday, we like to give you an insight into the incredible games that are set to rock your... | Read more »

Price Scanner via MacPrices.net

T-Mobile offers customers 64GB Apple iPhone X...
Trade in an eligible phone at T-Mobile, plus sign up for their $30/month Unlimited plan, and they will include a 64GB iPhone Xr for just $10/month extra for up to 4 lines. That’s $40 per month,... Read more
Amazon offers blowout prices on 10.5″ Apple i...
Amazon has 10.5″ Apple iPad Pros on sale for $150 off MSRP this week, with prices starting at $499. These are the cheapest prices currently available for 10″ iPad Pros from any Apple reseller, and... Read more
Lowest Apple Mac mini price ever: Only $379 t...
B&H Photo has previous-generation 1.4GHz Mac minis available today for $379 including free shipping. That’s $120 off original MSRP for this model, and it’s the cheapest price we’ve ever seen for... Read more
Space Gray and Silver 2018 13″ 1.6GHz/256GB M...
B&H Photo has new Space Gray and Silver 2018 13″ 1.6GHz/256GB MacBook Airs on sale for $100 off MSRP, or $1299. Their prices are the lowest available for these new models from any Apple reseller... Read more
Amazon has 2018 9.7″ Apple iPads on sale toda...
Amazon is offering 9.7″ WiFi iPads with Apple Pencil Support for $30 off MSRP today, with prices starting at $299. These are the same iPads found in Apple’s retail and online stores: – 9.7″ 32GB WiFi... Read more
Amazon has 2018 9.7″ Apple iPads on sale toda...
Amazon is offering 9.7″ WiFi iPads with Apple Pencil Support for $30 off MSRP today, with prices starting at $299. These are the same iPads found in Apple’s retail and online stores: – 9.7″ 32GB WiFi... Read more
Get a clearance 10.5″ iPad Pro from Apple sta...
Apple has Certified Refurbished 2017 10.5″ iPad Pros available for $100-$170 off original MSRP, depending on the model. An Apple one-year warranty is included with each model, shipping is free, and... Read more
Get a clearance 10.5″ iPad Pro from Apple sta...
Apple has Certified Refurbished 2017 10.5″ iPad Pros available for $100-$170 off original MSRP, depending on the model. An Apple one-year warranty is included with each model, shipping is free, and... Read more
B&H has 2017 13″ MacBook Airs on sale sta...
B&H Photo has 2017 13″ 1.8GHz MacBook Airs available today for $100-$270 off MSRP, with models available starting at $899. Shipping is free: – 13″ 1.8GHz/128GB MacBook Air (MQD32LL/A): $899, $100... Read more
B&H has 2017 13″ MacBook Airs on sale sta...
B&H Photo has 2017 13″ 1.8GHz MacBook Airs available today for $100-$270 off MSRP, with models available starting at $899. Shipping is free: – 13″ 1.8GHz/128GB MacBook Air (MQD32LL/A): $899, $100... Read more

Jobs Board

Operations Associate - *Apple* Blossom Mall...
Operations Associate - Apple Blossom Mall Location:Winchester, VA, United States- Apple Blossom Mall 1850 Apple Blossom Dr Job ID:1044618 Date:Today Job Read more
Sephora Product Consultant - *Apple* Blosso...
Sephora Product Consultant - Apple Blossom Mall Location:Winchester, VA, United States- Apple Blossom Mall 1850 Apple Blossom Dr Job ID:1056553 Date:Today Job Read more
Cashier - *Apple* Blossom Mall - JCPenney (...
Cashier - Apple Blossom Mall Location:Winchester, VA, United States- Apple Blossom Mall 1850 Apple Blossom Dr Job ID:1042611 Date:Today Job Description Read more
Omni-Channel Associate - *Apple* Blossom Ma...
Omni-Channel Associate - Apple Blossom Mall Location:Winchester, VA, United States- Apple Blossom Mall 1850 Apple Blossom Dr Job ID:1074107 Date:Today Job Read more
Home Commission Associate - *Apple* Blossom...
Home Commission Associate - Apple Blossom Mall Location:Winchester, VA, United States- Apple Blossom Mall 1850 Apple Blossom Dr Job ID:1079517 Date:October Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.