<select> your poison part 2: test all the things

TL:DR; Jump straight to the recommendations

In part 1 of this deep dive, we took a broad look at both how the native <select> element works across browsers and platforms, and the current state of accessible custom select component implementations. It’s necessary to look at how fragmented current select UX patterns are in order to understand why this is so difficult, and to sort through all the conflicting advice out there. However, it doesn’t tell you how to actually create a usable custom select.

A quick note on accessibility vs. usability:

It is entirely possible to make any of the semantic or behavioral variations outlined in part 1 “accessible,” at least in the sense of meeting the Web Content Accessibility Guidelines (WCAG) AA criteria. All of the provided examples meet Name, Role, Value, Keyboard, and Info and Relationships. This is true even if assistive tech support is lacking, or the keyboard implementation is so obscure that a real human would be effectively blocked.

Measuring whether any given select pattern is usable is an entirely different question: it requires caring about real-world browser and assistive tech support, how intuitive the controls are, and whether the behavior is predictable. The only way to really know which select component patterns are the most usable is to put them in front of people with disabilities and observe how well they perform in practice.

Let’s test this

As with any study, there was a limit to the scope of what was tested. As a developer and a bit of a standards wonk, I was more interested in how to best spec and code a select component than in the visual styles, label text, and so on. Those are all obviously extremely relevant to the eventual usability of a control, but are less specific to select components, and were not tested.

I ended up creating a set of custom select components specifically for the task, in order to have complete control over their markup, presentation, and behavior. That allowed me to ensure that the only differences between components were those intentionally being tested.

Another place where scope was limited was with the number of participants — there were 12 people in total. It would be very interesting to do a similar study sometime with a much larger number of people (do you have a large testing budget and want to collaborate? Get in touch!). However, keeping it small both made sense for practical reasons, and allowed me to individually review each interaction by each participant. It allowed me to take a qualitative deep dive into what was frustrating, what worked well, and the silent mistakes and assumptions by the participants that didn’t make it into usability scores and pass rates. On the other hand, the results are not statistically significant or a broad representation of all people with disabilities.

What was tested?

The major semantic variations tested were those listed in part 1, minus the ARIA 1.0 pattern:

• The ARIA 1.1 combobox pattern
• The ARIA 1.2 combobox pattern
• A popup button
• A listbox with aria-expanded

The second two options, the popup button and expanding listbox, were only tested as <select> alternatives and not editable comboboxes, since they do not support text input.

In addition to the above, two different read-only combobox variations were thrown in to test how best to create a <select> alternative that matches the prevailing Windows desktop pattern. The ARIA 1.1 specification for combobox did not support using the combobox role for a non-editable select component, though this has changed in the 1.2 draft by switching the wording from “textbox” to “input”, and more explicitly in the APG’s combobox description. There are still no clear rules on what the markup should be, so I tested two variations found in the wild:

• A standard ARIA combobox pattern, but with a readonly input element
• No input element at all, with a focusable <div role="combobox"> instead

I also tested the following behavioral variations for the editable combobox and multi-select components:

Editable Comboboxes:

• No filtering and no autoselection based on user input (though the first match was highlighted and scrolled into view)
• Filtering, but no autoselect
• Filtering and autoselect
• Inline autocomplete

Multi-selects:

• The native <select multiple> listbox
• Displaying current selections as an editable string of comma-separated values in the input
• Displaying current selections as a list of buttons above the combobox, and clearing the input after each selection
• Displaying current selections as a list of buttons inline within the combobox, followed by the input.

The study was run in two parts, the first with the bulk of the variations spread across four short forms, and a very short second study added after the ARIA 1.2 draft combobox pattern was finalized. Participants were asked to select values, change or delete selections, and review their chosen values. They were then asked to rate how usable the experience was, and were measured on whether they were able to complete the tasks.

You can follow along with the first study here, and the second mini study here.

Participants

There were a total of 12 participants in the study, including people who were blind, low vision, and who had mobility disabilities. All used some form of assistive tech device or software, with a range of skill levels and technical ability represented. The specific breakdown was as follows:

• 8 blind or very low vision participants, using the following screen readers:
  • Chrome & JAWS on Windows
  • Chrome & NVDA on Windows
  • Edge & Narrator on Windows
  • Edge, Narrator, and a refreshable braille display on Windows
  • Safari & VoiceOver on macOS
  • Safari & VoiceOver on iOS
• 2 screen magnification users on Windows
• 1 sighted keyboard-only user on Chrome on Windows
• 1 switch device user on Chrome on Windows

This study is notably missing participants with cognitive disabilities, primarily because it was so focused on semantics and the mechanics of keyboard and pointer interaction. It remains an open question how features like filtering and autocomplete impact people with cognitive disabilities, and would be better tested in context: whether autocomplete helps a doctor reduce errors when selecting a prescription drug on the job, for example.

A quick word of caution: as mentioned earlier, a 12-person study is not enough to be anywhere near statistically significant, so these findings function more to gesture in the right direction than to dictate irrefutable fact.

Known issues

Before passing all these off to real people, my coworker and I tested them with keyboard and screen readers in Windows, macOS, iOS and Android to both to catch any coding mistakes and to get a sense of support across the board. Discovering support gaps beforehand also allowed us to pay extra attention to how blocking they were in practice during the study.

Here are the current (as of November 2019) support gaps in the tested select components:

• ARIA 1.1. comboboxes did not show up in scan mode at all in NVDA
• ARIA 1.1 comboboxes had issues with exposing an accessible name with NVDA and JAWS
• The multiselect with inline buttons did not show up in scan mode in NVDA or JAWS
• VoiceOver on macOS advised a keyboard shortcut for all comboboxes that did nothing
• VoiceOver on macOS did not consistently respond to typing in comboboxes or arrowing through options
• VoiceOver on iOS did not always announce the edit field for comboboxes, and generally had buggy and inconsistent interaction with the options menu
• VoiceOver on iOS did not always move keyboard focus when VoiceOver focus entered a combobox
• Android Talkback had difficulty entering the dropdown options menu from the combobox input

Results

Unsurprisingly, the most conclusive result was that the native <select> element had the most understandable and usable experience across every platform, at least for single selection. It blew everything else out of the water, and not a single participant had a complaint about its behavior.

However, it doesn’t make much sense to compare the usability of a single-select to the usability of a multi-select, or measure an editable combobox against a basic read-only (i.e. lacking text input) select. Each of those interaction patterns has a place, depending on context and the needs of the user interface. Proving that the <select> element is more accessible than any custom control is not only unsurprising, it is also unhelpful when an autocomplete search field is needed, or a filtered multi-select. The rest of the results will therefore be broken down by category.

(Quick note: the two variations tested in the second study, the ARIA 1.2 pattern and expanding listbox, were not given individual usability scores by participants)

Read-only single select

Are you reading this just to figure out how to style the options menu? Does the <select> element meet the functional requirements of the UI? Just use that, really. OK? OK. Now that that’s out of the way, we can take a look at the how the other options performed.

Of the semantic variations, the <div role="combobox"> pattern with no input performed best, though it predictably did better with Windows screen reader users than with VoiceOver users on macOS or iOS. There was a minor issue with Narrator not automatically switching to application mode when entering the combobox, but otherwise no one using a Windows screen reader had trouble using the control. On the flip side, one participant using macOS VoiceOver said “I don’t like that field type, it should just be a regular popup,” and the iOS VoiceOver user said it “felt weird,” though both were eventually able to complete the task.

Both the ARIA 1.1 pattern and the listbox with aria-expanded had major blocking issues. The expandable listbox did not work at all with VoiceOver in macOS or iOS, and was buggy with Narrator. The known issues with the ARIA 1.1 pattern — namely that it was not exposed to NVDA in scan mode and sometimes did not read the label in NVDA and JAWS — ended up significantly slowing down participants using those screen readers.

The remaining two options, the popup button version similar to <select> on macOS and the <input readonly> combobox both had no blocking issues, but multiple people found the semantics unexpected. The popup button was (again, unsurprisingly) the most familiar to VoiceOver users, but not to Windows screen reader users.

Editable combobox

Of the two semantic variations tested, the ARIA 1.2 pattern was less buggy across the board compared to the ARIA 1.1 pattern. Assuming the version in the working draft becomes a W3C recommendation, it seems to be the clear choice for combobox markup going forward.

Filtering results ended up being surprisingly confusing to a number of the participants. Multiple people thought they were encountering a bug, particularly when the list of results would narrow to a single option after a selection was made. Filtering also caused confusion no results were returned after someone misspelled a word or searched for an option that did not exist. This could be perhaps be partially mitigated by displaying an unselectable “no results found” option. The confusion around filtering behavior was present for participants across the board, though those using screen readers were more likely to assume it was a bug and spend extra time trying to find the full list of options.

One common thread through the entire study was that participants relying on the keyboard, both with and without a screen reader, would tab away without hitting enter or space to select the active option. This seems to imply that autoselecting the active option is the way to go. Unfortunately, the only combobox tested with autoselect behavior also filtered results, so it inherited some user confusion from that behavior. Despite that, it had the highest usability scores, just barely edging out the unfiltered combobox.

The inline autocomplete version also did OK, though multiple screen reader users commented that they found it confusing that only the auto-inserted string following the cursor was announced, rather than the entire value. The underlying reason for this is that the inserted string is also selected to allow additional user input to overwrite it, so there does not seem to be any way around this behavior.

Multi-select

It can be fun to find the exception to the rule, and <select multiple> certainly demonstrates that not all native HTML features are accessible. It had the worst completion rates and usability scores of any component tested by far. Not a single participant relying on the keyboard or a desktop screen reader managed to select multiple non-contiguous options, and even the iOS VoiceOver user and pointer users found it confusing. One participant using VoiceOver on macOS spent a frustrated 10 minutes trying to find the right combination of keys to make it work, despite the task being capped at 5 minutes. Another participant using NVDA summed it up perfectly:

This is a broken listbox. This is entirely broken, I don’t see any way of doing anything with this.

The best performing multi-select ended up being the one that displayed selected options as a list of buttons above the combobox. The version with inline buttons was pretty similar, but caused issues for low vision and mobility impaired pointer users due to the diminishing size of the text input.

I was surprised by the steep learning curve for the multi-select combobox with a comma-separated text value. Sighted users learned the format relatively quickly, though even they were more likely to try to use the space key rather than the comma to select the current option. That expectation was also strongly present for screen reader users, and without the aid of visual cues most of the blind participants did not pick up on the comma behavior.

General feedback

There was also some feedback related to general functionality, not specific to any select pattern.

• Enter and space: different users expected either enter or space to expand the options menu. There was a recurring thread of confusion around enter and space not expanding the options menu, and a high incidence of accidental form submission with the enter key.
• Arrow icon: one of the screen magnification users expected the arrow icon to be a button that expanded and collapsed the dropdown. None of the keyboard or screen reader users had that expectation, so the ideal solution would seem to be a pointer-only expand/collapse button.
• Dropdown placement: the switch user found it particularly challenging to scroll the options menu into view when it extended below the browser’s viewport. It would have been better to conditionally render the dropdown above or below the field based on relative position in the viewport.

Roll your own

All the different custom select behaviors tested here are potentially valid choices, even if one example didn’t perform as well in this study. Take the filtered combobox, for instance: there may well be a use case that only it would fill, such as a search input that displays suggestions based on user input. Contextual interaction requirements should overrule the more general recommendations made here.

That said, here are the final recommendations for markup and behavior for a simple single-select, editable combobox, and multi-select:

1. Single-select: a <select> element! (Has the repetition worked yet? Please come to my TED talk where I repeat “just use a select element” for 40 minutes straight.) If that doesn’t cut it, try a <div role="combobox">, and bypass using an <input> element altogether. Move selection automatically with focus, and preserve the selection on blur.
   See the full single-select reference implementation
2. Editable combobox: the final recommendation is a hybrid of two tested patterns: it should autoselect the first matching option based on user input, but not filter the options (unless there is a strong need for filtering).
   See the full combobox reference implementation
3. Multi-select combobox: if there aren’t a large number of options, a group of checkboxes is almost certainly the way to go. For use cases that have enough options to make a combobox pattern beneficial, the combobox with buttons tested the best.
   See the full multi-select reference implementation

Context matters

In general, the more complex a custom component, the more its usability will likely be context-dependent. A more app-like product that has heavy repeat use can afford to design custom components that trade a steeper learning curve for higher efficiency once learned. On the other hand, a website that has more first-time visitors might benefit from sticking to the more widely-known <select> element, checkboxes, and radios, even if a pro user would be faster with a combobox.

In short: design for your specific needs, always test with real people, and never, ever use <select multiple>!

I want to drop a quick thanks to my manager, Gianugo Rabellino, for letting me take what seemed like a pretty basic question (how to create an accessible combobox) and really run with it, my coworker Ryan Shugart for patiently trying out everything I sent his way and giving excellent feedback, and Microsoft for investing in accessibility-focused user research.