Types of Accessibility Testing
There are four different types of accessibility testing:
- Automated testing
- Manual code review
- UX review
- User testing
The time you allocate to each of these will vary depending on the type of digital content you’re testing, as well as whether the asset is in development or live on the web. The chart below offers an example of what the testing and review process may look like for common digital assets, broken down by testing or review types:
| Digital Asset | % automated testing | % UX review | % manual code review | % user testing |
|---|---|---|---|---|
| Existing website | 20% | 10% | 50% | 20% |
| New website | 10% | 30% | 40% | 20% |
| Existing app | 10% | 60% | 0% | 30% |
| New app | 10% | 60% | 0% | 30% |
| PDFs | 70% | 0% | 20% | 10% |
Step 1: Automated Testing
Automated testing is an essential starting point for evaluating your site’s current level of accessibility. It lets you monitor key pages and conduct broad assessments of your whole site without having to look at every page individually. For a retailer with hundreds or thousands of templated product pages, automated testing is a good way to get started without feeling overwhelmed.
Here’s a checklist of considerations for choosing an effective automated testing solution:
- It tests your whole site, including login, shopping cart, user journeys, and other pages with dynamic content or functionality.
- It provides analysis of your accessibility status with real data to back it up.
- It provides results that are easily interpreted by all members of your team, including project managers, developers, QA testers, marketers, and your legal team.
- It easily integrates with your existing tech stack.
There are many easy-to-use automated testing tools, depending on your digital assets, including the following:
- For websites: Siteimprove will scan your site and provide a report with a summary of the basic errors, breaking them down by type and location on the page.
- For mobile apps: Apple’s VoiceOver, Android’s technical guide, and Accessibility Scanner all can help spot accessibility errors in your apps.
- For PDFs: Adobe Acrobat provides user-friendly tutorials on how to create and verify documents for accessibility and how to fix issues when you encounter them.
Step 2: Manual Code Review
Although automated testing is an essential part of web accessibility, it doesn’t cover all WCAG criteria. Whether free or commercial, these tools can only take you so far—a significant portion of the success criteria requires manual review. For example, an automated tool can identify alt text, but it can’t tell you whether that alt text is relevant for accessibility.
Where does Automatic Accessibility Testing fall short?
WCAG 2.0 and 2.1 establishes the guidelines and suggests the techniques to test pages through a number of Success Criteria at each level of conformance (A, AA, or AAA). An automated test can only be used for a small number of these Success Criteria, leaving the majority of criteria requiring a manual review. Many automatic testing tools try to overstate their completeness by stating how many automatic tests it can run, but rarely do they mention the number of Success Criteria that can NOT be fully tested automatically leading to misconceptions and incomplete results.
| Level A | WCAG | Fully Automatic | Auto & Manual Verification Needed | All Manual | Testing Workflow |
|---|---|---|---|---|---|
| 1.1.1 Non-text Content | 2.0 & 2.1 | X | AQA* & verify text & screen reader | ||
| 1.2.1 Audio-only and Video-only (Prerecorded) | 2.0 & 2.1 | X | Test with sound/screen off | ||
| 1.2.2 Captions (Prerecorded) | 2.0 & 2.1 | X | Test with sound/screen off | ||
| 1.2.3 Audio Description or Media Alternative (Prerecorded) | 2.0 & 2.1 | X | Test with sound/screen off | ||
| 1.3.1 Info and Relationships | 2.0 & 2.1 | X | AQA & verify text & verify code & screen reader | ||
| 1.3.2 Meaningful Sequence | 2.0 & 2.1 | X | Keyboard & verify code & screen reader | ||
| 1.3.3 Sensory Characteristics | 2.0 & 2.1 | X | Test with sound/screen off | ||
| 1.4.1 Use of Color | 2.0 & 2.1 | X | Visual verification | ||
| 1.4.2 Audio Control | 2.0 & 2.1 | X | AQA & Keyboard & Screen Reader | ||
| 2.1.1 Keyboard | 2.0 & 2.1 | X | AQA & Keyboard & Screen Reader | ||
| 2.1.2 No Keyboard Trap | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 2.1.4 Character Key Shortcuts | 2.1 | X | Keyboard & Screen Reader | ||
| 2.2.1 Timing Adjustable | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 2.2.2 Pause, Stop, Hide | 2.0 & 2.1 | X | AQA & Keyboard & Visual verification | ||
| 2.3.1 Three Flashes or Below Threshold | 2.0 & 2.1 | X | Visual verification | ||
| 2.4.1 Bypass Blocks | 2.0 & 2.1 | X | AQA | ||
| 2.4.2 Page Titled | 2.0 & 2.1 | X | AQA & verify text | ||
| 2.4.3 Focus Order | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 2.4.4 Link Purpose (In Context) | 2.0 & 2.1 | X | AQA & verify text & screen reader & verify code | ||
| 2.5.1 Pointer Gestures | 2.1 | X | Keyboard & Screen Reader & gestures | ||
| 2.5.2 Pointer Cancellation | 2.1 | X | Keyboard & Screen Reader & mouse events | ||
| 2.5.3 Label in Name | 2.1 | X | Keyboard & Screen Reader & verify code | ||
| 2.5.4 Motion Actuation | 2.1 | X | Keyboard & Screen Reader & real motions | ||
| 3.1.1 Language of Page | 2.0 & 2.1 | X | AQA | ||
| 3.2.1 On Focus | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 3.2.2 On Input | 2.0 & 2.1 | X | AQA & keyboard & Screen Reader | ||
| 3.3.1 Error Identification | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 3.3.2 Labels or Instructions | 2.0 & 2.1 | X | AQA | ||
| 4.1.1 Parsing | 2.0 & 2.1 | X | AQA | ||
| 4.1.2 Name, Role, Value | 2.0 & 2.1 | X | AQA & verify code | ||
| WCAG 2.0 – TOTAL Level A | 25 | 4 | 9 | 12 | |
| WCAG 2.1 – TOTAL Level A | 30 | 4 | 9 | 17 | |
| Level AA | WCAG | Fully Automatic | Auto & Manual Verification Needed | All Manual | |
| 1.2.4 Captions (Live) | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 1.2.5 Audio Description (Prerecorded) | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 1.3.4 Orientation | 2.1 | X | Keyboard & Screen Reader | ||
| 1.3.5 Identify Input Purpose | 2.1 | X | Keyboard & Screen Reader & code verification | ||
| 1.4.3 Contrast (Minimum) | 2.0 & 2.1 | X | AQA & visual verification & code verification | ||
| 1.4.4 Resize text | 2.0 & 2.1 | X | AQA & visual verification | ||
| 1.4.5 Images of Text | 2.0 & 2.1 | X | Visual verification & verify code | ||
| 1.4.10 Reflow | 2.1 | X | Visual verification | ||
| 1.4.11 Non Text Contrast | 2.1 | X | Visual verification & code verification | ||
| 1.4.12 Text Spacing | 2.1 | X | Visual verification | ||
| 1.4.13 Content on Hover or Focus | 2.1 | X | Keyboard & Screen Reader & mouse events | ||
| 2.4.5 Multiple Ways | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 2.4.6 Headings and Labels | 2.0 & 2.1 | X | AQA & verify text | ||
| 2.4.7 Focus Visible | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 3.1.2 Language of Parts | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 3.2.3 Consistent Navigation | 2.0 & 2.1 | X | Visual verification & verify code | ||
| 3.2.4 Consistent Identification | 2.0 & 2.1 | X | Keyboard & Screen Reader & visual verification & verify code | ||
| 3.3.3 Error Suggestion | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 3.3.4 Error Prevention (Legal, Financial, Data) | 2.0 & 2.1 | X | Keyboard & Screen Reader | ||
| 4.1.3 Status Messages | 2.1 | X | Keyboard & Screen Reader | ||
| WCAG 2.0 – TOTAL Level AA | 13 | 0 | 3 | 10 | |
| WCAG 2.1 – TOTAL Level AA | 20 | 0 | 3 | 17 | |
| WCAG | Fully Automatic | Auto & Manual Verification Needed | All Manual | ||
| WCAG 2.0 – TOTAL Level A and AA | 38 | 4 | 12 | 22 | |
| WCAG 2.1 – TOTAL Level A and AA | 50 | 4 | 12 | 34 |
Step 3: UX Review
During user experience review, it’s important to consider accessibility. Although accessibility is often confused with usability, the two terms are quite different:
- Accessibility generally refers to the adherence to standards, such as WCAG, when creating a website, app, or document. This can include everything from color and text to images and multimedia.
- Usability describes the actual experience of using the site, including user interface design.
The good news is that accessibility improvements can raise the level of usability for all users, regardless of disability. Another difference between the two is that usability testing can involve input from real people, not automation. This is where user testing comes in.
Step 4: User Testing
User testing should involve both usability experts and those who use assistive technology in their everyday lives. Companies across the globe recognize that automated tools don’t always catch every WCAG issue, and they are responding to the rising demand from the disability community for user testing.
Additionally, user testing makes good business sense. Today, you wouldn’t launch a website without making sure it works on a mobile device, so why launch a site or other digital asset without sign-off from a community of people who will be using it?
To launch a user testing program:
- Start in your own backyard. Look for local lighthouse groups or nonprofits who work with disabled communities.
- Bring in the experts. Make sure they can move at your pace and charge a price you can afford.
- Look inward. There may be people in your own company who are disabled and use assistive technology.
- Recruit your own testers using databases like Knowbility.