What an automated accessibility test can and cannot check

Three figures that measure different things

In 2017 the UK Government Digital Service ran ten tools against a test page with 143 known barriers. The best tool found 41% of them, the weakest 17%, and 29% were missed by every tool. The page was built to be broken, and the tools were those of 2017, but the conclusion still holds: tools cannot be relied on by themselves.

Deque, which makes the axe-core engine that Google Lighthouse also uses, reported in 2021 that automated rules covered 57% of the issues found in more than 2,000 of its own audits. That share is counted by volume: frequent problems such as low contrast weigh more. It is a vendor study.

The often quoted "20 to 30%" counts something else again: the share of WCAG success criteria a tool can test. These figures cannot be mixed in one sentence.

What a tool can decide on its own

Where a criterion is about something present or missing in the code, a rule can decide it for that page:

• images without alternative text, buttons and links without a name, form fields without a label
• text contrast below 4.5:1 (3:1 for large text), measured on the rendered page
• a missing or invalid lang attribute, a page without a title, duplicate IDs referenced by labels
• a meta viewport that blocks zoom or fixes the width, content that blinks or plays sound on its own
• CSS that removes the focus outline without a visible replacement, positive tabindex values

Even here the rule judges what it can see. Whether the alternative text describes the image well, or whether a heading really describes its section, is still a human question.

What only a person can judge

Other criteria depend on meaning, sequence or behaviour over time. No rule can tell whether:

• the reading order makes sense when the layout is removed (1.3.2)
• the keyboard focus can leave every widget again, for example a cookie dialog or a date picker (2.1.2)
• an error message suggests how to fix the input (3.3.3), or a payment step can be reviewed and corrected before it is sent (3.3.4)
• a status message such as "added to cart" reaches a screen reader without moving the focus (4.1.3)
• captions and audio descriptions match what is said and shown (1.2.x)

What we measured on real websites

On 21 October 2026 we combined the Lighthouse accessibility audit with Sitequiry's own checks of the HTML and the linked stylesheets on single pages of real websites, and counted for how many of the 55 WCAG 2.2 A and AA criteria an automated result existed: a problem, no problem found, nothing of that kind on the page, or not applicable.

The same 20 criteria were left to a person on both sites, among them reading order, keyboard traps, flashing content, error suggestions and status messages. Four of these 20 compare several pages: the same menu order, the same names for the same links, help on every page and more than one way to reach a page. One page cannot show them; a crawl of the whole site can.

"No problem found" is not a pass. It means the rules ran and found nothing; a person still confirms the criterion for the whole service.

How to use an automated test well

• Test one page of every template (home, category, product or article, form, checkout), not only the home page.
• Fix what the rules report first: those are certain problems and usually cheap to fix.
• Then walk through the main task with the keyboard only, and once with a screen reader (NVDA on Windows, VoiceOver on Mac or iPhone).
• Repeat the automated test after every larger change. It catches regressions that nobody looks for by hand.
• For a statement of conformity, an evaluation by a person is needed. A score of 100 in Lighthouse only says that the automated audits found nothing.

Frequently asked questions