Accountability in Accessibility Testing

While Accessibility Testing tools exist, all too often the testing process is incomplete, and many pages which claim conformance to a declared standard fail.

We've all worked hard, in our own ways, to educate, lead, instruct and guide those who come to the subject of Accessibility with "pure hearts". The emergence of numerous tools to assist in accessibility testing is testament that we have made some progress. We suspect then that the next topic for us to tackle is understanding and communicating what is involved in the testing process, and that the accountability of such testing must now also be addressed.

While the evaluation tools exist, all too often the testing process is incomplete, and many pages which claim conformance to a declared standard (be it W3C's WCAG, Section 508, or others) actually fail. This can occur for a variety of reasons, such as:

• a beginner with good intentions is not sure how to interpret the report output from testing tools
• false positives on the part of the evaluation tool
• the developer may not be able to complete the required manual verifications that these tools invariably flag as being required
• a once compliant page has changed, or new content has been introduced that was not present at testing time

We have discussed internally the need for accountability in accessibility testing as part of an overall strategy for Web Accessibility Testing, and we suggest that perhaps some pressure and active participation on the part of the WAI WG and members of Web Accessibility community could lead us to a possible solution.

We wonder out loud why the W3C's work on EARL (Evaluation and Report Language) and it's ability to document accessibility testing has not progressed further. A quick check on the W3C's EARL web page shows that there has been little to no activity in this area in some time. The last posted working draft is dated 6 December 2002 - a full year ago. Perhaps renewed activity in this area is required.

Consider the following: EARL is XML. P3P is XML.

Netscape 7.x, Mozilla, IE 6.x (and perhaps others?) now support the P3P initiative (a W3C Recommendation since April 2002), and expose the P3P statement for web pages which include them using the under-utilized <link > element. While enforcement of P3P policies is still a relatively new and often debated area, the framework exists. Why not look to add EARL reports to web pages/sites using the same methodology?

Organizations, companies and other entities which claim to support accessibility would now have a method to report their compliance beyond the Bobby icon and similar buttons, which have very little credibility or credence. Named accountability for accessibility would force developers to take the testing and evaluation process more seriously than clicking a button on a software application. The EARL report could/would clearly indicate which benchmarks/standards against which the page was tested, when the testing took place, and which tools and methodology were used to assess the resource. If you missed manual checks, the EARL report would indicate as such. When you have to sign your name to your work (perhaps using digital signatures) you are more likely to ensure that the report is accurate and thorough. Companies and other institutions could (should!) then make inclusion of EARL reports mandatory via Internal Policy, etc., thus closing the development loop.

This would also force software developers to address current shortcomings in their accessibility evaluation product or risk losing market share (the old carrot and stick method). If an evaluation tool fails to provide complete EARL reports as part of the reporting process (including the known requirement for manual checks), then the EARL report would not be available to link to the document...

What's required to make this a reality?

First, the W3C must see the EARL specification(s) move from a W3C Working Draft to an approved W3C Recommendation. (A W3C Recommendation is a specification or set of guidelines that, after extensive consensus-building, has received the endorsement of W3C Members and the Director. W3C recommends the wide deployment of its Recommendations. Note: W3C Recommendations are similar to the standards published by other organizations.) Following that, browser manufacturers need to implement a method to expose the EARL reports to users. Fortunately, as noted, this type of functionality is already being used for P3P reports. The addition of EARL reports should be relatively simple using the same techniques.

Next, Accessibility validation/testing tools must create the required EARL Reports -- already some of the current crop of tools do just that (HiSoftware's AccVerify^TM as well as WebThing's Accessibility Valet are two that come to mind)

Finally, developers must learn to do the required manual checks to ensure complete and accurate reports. Serious Web Accessibility practitioners already know and understand this; unfortunately this is not the norm but rather the exception. Perhaps this is a good next project for the W3C's WAI Education & Outreach Working Group.

Legislation in many territories around the world has made web accessibility a requirement. Education has convinced numerous companies, organizations and web developers on the ethical and financial reasons for achieving web accessibility. Accountability makes web accessibility both real and measurable. We believe the means to achieve this goal is both short and within our collective grasp.

Unless otherwise noted, this work is licensed under a Creative Commons License