Sex, lies and web accessibility: the use of accessibility logos and statements on e- commerce and financial websites

An investigation was conducted in the use of accessibility logos and statements in a large sample of e-commerce and financial websites in a number of countries. Firstly, a survey of the use of logos and statements on these websites was undertaken. For a selection of websites that displayed logos or provided statements, their accessibility was estimated using automated checking of their conformance to the Web Content Accessibility Guidelines (WCAG, Version 1.0) and user testing by people with disabilities. This allowed us to assess the accuracy of the claims made by the websites and compare their accessibility with previous research, particularly the recent survey of website accessibility undertaken by the Disability Rights Commission. The use of accessibility logos and statements on e-commerce and financial websites was not very high. Only 4% of e-commerce sites and 12% of financial sites had a logo or statement or a combination of the two. The accessibility of a sample of the sites with logos/statements was higher than expected when assessed with automated testing, when compared to the large sample of general websites tested for the DRC study, but not as high as with large samples from the museum, library and archive sector and the university sector. When the accessibility of the sites was assessed by disabled users, the accessibility was very similar to the general sample assessed for the DRC study. Thus, although these websites are stating that they are making particular efforts for disabled users, they are not any more accessible than a random sample of websites. More worrying, the sites are making exaggerated claims about their accessibility, with 30% of sites overstating their level of conformance to WCAG. The reasons for such a situation are explored.


INTRODUCTION
There are a number of logos that websites can display to show that they are accessible to users with disabilities.These include logos that only require self-certification e.g. the World Wide Web Consortium's (W3C) Web Content Accessibility Guidelines (WCAG) conformance logos [1] and the "Bobby Approved" logo [2], and logos that require an audit from an external organization e.g. the Royal National Institute for the Blind's "See it Right" logo [3].A range of these logos is shown in Table 1, below.Websites can also include statements about their achieved or aspirational level of accessibility.The use of such logos and statements is of interest for two reasons.Firstly do they currently provide an accurate and useful indication for users with disabilities that these websites will be accessible?Secondly, given that there are currently proposals to create a European quality mark for web content accessibility [4], what can the current situation tell us that might be useful for such a proposal.
Accessible Design in the Digital World Conference 2005 The current research investigated the use of accessibility logos and statements in a large sample of e-commerce and financial websites in a number of countries.Firstly, a survey of the use of logos and statements on these websites was undertaken.For a selection of websites that displayed logos or provided statements, their accessibility was estimated using automated checking of their conformance to the Web Content Accessibility Guidelines (WCAG, Version 1.0) [5] and user testing by people with disabilities.This allowed us to assess the accuracy of the claims made by the websites and compare their accessibility with previous research, particularly the recent survey of website accessibility undertaken by the Disability Rights Commission (DRC) [6].

A SURVEY OF THE USE OF E-COMMERCE AND FINANCIAL WEBSITES
For the survey of e-commerce websites, 250 sites either based in the UK (i.e. using the .co.uk domain) or clearly selling to the UK (e.g..comdomain sites, but quoting prices in pounds sterling) were sampled.Eight areas within ecommerce were targeted: sites selling books, clothing, CDs/DVDs, electronic goods and computers, home and garden products, shoes and accessories, travel sites (flights, hotels, car hire etc) and the sites of department or large multiproduct stores.Sites were found by consulting portal sites and conducting Google searches with the area keyword and ".co.uk" or ".com".
For the financial websites, 250 sites from the English-speaking world (Australia, Canada, Ireland, New Zealand, U.K., and U.S.A.) were sampled.Five areas within finance were targeted: banks (including High Street banks, online only banks, and commercial banks), building societies (and in Australia credit unions), credit card companies, financial advisors and insurance companies.Again sites were found by consulting portal sites and conducting appropriate Google searches.
For each site sampled, the site was thoroughly searched for any accessibility logos and statements by navigating around the site, consulting the site map if available and as a last resort, conducting searches with the terms "accessibility", "disabled" and "blind" (this often revealed that the organizations sponsored many disability related activities, even if their website was not accessible to disabled customers).
Of the 250 e-commerce websites, 10 sites (4%) had an accessibility statement.Statements sometimes made explicit claims about the accessibility of the site in terms of WCAG conformance, (e.g."Our pages conform at a minimum to Level A compliance as specified by the Web Content Accessibility Guidelines … The majority of Level AA and AAA requirements are also met") or sometimes made more general aspirational statements (e.g."We are working to make the X site as accessible as possible to be used by everybody.You will notice changes to the site over the coming months to help with navigation, reading text, colour schemes and use of screen readers").4 of these sites (1.6%) also had an accessibility logo.These included the self-certification Bobby Approved and W3C logos, and the externally validated RNIB "See It Right" logo.Several sites displayed more than one of the logos.No sites had a logo without having a statement.Table 2, below, shows the differences in the frequency of logos and statements between the different e-commerce areas.Three areas, books, home and garden, and shoes and accessories websites had no websites in the sample with logos or statements.On the other hand, over 10% of department store websites sampled had a logo or statement.
Accessible Design in the Digital World Conference 2005

AUTOMATED ACCESSIBILITY TESTING
20 of the websites with accessibility logos and/or statements (10 e-commerce and 10 financial) were selected for automated accessibility testing.Testing was conducted with Bobby Desktop Version 5.0 from Watchfire [7]. 10 key pages from each website were tested for conformance with those WCAG1 checkpoints that Bobby is able to check.
30% of websites (6 out of 20) passed all the automated Priority 1 tests, achieving Level A conformance, but none passed the automated Priority 2 or 3 tests to achieve Level A or AA conformance.This is substantially better than the results of the DRC study conducted in 2003 [6], when only 19% of a sample of 1000 websites passed all the automated Priority 1 tests (although a different tool was used for the DRC study, the accessibility module of WebXM, also from Watchfire [7]).However, this hardly indicates a high level of conformance to WCAG -the UK government target is for all public websites to achieve Level AA conformance [8].Indeed, a more recent survey of 325 websites in the museum, library and archive sector [9] found that the homepages of 41.6% of websites passed Level A conformance, and a study of UK university websites [10] in 2002 found that the homepages of 43.2% passed this Level.
In addition, when compared to the claims made by the use of accessibility logos and statements, it is clear that there is exaggeration of WCAG conformance claims: of the 20 sites investigated, 3 are claiming one level of conformance more than they actually achieve and 3 are claiming two levels of conformance more than they actually achieve.That means that 30% of the websites sampled with accessibility logos and/or statements are exaggerating their level of conformance, whether knowingly or perhaps without realising it.40% of websites, although they are displaying an accessibility logo or making an accessibility statement, are not making a specific claim for a level of conformance (for example they have wording such as "we are working towards Level AA accessibility").This leaves only 30% of websites sampled that are making accurate claims about their conformance levels.In addition, it needs to be remembered that the testing conducted was only with an automated tool, so sites might also fail a particular level of conformance on manual checking.The DRC study found that of 6 sites that passed AA conformance on automated Accessible Design in the Digital World Conference 2005 testing, only 2 passed the full manual assessment of AA conformance.So the level of exaggeration of the claims of accessibility logs and statements might be somewhat greater than indicated here.

USER TESTING BY PEOPLE WITH DISABILITIES
Based on the results of the automated WCAG testing, six websites were chosen for user testing.Three disabled people undertook the testing: a blind person using a screenreader, a partially sighted person using a screen magnification program and a dyslexic person, tested the sites.This is a very small sample of disabled people (time and resources were very limited), and can only give some indication of the accessibility of the sites as actually experienced by users, not a full evaluation.
On each site the testers initially browsed around the site and then were asked to complete two tasks which had been selected as being of key importance to using the site.In total 30 tasks were attempted in this evaluation.The target was 36 tasks (6 sites x 3 users x 2 tasks), but the blind user found the tasks so time consuming that she only undertook one task per site (an interesting indication of the accessibility of the sites in itself).
The overall success rate at the tasks was 73%.The breakdown for the different user groups is shown in Table 4.These are comparable with the corresponding success rate from the DRC study of 71%, but if the current sample of websites have made specific efforts at accessibility for people with disabilities, one might have hoped for a somewhat higher success rate.This is particularly true in light of the fact that a recent study on the usability of a sample of the sites tested by the DRC found that a matched sample of non-disabled people achieved a success rate of 96.7% [9]. 1.This total success rate is different from the total success rate reported by the DRC, as it has been calculated for blind, partially sighted and dyslexic participants only.
Participants were also asked to rate the ease of completing the task rating on a 7 point scale, where 1 means not at all easy and 7 means very easy.The mean ratings for the different user groups are shown in Table 5, below.For blind and dyslexic participants, the mean ease of task completion ratings for the websites in the current study were no different from the ratings by blind and dyslexic participants in the DRC study (for blind participants: t 5 = 0.73, n.s.; for dyslexic participants: t 6 = 1.79, n.s.).However, for partially sighted participants, the websites in the current study were rated significantly worse than those in the DRC study (t 10 = 3.96, p < 0.005).

DISCUSSION
This study has found that the use of accessibility logos and statements on e-commerce and financial websites is not very high.Only 4% of e-commerce sites and 12% of financial sites had a logo or statement or a combination of the two.The accessibility of a sample of the sites with logos/statements was higher than expected when assessed with automated testing, when compared to the large sample of general websites tested for the DRC study, but not as high as with large samples from the museum, library and archive sector and the university sector.When the accessibility of the sites was assessed by disabled users, the accessibility was very similar to the general sample assessed for the DRC study.Thus, although these websites are stating that they are making particular efforts for disabled users, they are not any more accessible than a random sample of websites.More worrying, the sites are making exaggerated claims about their accessibility, with 30% of sites overstating their level of conformance to WCAG.In spite of the title of this paper, these results do not mean that the owners of these websites are actually lying when they use accessibility logos or statements.There are two possibilities.Firstly, it is possible the site owners place logos or statements on their sites without fully understanding their implications.For example, they may think that if their site has passed an automated test of a particular conformance Level, then the site has passed all the checkpoints at that level, not just the automated ones.Or they may think that if a sample of pages, or even just the home page, passed a particular level of conformance, it means that the website can display the logo/statement.All these views have been encountered recently amongst website owners.Clearly all these problems only affect the self-certification logos such as the W3C and Bobby Approved logo.These possibilities suggest that an externally validated quality mark, such as the one being considered by the European Commission [4] is very important.The second possibility is that when the website was tested for accessibility, it did pass the level of conformance stated, but that no quality control procedures have been implemented to ensure that accessibility is maintained.As many people interested in accessibility have discovered (present authors included), it only takes one person who has access to a site to put up some interesting photos but forget to add descriptions using the "alt text" attribute, to destroy the accessibility level of the site.This is a problem for both self-certification and externally validated logos, unless the external validation process included control procedures that make reasonable efforts to maintain the accessibility of the site.

Accessible
Design in the Digital World Conference 2005 Accessible Design in the Digital World Conference 2005

Table 3 ,
above, shows the differences in frequency of logos and statements between the different financial areas.Only one area, credit card companies, had no examples of logos or statements.On the other hand, nearly 20% of banks and nearly 40% of insurance companies had a logo or a statement or both.