Add an optional layer of VPAT conformance ratings

[mitchellevan]

Story:

As an evaluator reporting WCAG results for a single product sold to international markets, I want to save time by using the WCAG-EM report generator as a decision support tool for the "Conformance Level" ratings in the VPAT WCAG tables.

International markets expect certain formats:

• Traditionally some US buyers, such as the US federal government and education institutions, expect WCAG results in ITI's VPAT format.
• In other regions where accessibility is a factor in procurement, such as EU, Canada, and Australia, WCAG is a standard but most buyers have not yet stated expectations for a reporting format.

Proposed solution:

• Keep the current "Results for entire sample" options (Not Checked, Passed, Failed, Not Present, Cannot Tell). These are a fine way to describe WCAG conformance.
• Allow users to enable an additional set of option lists, called "VPAT Conformance Level". The options are: Supports, Partially Supports, Does Not Support, Not Applicable, and Not Evaluated.
• Provide instructions for appropriate choices of values, based on the "Terms" sub-section of the "Essential Requirements for Authors" section of ITI's VPAT templates.
• Similar to Warn when then criteria result is different from the result of the pages #149, warn the user if their choice of a "VPAT Conformance Level" value is inconsistent with the "Results for entire sample" value.

The following pairs of values are consistent.

VPAT Conformance Level	Results for Entire Sample
Supports	Passed
Supports	Not Present
Partially Supports	Failed
Does Not Support	Failed
Not Evaluated	Not Checked - WCAG Level AAA criteria only

The "Not Applicable" value should always receive a warning.

[mitchellevan]

I originally called this issue "Overlay VPAT conformance ratings". I later noticed the unwanted implications of "overlay", so I changed the issue title.

[juleskuehn]

Like you, I'd like to see the tool produce both a detailed issues report and a summary VPAT report.
I would prefer the tool automatically populate the VPAT Conformance Level column, with warnings. "Partially Supports" can be automatically awarded when more than 50% of the sample is "Passed", but this should always receive a warning.
The relationship between WCAG "Not Applicable" and VPAT "Not Applicable" is troublesome. You can fairly say in WCAG that a SC which is "Not Applicable" is "Passed". As a result, WCAG "Not Applicable" can be said to correspond with VPAT "Supports". However, it is more useful to say "Not Applicable" in the VPAT for clarity. So I prefer that mapping.
The tool could also automatically populate the VPAT comments column, although these should always be manually reviewed.

[mitchellevan]

@juleskuehn I mostly agree with how to complete a VPAT. Nits:

The relationship between WCAG "Not Applicable" and VPAT "Not Applicable" is troublesome.

Did you mean "Not Present" in the WCAG-EM Report Tool? I prefer to follow ITI's instructions to avoid "Not Applicable" in VPAT WCAG tables, though it's fine in Chapter 4 Hardware, Chapter 5 Software, etc.

I would prefer the tool automatically populate the VPAT Conformance Level column, with warnings. "Partially Supports" can be automatically awarded when more than 50% of the sample is "Passed", but this should always receive a warning.

I would also prefer more automation, but we need to be cautious before locking this kind of logic into code too soon. If it's were just you and me deciding, I'm sure we could hash out other scenarios, such as "Does Not Support" caused by failure of CR 3 Complete Processes. But VPAT is not just for the US government anymore. I would rather suffer a bit more inconsistency in the short run, and allow time for more countries to sort out their ICT procurement expectations.

[mitchellevan]

My unstated assumption was that WCAG reporting is for international markets, yet my "story" in the issue originally mentioned only US buyers. I've edited the issue to clarify.

[SteveBarnett]

This would be really useful for us at Totara Learning. Many of our clients work with / in the public sector (in the UK, EU, and US) and often ask about an Accessibility Conformance Report in VPAT format.

At the moment we fill out our ACR (international edition, to go as wide as we can) manually from our own testing and (Jira) tickets. Being able to automate it a bit and connect it to our testing against the SC more closely would be really great.

[mgifford]

Hopefully we'll have an editor that is more geared to Sectio 508 & VPAT like results soon. https://github.com/GSA/open-product-accessibility-template

Just adding this as a thread here. Think everyone on this thread is already aware of OPAT.