Provide standardized way to turn JSON export into VPAT/custom audit report/etc.

[mra11yx]

Is your feature request related to a problem? Please describe.

The JSON export feature is currently only designed to facilitate re-importing a test into Accessibility Insights, rather than to export a VPAT or an audit report that could be used for monitoring the test results of a product over time.

I'm spinning this ticket off of #653

Describe the desired outcome

I'd love to see Insights' export feature include a list/data set of each WCAG success criterion, with each test, and that test's pass/fail/untested state, tied to the relevant success criterion/a.

This would make it easier to use Insights for true audit work, as testers could evaluate web content and build out VPAT-esque documents—as well as generate some kind of evaluation score, if appropriate—without having to understand each success criterion in depth.

Note: Although I'm focusing on manual tests here, it would probably also make sense to have any exported WCAG data also include any automated test results. If a tester wanted to have separate manual and automated scores, they could run an automated scan separate from a manual assessment.

Example flow:

1. Tester runs through all tests
2. Tester exports the document
3. Product owner imports the JSON file into their own software/web application/etc. to create a VPAT, score, or other piece of data to be used for accessibility scoring and monitoring

Describe alternatives you've considered

It's currently possible to load a .a11ywebassessment file into a webpage, parse it for passes, failures, and "not tested" values, map those to relevant WCAG success criterion/a, and output a file with the resulting data. However, this technique is brittle and will break if the structure of Insights' exported .a11ywebassessment files changes, or if tests are added, removed, or remapped to different WCAG success criteria.

Additional context

Certain auditing products sold by accessibility companies already provide features similar to this.

[ghost]

This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!

[ferBonnin]

Thanks for the feature request @mra11yx! As you mentioned, this is related to part 2 of #653 and after reviewing with the team it sounds like an interesting scenario.

We would like to have a consensus within the community for the format for this file; do you have a sample or suggestion for how this should look like? 🙂

[ghost]

The team requires additional author feedback; please review their replies and update this issue accordingly. Thank you for contributing to Accessibility Insights!

[mra11yx]

Hi @ferBonnin, I'm thinking something like this (could potentially be in a format other than JSON, but I'm using this for simplicity's sake):

{
  412NameRoleValue: {
    //info about success criterion, for data presentation purposes
    wcagInfo: {
      num: "4.1.2",
      name: "Name, Role, Value",
      level: "A",
      status: "fail"
    }
    //tests, with pass/fail information
    roleStateProperty: {
      status: "fail",
    //instances for each test, if a test has these (the "role, state, property" test is an example of one that does), so that data can be broken down further if needed
      instances: {
        instance0: {
          pattern: "combobox",
          accessibleName: "Search",
          status: "pass"
        },
        instance1: {
          pattern: "accordion",
          accessibleName: "More settings",
          status: "fail",
          notes: "The 'more settings' accordion control does not convey an expanded or collapsed state to assistive technologies."
        }
     //other instances
      }
  //other tests for this success criteria
  }
//other success criteria
}

[ghost]

This issue requires additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, we will update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!

[mgifford]

Would be really nice to be able to integrate it with this machine-readable accessibility compliance report format -https://github.com/GSA/open-product-accessibility-template

[ferBonnin]

Marking this as active since are starting the feature work. The JSON will follow a simple format with the WCAG number and the requirements under that WCAG.

[mgifford]

Is it possible to use the ID alone? OPAT is going with id: 2.1.2 in YAML, but it would be easier to map to the numeric ID than a combination of number and name.

We've tried to align with the W3C's by listing an id & alt_id

      - id: 2.1.2
        handle: No Keyboard Trap
        alt_id: keyboard-operation-trapping

In our catalog here:
https://github.com/GSA/open-product-accessibility-template/blob/main/catalog/2.4-edition-wcag-2.0-508-en.yaml

We've got some other datasets here that might be convenient to repurpose.
https://github.com/GSA/open-product-accessibility-template/tree/main/data

Unfortunately, the W3C WAI does not yet provide a central list of JSON files that they are maintaining. Not yet in any case.

That said, still useful to align with their practice.

[ferBonnin]

Is it possible to use the ID alone? OPAT is going with id: 2.1.2 in YAML, but it would be easier to map to the numeric ID than a combination of number and name.

@mgifford yes, we can have the WCAG Number separated from the WCAG Name

[ghost]

The team requires additional author feedback; please review their replies and update this issue accordingly. Thank you for contributing to Accessibility Insights!

[ferBonnin]

This new feature is now live in the latest version of Accessibility Insights for Web. Please let us know what you think! 🙂

[ghost]

This issue has been automatically marked as stale because it is marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thank you for contributing to Accessibility Insights!

[mgifford]

@ferBonnin two observations:

1. You can't export it from the Fast Pass. That dropdown only appears when you are on the Assessment tab. Would be good if both exports offered the same pattern.
2. I tried to import this into WCAG-EM and it didn't work. Is it in a JSON-LD format that should work with common EARL implementations?

Just remembered that you were hoping this would be a community contribution..
#4331

[ferBonnin]

1. This feature request talked about assessment in particular. Also, although the automated checks in FastPass are extensive, they do not cover full WCAG so we cannot make a pass / fail determination for that S.C.
2. Please check out or FAQ on the JSON Schema we use in Assessment