Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate visual testing mechanism to the latest KDS version #901

Merged
merged 62 commits into from
Jan 29, 2025
Merged
Show file tree
Hide file tree
Changes from 59 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
0236bf6
Add and configure percy for visual tests
KshitijThareja Jun 21, 2024
2434700
Change server configuration and file names for visual tests
KshitijThareja Jun 29, 2024
b3f0bb4
Add component rendering mechanism for visual testing
KshitijThareja Jun 30, 2024
7d55f4b
Remove unused packages
KshitijThareja Jun 30, 2024
c91086f
Add JSDoc for testing playground and modify server script
KshitijThareja Jul 2, 2024
e251350
Update KImg unit tests and eslint configuration
KshitijThareja Jul 6, 2024
cefdfe6
Use concurrently to manage startup and termination of tests
KshitijThareja Jul 9, 2024
63d5144
Update waitForServer function to use http
KshitijThareja Jul 10, 2024
b6044ad
Remove unused dependencies and separate visual tests from yarn test
KshitijThareja Jul 10, 2024
fc09854
Separate visual tests from yarn test command
KshitijThareja Jul 10, 2024
cf41d8c
Merge pull request #670 from KshitijThareja/visual-testing
bjester Jul 10, 2024
6c3fd69
Introduce abstraction for component rendering functions and support f…
KshitijThareja Jul 11, 2024
985c0d4
Initial workflow config
KshitijThareja Jul 14, 2024
e5d792d
Fix the initial workflow for visual tests job
KshitijThareja Jul 16, 2024
8823a8f
Merge pull request #677 from KshitijThareja/vt-add-concurrently
bjester Jul 16, 2024
cc9417c
Add automatic comments job for visual testing workflow
KshitijThareja Jul 17, 2024
2ef82b1
Fix lint errors
KshitijThareja Jul 17, 2024
9ad6f8f
Abstraction for page navigation
KshitijThareja Jul 17, 2024
1b024f9
Fix lint errors
KshitijThareja Jul 17, 2024
4454b60
Introduce visual testing to the existing Javascript tests workflow
KshitijThareja Jul 17, 2024
8119ec9
Replace usage of custom describe blocks with test regex patterns
KshitijThareja Jul 20, 2024
e1a54ba
Remove unintentional changes
KshitijThareja Jul 20, 2024
f55a831
Add comments for rendering function and make variable names clear
KshitijThareja Jul 23, 2024
475827f
Remove run server step
KshitijThareja Jul 23, 2024
e325e34
Remove pipes where not required
KshitijThareja Jul 24, 2024
622a22c
Add special test blocks for visual tests to append Visual tag automat…
KshitijThareja Jul 26, 2024
55bff15
Merge pull request #685 from KshitijThareja/vt-abstraction
AlexVelezLl Jul 26, 2024
02d0dbb
Add support for default slots rendering
KshitijThareja Jul 31, 2024
7c117e7
Change workflow to run on separate environment
KshitijThareja Aug 6, 2024
2d2a9bd
Merge branch 'gsoc/visual-testing' into vt-ghaction
KshitijThareja Aug 6, 2024
8c1cfdb
Add option to specify custom dimensions for snapshots
KshitijThareja Aug 6, 2024
1e3c790
Add puppeteer config for CI environments
KshitijThareja Aug 6, 2024
a0423f5
Remove .puppeteerrc.cjs
KshitijThareja Aug 7, 2024
5a76c43
Checkout pr head
AlexVelezLl Aug 7, 2024
563fd8f
Update should skip to include style changes
AlexVelezLl Aug 7, 2024
3fbaf41
Initial documentation for visual tests
KshitijThareja Aug 9, 2024
1e664e2
Update comment job to use github-script action
KshitijThareja Aug 9, 2024
e61a2de
Update visual testing docs with links to mentioned files
KshitijThareja Aug 9, 2024
348e47c
Update documentation
KshitijThareja Aug 13, 2024
e31e203
Updat comment job to use utils file
KshitijThareja Aug 13, 2024
690cfd8
Migrate findComment and generateComment functions to githubUtils.js
KshitijThareja Aug 13, 2024
21a0d46
Add reference to documentation in getting-started doc
KshitijThareja Aug 13, 2024
a6d1312
Update jsdoc
AlexVelezLl Aug 14, 2024
c5e0df5
Merge branch 'gsoc/visual-testing' into vt-test
AlexVelezLl Aug 14, 2024
1c7ec1e
Merge branch 'gsoc/visual-testing' into vt-test
AlexVelezLl Aug 14, 2024
7d6c634
Modify testing playground to render named and default slots in a sing…
KshitijThareja Aug 15, 2024
09820a1
Add jsdocs for renderComponent and takeSnapshot functions
KshitijThareja Aug 15, 2024
05063ac
Fix lint error
KshitijThareja Aug 15, 2024
aaa4411
Remove async from describe block
KshitijThareja Aug 15, 2024
6be488b
Add snapshot options specifying test widths
KshitijThareja Aug 16, 2024
15d8ae5
Merge pull request #710 from KshitijThareja/vt-test
AlexVelezLl Aug 16, 2024
1f09d07
Update test command to work for all test files
KshitijThareja Aug 18, 2024
083532a
Add additional info for rendering complex components
KshitijThareja Aug 20, 2024
86d7402
Add link to percy documentation
KshitijThareja Aug 21, 2024
e4b8986
Update info for usage of test blocks for visual tests
KshitijThareja Aug 21, 2024
c8e4d0e
Update required node version specification
KshitijThareja Aug 22, 2024
b0cc1e7
Fix errors in integrating visual testing setup with the codebase
KshitijThareja Jan 20, 2025
924f782
Update percy and puppeteer to latest versions
KshitijThareja Jan 20, 2025
d07775a
Update package.json
KshitijThareja Jan 21, 2025
7b7395d
Update visual test description for KButton tests and fix http import
KshitijThareja Jan 23, 2025
78324ba
Resolve merge conflicts with the develop branch
KshitijThareja Jan 23, 2025
d4d12b1
Merge branch 'develop' into gsoc/vis-testing
KshitijThareja Jan 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,14 @@ esLintConfig.settings['import/resolver'].nuxt = {
nuxtSrcDir: 'docs',
};

// Remove linting errors for the globals defined in the jest-puppeteer package and testUtils
esLintConfig.globals = {
...esLintConfig.globals,
page: true,
browser: true,
context: true,
puppeteerConfig: true,
jestPuppeteer: true,
};

module.exports = esLintConfig;
43 changes: 43 additions & 0 deletions .github/githubUtils.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
async function generateComment(percyUrl) {
return `
### Percy Visual Test Results

**Percy Dashboard:** [View Detailed Report](${percyUrl})

**Environment:**
- **Node.js Version:** 18.x
- **OS:** Ubuntu-latest

**Instructions for Reviewers:**
- Click on the [Percy Dashboard](${percyUrl}) link to view detailed visual diffs.
- Review the visual changes highlighted in the report.
- Approve or request changes based on the visual differences.
`;
}

async function findComment(github, context, issue_number) {
let comment;
let page = 1
while (!comment) {
const request = await github.rest.issues.listComments({
issue_number,
owner: context.repo.owner,
repo: context.repo.repo,
page,
})
const comments = request.data
if (!comments.length) {
return;
}
comment = comments.find(c => c.body && c.body.includes('### Percy Visual Test Results'));
if (comment) {
return comment.id.toString()
}
page += 1;
}
}

module.exports = {
findComment,
generateComment,
}
109 changes: 109 additions & 0 deletions .github/workflows/visual_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
name: Percy Visual Tests

on: [pull_request_target]

jobs:
pre_job:
name: Path match check
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
should_skip: ${{ steps.skip_check.outputs.should_skip }}
steps:
- id: skip_check
uses: fkirc/skip-duplicate-actions@master
with:
github_token: ${{ github.token }}
paths: '["**.vue", "**.js", "**.css", "**.scss", "yarn.lock"]'

visual_tests:
name: Frontend Visual Tests
needs: pre_job
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
runs-on: ubuntu-latest
environment: percy_tests
outputs:
percy_url: ${{ steps.extract-url.outputs.percy_url }}
steps:
- name: Checkout code from PR
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Use Node.js
uses: actions/setup-node@v4
with:
node-version: '18.x'
- name: Cache Node.js modules
uses: actions/cache@v4
with:
path: '**/node_modules'
key: ${{ runner.OS }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.OS }}-node-
- name: Install dependencies
run: |
yarn --frozen-lockfile
npm rebuild node-sass
- name: Download Chromium
run: npx puppeteer browsers install chrome
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious if this is cached, and if not, if we should

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

From what I can infer from the test run, puppeteer does cache the browser after the first workflow run and uses the same for successive runs. Is there a need to add another step to the workflow for the same?

- name: Extract jsdocs and environment info
run: yarn pregenerate
- name: Run visual tests
run: yarn test:visual 2>&1 | tee test-output.log
env:
PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}
- name: Extract Percy build URL
id: extract-url
run: |
url=$(grep -o 'https://percy.io/[a-zA-Z0-9/_-]*' test-output.log | tail -1)
echo "percy_url=$url" >> $GITHUB_OUTPUT

comment:
name: Comment Percy results
needs: visual_tests
if: ${{ needs.visual_tests.result == 'success' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code from PR
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Define comment body
id: comment-text
uses: actions/github-script@v7
with:
script: |
const percyUrl = "${{ needs.visual_tests.outputs.percy_url }}";
const utils = require('./.github/githubUtils.js');
return await utils.generateComment(percyUrl);
- name: Find existing comment
id: find-comment
uses: actions/github-script@v7
with:
script: |
const utils = require('./.github/githubUtils.js');
return await utils.findComment(github, context, context.issue.number);
- name: Create build comment
if: ${{!steps.find-comment.outputs.result}}
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: ${{ steps.comment-text.outputs.result }}
})
- name: Update build comment
if: ${{steps.find-comment.outputs.result}}
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: ${{steps.find-comment.outputs.result}},
body: ${{ steps.comment-text.outputs.result }}
})
19 changes: 19 additions & 0 deletions .percy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
version: 2
snapshot:
widths:
- 375
- 1280
minHeight: 1024
percyCSS: ""
enableJavaScript: false #disable Javascript by default to capture initial page state without JS-driven changes
cliEnableJavaScript: true #enable Javascript when running Percy through CLI, for dynamic content
disableShadowDOM: false
discovery:
allowedHostnames: []
disallowedHostnames: []
networkIdleTimeout: 100
captureMockedServiceWorker: false
upload:
files: "**/*.{png,jpg,jpeg}"
ignore: ""
stripExtensions: false
1 change: 1 addition & 0 deletions dev_docs/01_getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ You're now ready to code!
- If you'd like to update the component library, continue to [How to update the component library](./03_how_to_update_library.md).

The guidelines referenced above should be sufficient for the most common tasks. There are a few additional developer documentation pages available. However, these pages contain information that is more internal in nature or related to specialized tasks:
- [Visual Testing](./07_visual_testing_guide.md)
- [How to update the documentation website](./04_how_to_update_docs.md)
- [Icons](./05_icons.md)
- [Miscellaneous](./06_misc.md)
131 changes: 131 additions & 0 deletions dev_docs/07_visual_testing_guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Visual Testing

KDS has a visual testing system that allows you to take snapshots of how KDS Components would look like in different browsers, and compare them with the previously set baseline versions of these components, allowing you to see the visual differences in a PR's changes more quickly.

## Prerequisites

- Make sure that your Node version is >= 18.x with all the requirements (listed in package.json) installed.
- Ensure that you declare the `PERCY_TOKEN` environment variable when running visual tests locally.
- You should not have anything running on `port:4000` as the visual testing server utilizes that port to run the tests.

## Guide to using the visual testing mechanism

### Running visual tests locally

1. **Setup**:
- Ensure all dependencies are installed:

```bash
yarn install
```

- Set the `PERCY_TOKEN` environment variable:

```bash
export PERCY_TOKEN=<your-percy-token>
```

2. **Run Visual Tests**:
- Execute the tests with the visual testing configuration:

```bash
yarn test:visual
```

3. **Check results**:
- Head over to the Percy Dashboard and check the results for the latest Percy build.

### Visual testing workflow

We use GitHub Actions to execute the visual testing workflow on every PR. When changes to a PR are made, a deployment request is initiated. Only **Learning Equality team members** can approve this request. Once approved, the deployment is executed, and visual tests are run automatically. The results of the test run are surfaced in the form of an automated comment containing a link to the Percy build.

**Note:** Developers outside of Learning Equality can only review the visual changes for local test runs. To review the visual tests executions that run on GitHub Actions, one needs to be a part of Learning Equality's Browserstack team.

### Writing visual tests

You can write the visual tests alongside the unit tests, i.e. in the same test file. Take a look at [`KButton.spec.js`](../lib/buttons-and-links/__tests__/KButton.spec.js) to familiarize yourself with writing visual tests.

1. **Import Utility Functions**:
- Import the utility functions from [`visual.testUtils.js`](../jest.conf/visual.testUtils.js):

```javascript
import { renderComponent, takeSnapshot } from './visual.testUtils';
```

You can import and use the utility functions for managing component's visual states as needed. Different utility functions available are:

- `renderComponent(component, props, slots)`: Renders the specified component with given props and slots in the visual testing playground.
- `takeSnapshot(name, options)`: Takes a Percy snapshot with the given name and options.
- `click(selector)`, `hover(selector)`, `scrollToPos(selector, scrollOptions)`, `waitFor(selector)`, `delay(time)`: Utility functions for simulating user interactions.

2. **Write Tests**:
- Use the utility functions to render components and take snapshots. For example:

```javascript
describe.visual('KButton Visual Tests', () => {
it('Sample test for KButton', async () => {
await renderComponent('KButton', { text: 'Raised Button', appearance: 'raised-button' });
await takeSnapshot('KButton - Raised Button', { widths: [375, 520] });
});
});
```
Note that the `widths` parameter passed to the `takeSnaphot` function is a part of Percy CLI's snapshot options. For a full list of available options, refer the [Percy documentation](https://www.browserstack.com/docs/percy/take-percy-snapshots/snapshots-via-scripts#per-snapshot-configuration).

- For rendering complex commponents, refer to the following:

- **Example with slots:** For components that involve slots, you can render them with `renderComponent` by passing the slot structure using element and elementProps. You can pass multiple slots at once.

```javascript
await renderComponent('KIconButton', { icon: 'add' }, {
menu: { // slot named #menu
element: 'KDropdownMenu',
elementProps: {
items: ['Option 1', 'Option 2'],
},
},
});
```

**Note:** Use `'default'` key for passing default slots, with the HTML content specified using `innerHTML` prop. Checkout [`KButton.spec.js`](../lib/buttons-and-links/__tests__/KButton.spec.js) for reference.

- **Example involving more complex component structures:** When dealing with more complex component structures, it's recommended to create a dedicated Vue component for visual testing purposes. Add all the use cases in a Vue file and then render the custom component using the `renderComponent` function.

```javascript
await renderComponent('CustomVueComponent');
```

This approach ensures that all necessary child components and slots are correctly set up and rendered.

- Make sure to use `describe.visual` or `it.visual` instead of the default notations for writing test blocks containing visual tests so as to prevent any unexpected behavior. These custom blocks add a `[Visual]` tag to the test name whose presence or absence are then checked using a regex pattern based on the type of tests executed.
- Anything inside these blocks will not be executed when running unit tests. The default `describe` and `it` blocks can be used inside a parent `describe.visual` block, which itelf can be placed within a `describe` block as its parent (as `describe` blocks just group the tests placed within them).
- In simple terms, any test block with a `[Visual]` tag will be executed when running visual tests, regardless of the type of test blocks used within it, and will be ignored when running unit tests. Using `describe.visual` or `it.visual` automatically appends this tag to the test name.
- This implementation helps determine which test blocks should be executed by Jest and which ones should be skipped.

3. **Simulate User Interactions**:
- Use the custom commands to simulate user interactions. For example, to simulate the *'click'* user event, you can do something like:

```javascript
await click('button');
```

Here, *'button'* is the CSS selector for the component. You can pass different selectors to the functions, exposed by [`visual.testUtils.js`](../jest.conf/visual.testUtils.js), to simulate user interaction as per requirement.

## Implementation details

The visual testing mechanism uses the following dependencies to ensure components render correctly under various conditions:

- **Puppeteer:** for interacting with the testing environment and the rendered components.
- **Jest-Puppeteer:** to provide all required configuration to run tests using Puppeteer.
- **Percy:** to take snapshots for comparing visual diffs.

The key parts of the mechanism include:

1. **Configuration Files**: Since we are using Jest for both unit and visual tests, there are two separate configuration files for visual tests apart from the ones being used for unit tests so as to ensure separation of logic needed for running both types of tests.
- [***visual.index.js***](../jest.conf/visual.index.js): Configures Jest-Puppeteer and includes server checks to ensure the visual testing playground is up and running.
- [***visual.setup.js***](../jest.conf/visual.setup.js): Sets up global functions and constants needed for visual tests.

2. **Utility Functions** [***(visual.testUtils.js)***](../jest.conf/visual.testUtils.js): We are also using a separate file that contains all the utility functions that are needed for writing visual tests.

3. **Visual Testing Playground** ([***testing-playground.vue***](../docs/pages/testing-playground.vue)):
- A dedicated page rendered by the devserver for component visual testing, ensuring expected visual behavior under various conditions.
- The visual test command runs the devserver and once the server is up and the testing playground page is loaded, the visual tests are executed and the required components are rendered dynamically based on messages received from the test runner.
Loading
Loading