-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate visual testing mechanism to the latest KDS version #901
Merged
AlexVelezLl
merged 62 commits into
learningequality:develop
from
KshitijThareja:gsoc/vis-testing
Jan 29, 2025
+3,867
−2,780
Merged
Changes from 59 commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
0236bf6
Add and configure percy for visual tests
KshitijThareja 2434700
Change server configuration and file names for visual tests
KshitijThareja b3f0bb4
Add component rendering mechanism for visual testing
KshitijThareja 7d55f4b
Remove unused packages
KshitijThareja c91086f
Add JSDoc for testing playground and modify server script
KshitijThareja e251350
Update KImg unit tests and eslint configuration
KshitijThareja cefdfe6
Use concurrently to manage startup and termination of tests
KshitijThareja 63d5144
Update waitForServer function to use http
KshitijThareja b6044ad
Remove unused dependencies and separate visual tests from yarn test
KshitijThareja fc09854
Separate visual tests from yarn test command
KshitijThareja cf41d8c
Merge pull request #670 from KshitijThareja/visual-testing
bjester 6c3fd69
Introduce abstraction for component rendering functions and support f…
KshitijThareja 985c0d4
Initial workflow config
KshitijThareja e5d792d
Fix the initial workflow for visual tests job
KshitijThareja 8823a8f
Merge pull request #677 from KshitijThareja/vt-add-concurrently
bjester cc9417c
Add automatic comments job for visual testing workflow
KshitijThareja 2ef82b1
Fix lint errors
KshitijThareja 9ad6f8f
Abstraction for page navigation
KshitijThareja 1b024f9
Fix lint errors
KshitijThareja 4454b60
Introduce visual testing to the existing Javascript tests workflow
KshitijThareja 8119ec9
Replace usage of custom describe blocks with test regex patterns
KshitijThareja e1a54ba
Remove unintentional changes
KshitijThareja f55a831
Add comments for rendering function and make variable names clear
KshitijThareja 475827f
Remove run server step
KshitijThareja e325e34
Remove pipes where not required
KshitijThareja 622a22c
Add special test blocks for visual tests to append Visual tag automat…
KshitijThareja 55bff15
Merge pull request #685 from KshitijThareja/vt-abstraction
AlexVelezLl 02d0dbb
Add support for default slots rendering
KshitijThareja 7c117e7
Change workflow to run on separate environment
KshitijThareja 2d2a9bd
Merge branch 'gsoc/visual-testing' into vt-ghaction
KshitijThareja 8c1cfdb
Add option to specify custom dimensions for snapshots
KshitijThareja 1e3c790
Add puppeteer config for CI environments
KshitijThareja a0423f5
Remove .puppeteerrc.cjs
KshitijThareja 5a76c43
Checkout pr head
AlexVelezLl 563fd8f
Update should skip to include style changes
AlexVelezLl 3fbaf41
Initial documentation for visual tests
KshitijThareja 1e664e2
Update comment job to use github-script action
KshitijThareja e61a2de
Update visual testing docs with links to mentioned files
KshitijThareja 348e47c
Update documentation
KshitijThareja e31e203
Updat comment job to use utils file
KshitijThareja 690cfd8
Migrate findComment and generateComment functions to githubUtils.js
KshitijThareja 21a0d46
Add reference to documentation in getting-started doc
KshitijThareja a6d1312
Update jsdoc
AlexVelezLl c5e0df5
Merge branch 'gsoc/visual-testing' into vt-test
AlexVelezLl 1c7ec1e
Merge branch 'gsoc/visual-testing' into vt-test
AlexVelezLl 7d6c634
Modify testing playground to render named and default slots in a sing…
KshitijThareja 09820a1
Add jsdocs for renderComponent and takeSnapshot functions
KshitijThareja 05063ac
Fix lint error
KshitijThareja aaa4411
Remove async from describe block
KshitijThareja 6be488b
Add snapshot options specifying test widths
KshitijThareja 15d8ae5
Merge pull request #710 from KshitijThareja/vt-test
AlexVelezLl 1f09d07
Update test command to work for all test files
KshitijThareja 083532a
Add additional info for rendering complex components
KshitijThareja 86d7402
Add link to percy documentation
KshitijThareja e4b8986
Update info for usage of test blocks for visual tests
KshitijThareja c8e4d0e
Update required node version specification
KshitijThareja b0cc1e7
Fix errors in integrating visual testing setup with the codebase
KshitijThareja 924f782
Update percy and puppeteer to latest versions
KshitijThareja d07775a
Update package.json
KshitijThareja 7b7395d
Update visual test description for KButton tests and fix http import
KshitijThareja 78324ba
Resolve merge conflicts with the develop branch
KshitijThareja d4d12b1
Merge branch 'develop' into gsoc/vis-testing
KshitijThareja File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
async function generateComment(percyUrl) { | ||
return ` | ||
### Percy Visual Test Results | ||
|
||
**Percy Dashboard:** [View Detailed Report](${percyUrl}) | ||
|
||
**Environment:** | ||
- **Node.js Version:** 18.x | ||
- **OS:** Ubuntu-latest | ||
|
||
**Instructions for Reviewers:** | ||
- Click on the [Percy Dashboard](${percyUrl}) link to view detailed visual diffs. | ||
- Review the visual changes highlighted in the report. | ||
- Approve or request changes based on the visual differences. | ||
`; | ||
} | ||
|
||
async function findComment(github, context, issue_number) { | ||
let comment; | ||
let page = 1 | ||
while (!comment) { | ||
const request = await github.rest.issues.listComments({ | ||
issue_number, | ||
owner: context.repo.owner, | ||
repo: context.repo.repo, | ||
page, | ||
}) | ||
const comments = request.data | ||
if (!comments.length) { | ||
return; | ||
} | ||
comment = comments.find(c => c.body && c.body.includes('### Percy Visual Test Results')); | ||
if (comment) { | ||
return comment.id.toString() | ||
} | ||
page += 1; | ||
} | ||
} | ||
|
||
module.exports = { | ||
findComment, | ||
generateComment, | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
name: Percy Visual Tests | ||
|
||
on: [pull_request_target] | ||
|
||
jobs: | ||
pre_job: | ||
name: Path match check | ||
runs-on: ubuntu-latest | ||
# Map a step output to a job output | ||
outputs: | ||
should_skip: ${{ steps.skip_check.outputs.should_skip }} | ||
steps: | ||
- id: skip_check | ||
uses: fkirc/skip-duplicate-actions@master | ||
with: | ||
github_token: ${{ github.token }} | ||
paths: '["**.vue", "**.js", "**.css", "**.scss", "yarn.lock"]' | ||
|
||
visual_tests: | ||
name: Frontend Visual Tests | ||
needs: pre_job | ||
if: ${{ needs.pre_job.outputs.should_skip != 'true' }} | ||
runs-on: ubuntu-latest | ||
environment: percy_tests | ||
outputs: | ||
percy_url: ${{ steps.extract-url.outputs.percy_url }} | ||
steps: | ||
- name: Checkout code from PR | ||
uses: actions/checkout@v4 | ||
with: | ||
ref: ${{ github.event.pull_request.head.sha }} | ||
- name: Use Node.js | ||
uses: actions/setup-node@v4 | ||
with: | ||
node-version: '18.x' | ||
- name: Cache Node.js modules | ||
uses: actions/cache@v4 | ||
with: | ||
path: '**/node_modules' | ||
key: ${{ runner.OS }}-node-${{ hashFiles('**/yarn.lock') }} | ||
restore-keys: | | ||
${{ runner.OS }}-node- | ||
- name: Install dependencies | ||
run: | | ||
yarn --frozen-lockfile | ||
npm rebuild node-sass | ||
- name: Download Chromium | ||
run: npx puppeteer browsers install chrome | ||
- name: Extract jsdocs and environment info | ||
run: yarn pregenerate | ||
- name: Run visual tests | ||
run: yarn test:visual 2>&1 | tee test-output.log | ||
env: | ||
PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }} | ||
- name: Extract Percy build URL | ||
id: extract-url | ||
run: | | ||
url=$(grep -o 'https://percy.io/[a-zA-Z0-9/_-]*' test-output.log | tail -1) | ||
echo "percy_url=$url" >> $GITHUB_OUTPUT | ||
|
||
comment: | ||
name: Comment Percy results | ||
needs: visual_tests | ||
if: ${{ needs.visual_tests.result == 'success' }} | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout code from PR | ||
uses: actions/checkout@v4 | ||
with: | ||
ref: ${{ github.event.pull_request.head.sha }} | ||
- name: Define comment body | ||
id: comment-text | ||
uses: actions/github-script@v7 | ||
with: | ||
script: | | ||
const percyUrl = "${{ needs.visual_tests.outputs.percy_url }}"; | ||
const utils = require('./.github/githubUtils.js'); | ||
return await utils.generateComment(percyUrl); | ||
- name: Find existing comment | ||
id: find-comment | ||
uses: actions/github-script@v7 | ||
with: | ||
script: | | ||
const utils = require('./.github/githubUtils.js'); | ||
return await utils.findComment(github, context, context.issue.number); | ||
- name: Create build comment | ||
if: ${{!steps.find-comment.outputs.result}} | ||
uses: actions/github-script@v7 | ||
with: | ||
github-token: ${{ secrets.GITHUB_TOKEN }} | ||
script: | | ||
github.rest.issues.createComment({ | ||
issue_number: context.issue.number, | ||
owner: context.repo.owner, | ||
repo: context.repo.repo, | ||
body: ${{ steps.comment-text.outputs.result }} | ||
}) | ||
- name: Update build comment | ||
if: ${{steps.find-comment.outputs.result}} | ||
uses: actions/github-script@v7 | ||
with: | ||
github-token: ${{ secrets.GITHUB_TOKEN }} | ||
script: | | ||
github.rest.issues.updateComment({ | ||
owner: context.repo.owner, | ||
repo: context.repo.repo, | ||
comment_id: ${{steps.find-comment.outputs.result}}, | ||
body: ${{ steps.comment-text.outputs.result }} | ||
}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
version: 2 | ||
snapshot: | ||
widths: | ||
- 375 | ||
- 1280 | ||
minHeight: 1024 | ||
percyCSS: "" | ||
enableJavaScript: false #disable Javascript by default to capture initial page state without JS-driven changes | ||
cliEnableJavaScript: true #enable Javascript when running Percy through CLI, for dynamic content | ||
disableShadowDOM: false | ||
discovery: | ||
allowedHostnames: [] | ||
disallowedHostnames: [] | ||
networkIdleTimeout: 100 | ||
captureMockedServiceWorker: false | ||
upload: | ||
files: "**/*.{png,jpg,jpeg}" | ||
ignore: "" | ||
stripExtensions: false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
# Visual Testing | ||
|
||
KDS has a visual testing system that allows you to take snapshots of how KDS Components would look like in different browsers, and compare them with the previously set baseline versions of these components, allowing you to see the visual differences in a PR's changes more quickly. | ||
|
||
## Prerequisites | ||
|
||
- Make sure that your Node version is >= 18.x with all the requirements (listed in package.json) installed. | ||
- Ensure that you declare the `PERCY_TOKEN` environment variable when running visual tests locally. | ||
- You should not have anything running on `port:4000` as the visual testing server utilizes that port to run the tests. | ||
|
||
## Guide to using the visual testing mechanism | ||
|
||
### Running visual tests locally | ||
|
||
1. **Setup**: | ||
- Ensure all dependencies are installed: | ||
|
||
```bash | ||
yarn install | ||
``` | ||
|
||
- Set the `PERCY_TOKEN` environment variable: | ||
|
||
```bash | ||
export PERCY_TOKEN=<your-percy-token> | ||
``` | ||
|
||
2. **Run Visual Tests**: | ||
- Execute the tests with the visual testing configuration: | ||
|
||
```bash | ||
yarn test:visual | ||
``` | ||
|
||
3. **Check results**: | ||
- Head over to the Percy Dashboard and check the results for the latest Percy build. | ||
|
||
### Visual testing workflow | ||
|
||
We use GitHub Actions to execute the visual testing workflow on every PR. When changes to a PR are made, a deployment request is initiated. Only **Learning Equality team members** can approve this request. Once approved, the deployment is executed, and visual tests are run automatically. The results of the test run are surfaced in the form of an automated comment containing a link to the Percy build. | ||
|
||
**Note:** Developers outside of Learning Equality can only review the visual changes for local test runs. To review the visual tests executions that run on GitHub Actions, one needs to be a part of Learning Equality's Browserstack team. | ||
|
||
### Writing visual tests | ||
|
||
You can write the visual tests alongside the unit tests, i.e. in the same test file. Take a look at [`KButton.spec.js`](../lib/buttons-and-links/__tests__/KButton.spec.js) to familiarize yourself with writing visual tests. | ||
|
||
1. **Import Utility Functions**: | ||
- Import the utility functions from [`visual.testUtils.js`](../jest.conf/visual.testUtils.js): | ||
|
||
```javascript | ||
import { renderComponent, takeSnapshot } from './visual.testUtils'; | ||
``` | ||
|
||
You can import and use the utility functions for managing component's visual states as needed. Different utility functions available are: | ||
|
||
- `renderComponent(component, props, slots)`: Renders the specified component with given props and slots in the visual testing playground. | ||
- `takeSnapshot(name, options)`: Takes a Percy snapshot with the given name and options. | ||
- `click(selector)`, `hover(selector)`, `scrollToPos(selector, scrollOptions)`, `waitFor(selector)`, `delay(time)`: Utility functions for simulating user interactions. | ||
|
||
2. **Write Tests**: | ||
- Use the utility functions to render components and take snapshots. For example: | ||
|
||
```javascript | ||
describe.visual('KButton Visual Tests', () => { | ||
it('Sample test for KButton', async () => { | ||
await renderComponent('KButton', { text: 'Raised Button', appearance: 'raised-button' }); | ||
await takeSnapshot('KButton - Raised Button', { widths: [375, 520] }); | ||
}); | ||
}); | ||
``` | ||
Note that the `widths` parameter passed to the `takeSnaphot` function is a part of Percy CLI's snapshot options. For a full list of available options, refer the [Percy documentation](https://www.browserstack.com/docs/percy/take-percy-snapshots/snapshots-via-scripts#per-snapshot-configuration). | ||
|
||
- For rendering complex commponents, refer to the following: | ||
|
||
- **Example with slots:** For components that involve slots, you can render them with `renderComponent` by passing the slot structure using element and elementProps. You can pass multiple slots at once. | ||
|
||
```javascript | ||
await renderComponent('KIconButton', { icon: 'add' }, { | ||
menu: { // slot named #menu | ||
element: 'KDropdownMenu', | ||
elementProps: { | ||
items: ['Option 1', 'Option 2'], | ||
}, | ||
}, | ||
}); | ||
``` | ||
|
||
**Note:** Use `'default'` key for passing default slots, with the HTML content specified using `innerHTML` prop. Checkout [`KButton.spec.js`](../lib/buttons-and-links/__tests__/KButton.spec.js) for reference. | ||
|
||
- **Example involving more complex component structures:** When dealing with more complex component structures, it's recommended to create a dedicated Vue component for visual testing purposes. Add all the use cases in a Vue file and then render the custom component using the `renderComponent` function. | ||
|
||
```javascript | ||
await renderComponent('CustomVueComponent'); | ||
``` | ||
|
||
This approach ensures that all necessary child components and slots are correctly set up and rendered. | ||
|
||
- Make sure to use `describe.visual` or `it.visual` instead of the default notations for writing test blocks containing visual tests so as to prevent any unexpected behavior. These custom blocks add a `[Visual]` tag to the test name whose presence or absence are then checked using a regex pattern based on the type of tests executed. | ||
- Anything inside these blocks will not be executed when running unit tests. The default `describe` and `it` blocks can be used inside a parent `describe.visual` block, which itelf can be placed within a `describe` block as its parent (as `describe` blocks just group the tests placed within them). | ||
- In simple terms, any test block with a `[Visual]` tag will be executed when running visual tests, regardless of the type of test blocks used within it, and will be ignored when running unit tests. Using `describe.visual` or `it.visual` automatically appends this tag to the test name. | ||
- This implementation helps determine which test blocks should be executed by Jest and which ones should be skipped. | ||
|
||
3. **Simulate User Interactions**: | ||
- Use the custom commands to simulate user interactions. For example, to simulate the *'click'* user event, you can do something like: | ||
|
||
```javascript | ||
await click('button'); | ||
``` | ||
|
||
Here, *'button'* is the CSS selector for the component. You can pass different selectors to the functions, exposed by [`visual.testUtils.js`](../jest.conf/visual.testUtils.js), to simulate user interaction as per requirement. | ||
|
||
## Implementation details | ||
|
||
The visual testing mechanism uses the following dependencies to ensure components render correctly under various conditions: | ||
|
||
- **Puppeteer:** for interacting with the testing environment and the rendered components. | ||
- **Jest-Puppeteer:** to provide all required configuration to run tests using Puppeteer. | ||
- **Percy:** to take snapshots for comparing visual diffs. | ||
|
||
The key parts of the mechanism include: | ||
|
||
1. **Configuration Files**: Since we are using Jest for both unit and visual tests, there are two separate configuration files for visual tests apart from the ones being used for unit tests so as to ensure separation of logic needed for running both types of tests. | ||
- [***visual.index.js***](../jest.conf/visual.index.js): Configures Jest-Puppeteer and includes server checks to ensure the visual testing playground is up and running. | ||
- [***visual.setup.js***](../jest.conf/visual.setup.js): Sets up global functions and constants needed for visual tests. | ||
|
||
2. **Utility Functions** [***(visual.testUtils.js)***](../jest.conf/visual.testUtils.js): We are also using a separate file that contains all the utility functions that are needed for writing visual tests. | ||
|
||
3. **Visual Testing Playground** ([***testing-playground.vue***](../docs/pages/testing-playground.vue)): | ||
- A dedicated page rendered by the devserver for component visual testing, ensuring expected visual behavior under various conditions. | ||
- The visual test command runs the devserver and once the server is up and the testing playground page is loaded, the visual tests are executed and the required components are rendered dynamically based on messages received from the test runner. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious if this is cached, and if not, if we should
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From what I can infer from the test run, puppeteer does cache the browser after the first workflow run and uses the same for successive runs. Is there a need to add another step to the workflow for the same?