Skip to content

Latest commit

 

History

History
149 lines (112 loc) · 6.32 KB

testing.md

File metadata and controls

149 lines (112 loc) · 6.32 KB

Testing

Table of Contents

Overview

Task Command
Run package test yarn test:e2e
Run playwright tests yarn playwright test
Run a specific playwright test yarn playwright test path/to/test-e2e.js
Run playwright tests in a specific browser yarn playwright test --browser=chromium
Run playwright tests in a specific project yarn playwright test --project=chromium
Debug playwright tests yarn playwright test --debug
Run playwright with browser visible yarn playwright test --project=chromium --headed
Run playwright tests that match a specific tag yarn playwright test --grep @tag-name
Run playwright tests that do not match a specific tag yarn playwright test --grep-invert @tag-name

End-to-end

We use our end-to-end tests to verify the packaging of libraries along with the look and behavior of components in our design system. End-to-end tests for packages are run using Jest, components are run using Playwright.

Playwright

We use Playwright to run end-to-end tests against our components in our Design System. We also use it in other situations to prevent visual regressions, like in our elements site.

These tests are authored within the e2e directory and match the file pattern: *-test.e2e.js.

Tags

Playwright tests are divided into different tag categories for reporting purposes. @avt test tags are used to populate accessibility test statuses within component pages on https://carbondesignsystem.com

For avt tests, the test title should always include one of the following:

Tag Description
@avt High level/root tag that should wrap all avt tests. This is usually placed in a describe block title.
@avt-default-state Sub-tag of @avt, used to tag individual tests covering the default state of a component.
@avt-advanced-states Sub-tag of @avt, used to tag individual tests covering advanced states of a component (open/close, invalid, expanded, etc.).
@avt-keyboard-nav Sub-tag of @avt, used to tag individual tests covering keyboard navigation flows.

Developing

When working with Playwright locally, it's important to start up the service that you're testing against. For components in React, this will mean starting up the React storybook locally by doing the following from the root of the project:

cd packages/react
yarn storybook

Once the storybook is loaded, you can run tests against it using the storybook test-utils found in e2e/test-utils/storybook. A common use-case for testing a component is to use Percy to take a snapshot of a component in a particular theme from a specific story in storybook.

You can do this by writing the following:

// e2e/components/component/component-test.e2e.js

'use strict';

const { test } = require('@playwright/test');
const { themes } = require('../../test-utils/env');
const { snapshotStory } = require('../../test-utils/storybook');

test('component-name @vrt', ({ page }) => {
  await snapshotStory(page, {
    component: 'component',
    story: 'story-name',
    theme: 'white',
  });
});

You can test this component in multiple themes by writing the following:

// e2e/components/component/component-test.e2e.js

'use strict';

const { test } = require('@playwright/test');
const { themes } = require('../../test-utils/env');
const { snapshotStory } = require('../../test-utils/storybook');

test.describe('component-name @vrt', () => {
  themes.forEach((theme) => {
    test(theme, async ({ page }) => {
      await snapshotStory(page, {
        component: 'component',
        story: 'story-name',
        theme,
      });
    });
  });
});

If you would like to debug or interact with the test suite, you can use Playwright's VS Code integration or run yarn playwright test with the --debug flag to open up the inspector. This will allow you to go through the test step-by-step to debug what's going on. It will also allow you to interact with the page to quickly find selectors you can use to find items to run tests against.

Working with snapshots locally

Sometimes you'll want to debug snapshots locally instead of relying on an external service to get feedback. To do so, you can use the ENABLE_LOCAL_SNAPSHOTS environment variable to store snapshots locally. Almost any playwright command you run can be prefixed with this value in order to store screenshots locally.

ENABLE_LOCAL_SNAPSHOTS=1 yarn playwright test --project chromium --grep @vrt component-test.e2e.js

Note: it's important to narrow down tests in order to not generate a lot of screenshots locally

The first time you'll run this command, it will need to generate the baseline snapshots for this component. The second time you run it, it will compare the snapshots for the current page with what is stored in the screenshot. If the two do not match, Playwright will report a failure and will provide a link to the diff image on your machine.

FAQ

Why am I seeing browserType.launch: Executable doesn't exist at ../path?

The browser executables need to be installed so that Playwright can run tests inside Chromium, Firefox, etc. They can be installed by running yarn playwright install