Add playwright end-to-end tests #322
Conversation
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
| strategy: | ||
| matrix: | ||
| node: [20] | ||
| os: [ubuntu-latest] |
There was a problem hiding this comment.
If the idea is to test the built artifacts, should we test more targets?
There was a problem hiding this comment.
I thought the artefacts would hopefully be guaranteed to be the same across environments (and that other tests would aim to guarantee that). So then it'd be a waste to run playwright tests on all OSs/versions.
There was a problem hiding this comment.
Depends on what you mean by artifacts in this case, but the Rust artifacts would be per platform. They get copied per arch / os in the release workflow.
There was a problem hiding this comment.
Sorry. Let me clarify;
- The intention of this PR is to load javascript, CSS and HTML files onto a web browser to test certain E2E functionality that is currently not covered by Node.js test suites (which load bundles into the Node.js VM module)
- For example: certain async loading functionality, all of HTML transform/bundle/packaging functionality, other browser specific functionality
So by 'built artefact' I mean: HTML, CSS, JavaScript, Images, built by atlaspack itself.
It is true that atlaspack itself might be running on different versions of Node.js or running binaries compiled for different operating systems or architectures. Those compiled binaries could also be called 'build artefacts'.
I do not think it's necessary to test the 'bundler artefacts' (HTML/CSS/JavaScript) for each compiled binary/node.js version target (Linux/v22/macOS/v18), because unless there is a platform specific bug or a compiler bug the 'bundler artefacts' (HTML/CSS/JavaScript) should be the same across platforms.
Since we already have "integration tests" running per platform and version, I thought those tests can be thought to/aim to guarantee that the output (the 'bundler artefacts' HTML/CSS/JavaScript) is correct and the same regardless of the environment (Linux/v22/macOS/v18).
Browser based E2E tests would guarantee that provided the bundler runs, its outputs also work on a web browser.
Because these are browser based tests, which ought to be slow and have certain drawbacks, I thought it'd be enough to run them on their own job on linux only.
There was a problem hiding this comment.
We may add more browsers / OSs too, but I'd want to avoid adding 4 new jobs for this. I can instead just copy this into the integration test suite if that seems better, I'll just need to provision playwright onto the integration test jobs.
There was a problem hiding this comment.
Yeah, the terminology is ambiguous which is why I was confused. What you've said makes sense, all good.
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
Created using spr 1.3.5
There was a problem hiding this comment.
Can we start writing tests in TypeScript?
The Babel register function already supports TypeScript and Nodejs now supports TS natively without flags (only caveat is Nodejs TS support requires file extensions to be specified in the import)
There was a problem hiding this comment.
I suppose we could try to do that
Adds E2E playwright based tests. These allow us to test that built artifacts
work on browsers end-to-end.
Test Plan: Run
yarn test:end-to-end