Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically add a results table for CI runs on pull requests #4118

Open
ptomato opened this issue Jun 26, 2024 · 1 comment
Open

Automatically add a results table for CI runs on pull requests #4118

ptomato opened this issue Jun 26, 2024 · 1 comment

Comments

@ptomato
Copy link
Contributor

ptomato commented Jun 26, 2024

I've noticed several times that the meaning of the CI results is confusing for contributors:

image

This looks like it is saying "all engines passed these tests." What it really means is "all engines installed correctly and executed these tests without crashing, regardless of pass or fail."

image

Likewise, this looks like it means "ChakraCore failed these tests, all other engines passed." What it really means is "ChakraCore failed to install."

I think we could reduce this confusion by automatically posting individual test results to a table in a PR comment. For example, the table for #4052 might look like this:

Test name engine262 ESMeta JSC SpiderMonkey V8 V8 --harmony XS
language/expressions/assignment/target-member-computed-reference.js
language/expressions/assignment/target-super-computed-reference.js

(pass/fail status is entirely made up and not a comment on any engine)

ci_test.sh already runs only the added and modified tests on each engine, and just discards the results; we could easily consume those results to build the table.

@ljharb
Copy link
Member

ljharb commented Jun 26, 2024

That would be great. What would be even better if the table only included results that were different from the last successful run on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants