Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suspicious time diff at https://perf.rust-lang.org/status.html in took/expected #1713

Open
klensy opened this issue Aug 31, 2023 · 8 comments

Comments

@klensy
Copy link
Contributor

klensy commented Aug 31, 2023

Time displayed at https://perf.rust-lang.org/status.html for took/expected heavily differs for some runs, while perf result shows no difference.

For example, for run rust-lang/rust#115388 (comment)

Step Took Expected
await-call-tree 0m28s 0m28s
bitmaps-3.1.0 1m06s 0m59s
cargo-0.60.0 8m24s 5m23s
clap-3.1.6 1m43s 1m32s
coercions 0m56s 0m54s
cranelift-codegen-0.82.1 9m50s 3m02s
ctfe-stress-5 1m24s 1m11s
deeply-nested-multi 2m43s 0m32s
deep-vector 1m34s 1m24s
derive 4m13s 0m49s
diesel-1.4.8 7m15s 2m30s
exa-0.10.1 3m00s 3m00s
externs 0m38s 0m39s
helloworld 0m37s 0m37s
helloworld-tiny 0m11s 0m11s
html5ever-0.26.0 1m18s 1m08s
hyper-0.14.18 2m11s 2m10s
image-0.24.1 2m00s 2m01s
issue-46449 1m15s 1m16s
issue-58319 0m32s 0m32s
issue-88862 0m32s 0m32s
libc-0.2.124 1m01s 1m02s
many-assoc-items 0m57s 0m59s
match-stress 0m51s 0m52s
projection-caching 0m41s 0m41s
regex-1.5.5 2m58s 2m56s
regression-31157 0m54s 1m01s
ripgrep-13.0.0 2m04s 2m02s
ripgrep-13.0.0-tiny 1m05s 1m04s
serde-1.0.136 1m33s 1m33s
serde_derive-1.0.136 1m38s 1m38s
stm32f4-0.14.0 3m49s 6m10s
syn-1.0.89 1m28s 1m24s
token-stream-stress 0m28s 0m32s
tt-muncher 0m28s 1m00s

Notice cranelift, cargo, derive time diff for took/expected, while results clean https://perf.rust-lang.org/compare.html?start=b1b244da6527cf2ba36e88d02275f4c64a0c90d8&end=24259321f2e7a82959b47b86ded3d1073f281746&stat=instructions:u

And next perf for 6ff94474e1d11 (rust-lang/rust#115391 (comment)) run bounces back:

Step Took Expected
await-call-tree 0m30s 0m28s
bitmaps-3.1.0 1m02s 1m06s
cargo-0.60.0 5m26s 8m24s
clap-3.1.6 1m32s 1m43s
coercions 0m55s 0m56s
cranelift-codegen-0.82.1 3m04s 9m50s
ctfe-stress-5 1m12s 1m24s
deeply-nested-multi 0m33s 2m43s
deep-vector 1m25s 1m34s
derive 0m51s 4m13s
diesel-1.4.8 2m32s 7m15s
exa-0.10.1 2m59s 3m00s
externs 0m38s 0m38s
helloworld 0m37s 0m37s
helloworld-tiny 0m11s 0m11s
html5ever-0.26.0 1m09s 1m18s
hyper-0.14.18 2m10s 2m11s
image-0.24.1 2m02s 2m00s
issue-46449 1m16s 1m15s
issue-58319 0m33s 0m32s
issue-88862 0m33s 0m32s
libc-0.2.124 1m01s 1m01s
many-assoc-items 0m58s 0m57s
match-stress 0m52s 0m51s
projection-caching 0m41s 0m41s
regex-1.5.5 2m56s 2m58s
regression-31157 0m55s 0m54s
ripgrep-13.0.0 2m03s 2m04s
ripgrep-13.0.0-tiny 1m05s 1m05s
rustc 7m45s 13m12s
serde-1.0.136 1m37s 1m33s
serde_derive-1.0.136 1m41s 1m38s
stm32f4-0.14.0 3m50s 3m49s
syn-1.0.89 1m26s 1m28s
token-stream-stress 0m28s 0m28s
tt-muncher 0m45s 0m44s
tuple-stress 0m58s 0m56s
ucd 1m04s 1m03s
unicode-normalization-0.1.19 0m52s 0m51s
unify-linearly 0m39s 0m38s
unused-warnings 1m00s 1m01s
webrender-2022 3m46s 3m46s
wf-projection-stress-65510 0m31s 0m31s
wg-grammar 0m54s 0m54s

Quite sus number diff, while nothing interesting can be seen in perf results.

@lqd
Copy link
Member

lqd commented Aug 31, 2023

cranelift is already back to 3 mins

image

@klensy
Copy link
Contributor Author

klensy commented Aug 31, 2023

Yeph, in first message there second table, where it can be seen:

cranelift-codegen-0.82.1 3m04s 9m50s

@klensy
Copy link
Contributor Author

klensy commented Aug 31, 2023

Both runs was try runs, so possible difference in try/post-merge runs can't be here.

@lqd
Copy link
Member

lqd commented Aug 31, 2023

The second table still has the estimate at 9m50s, while I'm saying the estimate is now fixed.

Some try runs may have been slow and I also indeed would like to know whether we use them for the time estimates: it seems possible if we're actually using the "last run", but I don't know for sure. We'll see when jakub is back from vacation.

@klensy
Copy link
Contributor Author

klensy commented Aug 31, 2023

Looking at digits, Expected numbers is really taken from latest run:

cranelift-codegen-0.82.1 9m50s 3m02s
cranelift-codegen-0.82.1 3m04s 9m50s
And your linked times for cranelift: 3m08s 3m04s.

We can try to check source code to dig possible truth :-)

@lqd
Copy link
Member

lqd commented Aug 31, 2023

It's my recollection that we're using the last run but wasn't sure if try builds were used or not -- and as you say it seems like they are ^^

@Kobzol
Copy link
Contributor

Kobzol commented Aug 31, 2023

Yeah, it's always estimated from the most recent run. We should probably ignore try runs though, I'll look into it.

@klensy
Copy link
Contributor Author

klensy commented Aug 31, 2023

Yeah, it's always estimated from the most recent run. We should probably ignore try runs though, I'll look into it.

This probably shouldn't be fixed until reason why this difference shows up will be found. Or maybe didn't changed at all, with current behavior we can at least saw difference and start digging; without that - noone will catch this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants