Experiment, parallelize some tests

[majocha]

Just to see if it works and if it's worth it.

[github-actions]

❗ Release notes required

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
src/Compiler	docs/release-notes/.FSharp.Compiler.Service/9.0.200.md

[majocha]

Yeah this will not do much.

A lot of test cases write to stdout. We do captureConsoleOutputs in CompilerAssert but in a way that does not allow parallel execution. Not to mention in a lot of places there are printfn calls that will mangle outputs even more. All of those tests must run sequentially.

This needs a systemic approach to deal with stdout capture, maybe xUnit have some mechanism to do this in parallel.

[psfinaki]

@majocha guess what, I was going to try the same today :)

Thanks for looking into that. Indeed, it seems like we should just damn stop printing everything to the console, it's likely an artifact of older testing approaches. I cannot see any reason to do this instead of just in memory processing.

[majocha]

xUnit does not run tests from the same module in parallel. It also does not parallelize Theories.
This can slow things down even with parallel execution enabled. In FSharpSuite we have modules with lots of slow tests.

This can be mitigated by customizing xUnit in code, I think?

[psfinaki]

By customizing xUnit you mean setting up some special runner settings and assembly attributes?

We can do that. However - I am not a fan of this idea. xUnit's philosophy is really to apply good coding practices to tests. As in, write tests as you write code. Hence, e.g. compared to Nunit, it offers a very limited test platform voodoo (think fixtures, setup/teardown and so on), instead making as much as possible of the builtin language capabilities.

And so I would instead prefer keeping up with this philosophy. If, by default, xUnit parallelizes execution on the module level, then we should actually split modules into smaller ones - thereby it will improve code clarity and will generally add to better code organization :)

What do you think?

[majocha]

By customizing I meant something like https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

But this is not the most pressing thing and probably not needed if splitting modules would do.

The biggest hurdle for now is correctly isolating the console when running tests in parallel. Redirecting with Console.SetOut for each test won't work anymore when another thread can also redirect it unpredictably.

Console writes come from multiple sources:

1. In process executing fsi, probably also fsc
2. Various printfn and log calls sprinkled through the test cases and helper code.
3. Source code compiled and executed in AssemblyLoadContext (or AppDomain in case of net472).

While we can manage 1. and 2., 3. is a bit harder.

[majocha]

I've been chatting with Bing / Copilot about it, and it actually proposed a not bad idea:

Don't redirect the Console at all for individual tests. Instead install a custom thread splitting TextWriter upfront.
That splitting writer will keep a ThreadLocal inner TextWriter and direct all writes to it.

[psfinaki]

Thanks for the further investigations here.

In the spirit of my comment above - I just vote for reinventing as few wheels as possible and removing those we've already reinvented here :) Unit tests rarely need any output at all, but if they do - it's good to use those few means that xUnit provides for this, which are basically "plug in the writer if and when you need to".

I think it aligns with your thoughts above? It's important to make gradual changes here, probably actually in the way you outline it above. The current direction you're taking (removing stuff) looks promising!

---

Note, I am off until Monday with limited internet connection so cannot play with the code myself. Also, we've discussed this PR internally yesterday and were all very happy that things are moving in this space!

[majocha]

This is at a state that can be run locally in VS test explorer or from the console with build -c Release -testCoreClr
I think it's shaving a few minutes from build -testCoreClr locally, this could be further improved by breaking up large modules in ComponentTests and FSharpSuite. Definitely I see much better CPU utilization.

In the CI there's that weird TaskCancelledException in random tests, no idea where it comes from.

Still, there are some minor fixes here that I'll try to extract to another PR.

[psfinaki]

Which ones do you mean in particular? Maybe I can try hacking something locally.

[majocha]

In the previous ci run, I think. I'm on the phone right now, so I can't easily find the link, sorry 🙂

[psfinaki]

Well I guess eventually, all the problems we might be getting here would be due to the fact that we are running in parallel something that shouldn't or cannot work like that yet. So disabling parallelization should heal this as well? Or I am looking at the wrong place?

[majocha]

It's a bit worse than that, the failures in CI I usually cannot repro locally. But you're right, I should put a strictly maxthreads: 1 run from time to time as a sanity check :D

[majocha]

@T-Gro, I changed the build script to disable parallelization in ci.
I agree running this conservatively in the CI, and with full speed locally makes the most sense for now.

I'll try to solidify the work here. Concentrating on cleaning this up, reenabling the skipped tests, extracting the ones I modified into separate PRs.

[psfinaki]

Thanks Jakub, we're following this :)