Automatically disable p4tc tests when the necessary environment is missing

[fruffy]

Do not add P4TC STF tests when certain components are missing. Currently this is just brctland clang-15, but there might be other things we can check for.

Also add some typing to the run-tc-test.py and fix some incorrect error output and handling.

@vbnogueira Could you give this a review?

[vlstill]

With the caveat I have no TC knowledge the changes look good to me, but I don't see how are the python changes related. I'll wait from someone from TC to take a look before I approve.

[fruffy]

With the caveat I have no TC knowledge the changes look good to me, but I don't see how are the python changes related. I'll wait from someone from TC to take a look before I approve.

The TC script had a problem where they caused exceptions to be thrown when a binary was missing. The Python changes handle these exceptions gracefully. Let me move them to a separate PR.

[vbnogueira]

Unless I'm misunderstanding something, with this PR the p4tc compilation tests have stopped running even when the p4tc tag is on

[fruffy]

Unless I'm misunderstanding something, with this PR the p4tc compilation tests have stopped running even when the p4tc tag is on

Guess I was a bit too aggressive. Need to have two separate checks for the normal compilation tests and the STF ones.

[fruffy]

@vbnogueira I remember now why the tests were disabled. Could you look at the failures and tell me on what system or CI they are meant to be run?

[vbnogueira]

@vbnogueira I remember now why the tests were disabled. Could you look at the failures and tell me on what system or CI they are meant to be run?

So the mac tests shouldn't run any of the p4tc compilation tests
The fedora one should work fine and the ubuntu 18.04 as well
If you look at the last PR, the p4tc tests ran succesfully (https://github.com/p4lang/p4c/pull/5172/checks)
Looking at the fedora fail and the escape issue seems very strange
It's even stranger given the fact that this line has been there since sosutha's original commit, I just moved it to another file
I looked at the changes you did to the python script, but couldn't identify any obvious culprit
Trying to figure the ubuntu issue as well

[vbnogueira]

Trying to figure the ubuntu issue as well

It seems like the ubuntu-18 test was skipped on all the previous PRs, so we didn't catch this
The kernel they have in the ubuntu-18 image is quite old and struct __sk_buff doesn't have the tstamp field which was included in 2018
So we should avoid running the p4tc tests in ubuntu-18 as well

[fruffy]

So we should avoid running the p4tc tests in ubuntu-18 as well

Let me see how we can check for that. Ideally I want to get to a point where we do not have to add all these filters to ctest.

[vbnogueira]

So we should avoid running the p4tc tests in ubuntu-18 as well

Let me see how we can check for that. Ideally I want to get to a point where we do not have to add all these filters to ctest.

For ubuntu, you can try checking whether the linux-libc-dev version is greater than 5.0
Dummy example in bash:

VERSION=$(dpkg-query -W -f='${Version}' linux-libc-dev)

if dpkg --compare-versions "$VERSION" gt 5.0; then
    echo "linux-libc-dev is greater than 5.0"
else
    echo "linux-libc-dev is not greater than 5.0"
fi

[fruffy]

It looks like tests are now running as expected and are also being filtered correctly. @vbnogueira let me know whether the CMake requirements and changes make sense to you.

Edit: Actually need to make sure to check the package correctly to filter the 18.04 tests.

[vbnogueira]

It looks like tests are now running as expected and are also being filtered correctly. @vbnogueira let me know whether the CMake requirements and changes make sense to you.

I'll wait for the tests to finish to approve, but it looks ok to me

[fruffy]

I'll wait for the tests to finish to approve, but it looks ok to me

Working on some tweaks which are causing failures. Hope to resolve that soon.

The problem is when you run in Docker, the kernel version can be mismatched with the Linux headers, which is what I am running into here.

[vbnogueira]

Just a minor nit, shouldn't you also check the linux libc version as well here for consistency?

[fruffy]

Was thinking about this, yeah. But wasn't sure whether that could have unintended side effects. I think it is safe though.

[vbnogueira]

I see
I don't think there would be any side effects, the check shouldn't cause any trouble
It's just that you could encounter the same issue in the future where the host's kernel version is supported, but the libc version in the container isn't for the ebpf backend as well
It should be a rare case, but could happen

[fruffy]

There was a bunch of commit spam because I ran into an (unrelated) bug. The latest version should include the additional check.

[vbnogueira]

Looks good to me