Run acceptance tests against Prefect server in Docker Compose

[mitchnielsen]

Summary

In #186, we added support for running Prefect in Docker Compose and being able to test terraform plan/apply commands against it.

This is helpful for local development and testing, but we may also be able to use this in CI.

We could replace the PREFECT_API_URL value in the CI file to point to something like http://localhost:4200, and tell CI to spin up Prefect using the Docker Compose configuration.

This means we'd be testing in an ephemeral instance of Prefect instead of against dev/staging environments, which can cause transient errors due to environment pollution.

One caveat mentioned by @parkedwards:

just a caveat - one wrinkle is that our OSS server isn't an exact match/port of our SaaS (which the current acceptance tests are mostly oriented towards supporting) - the OSS server has a subset of features and endpoints. so there may be some extra potholes we hit as we try to marry the tests<>this local prefect server instance. but i like where this is going

Acceptance criteria

• Acceptance tests can be run against an instance of Prefect started via Docker Compose
• CI uses this same setup (we may choose to break this into a separate PR)

[parkedwards]

something to think about - not sure if we can do this with golang tests, but in pytest you can create test groups and invoke them separately. i really do like this local testing facility, but if it only supports non-Cloud-specific features/resource, maybe we should group our tests to be OSS vs. Cloud and run them with separate make commands. Cloud would be the full thing, but OSS would be a subset - the benefit would be that we're testing provider compatibility with OSS, but also shorter feedback loop. and then the Cloud tests can run in CI/CD

[mitchnielsen]

Yeah good points, we'll have to delineate which ones are Cloud-only for sure.

I do think the OSS tests could run in CI/CD too if we wanted, since all we need is that prefecthq/prefect image running.

[mitchnielsen]

As for grouping them in Golang, I think there's usually a few options:

• Build tags and then calling go test -tags=<tag>
• Common test name prefix and then calling go test -run CommonPrefix ...
• Separate directories?
• Others?

[mitchnielsen]

https://pkg.go.dev/testing#hdr-Skipping

[parkedwards]

tags sound like the way to go 🙌

[mitchnielsen]

After some discussion, it may be more valuable to prioritize #218 for now as that would increase the reliability of tests in Prefect Cloud (which can run all tests, rather than just a subset that could run when testing against OSS).