CI/CD bot support for GitLab

[almereyda]

It could be nice to build MR-based DataOps workflows in GitLab.

Is this a considerable option? I've seen that the CI/CD bot already knows about a type: github, which invites for additions. Would type: gitlab be generally favoured and considered useful to be included in the roadmap?

[isaiahgm]

Second this, GitLab support is blocking us from using SQL Mesh to its fullest potential. Our company exclusively hosts code on GitLab, so the GitHub CI/CD isn't an option.

[almereyda]

Thanks to the well structured, modular and documented code in a self-contained repository, we find all artifacts to adapt in cleanly separated places:

• sqlmesh/docs/integrations/github at main · TobikoData/sqlmesh
• sqlmesh/sqlmesh/cicd at main · TobikoData/sqlmesh
• sqlmesh/sqlmesh/integrations/github/cicd at main · TobikoData/sqlmesh
• sqlmesh/tests/integrations/github/cicd at main · TobikoData/sqlmesh

How much effort would we assume it takes to adapt all this to the GitLab GraphQL API? Could the work be distributed across interested developers? While screenshots for the docs can only be generated after the fact, it may be useful to consider writing the tests first and adapting the CI/CD bot and the gitlab integration from the perspective of the expected API surface?

[eakmanrq]

Is this a considerable option? I've seen that the CI/CD bot already knows about a type: github, which invites for additions.

Yes that is correct that this was designed to help facilitate more bots in the future. Definitely open to contributions to add bots for other platforms. It would require some extensive tests though (similar to the current bot) in order to help with the support burden of having these different bots.

How much effort would we assume it takes to adapt all this to the GitLab GraphQL API?

Sadly I'm just not familiar with the Gitlab GraphQL API to have a sense on that complexity. Sadly I'm also not familiar with Gitlab at all since I've never used it. So this is the biggest issue for me personally to estimate the complexity involved. I'm also curious what features from the Github bot would also work on Gitlab given the constraints of that platform.

it may be useful to consider writing the tests first and adapting the CI/CD bot and the gitlab integration from the perspective of the expected API surface?

Yeah I think this is a good idea. I think the task might be finding each API request in the Github bot and then mapping that to a corresponding gitlab endpoint. That would also let us know if there is any functionality gap in Gitlab that we should be aware of before investing further. Then tests could be written to mock out those endpoints and verify the expected behavior.

I think this would be great contribution to the SQLMesh project. The two key things that I would want to make sure we are tracking are:

• What are the functionality differences between the two bots and is the functionality that is possible on Gitlab enough to justify supporting it?
• How can we minimize the support burden for OSS developers and not require deep Gitlab expertise to do standard maintenance improvements to the bot over time?

[isaiahgm]

I'd be happy to contribute some time to help develop this, but I am also pretty unfamiliar with developing gitlab bots. My experience ends at writing simple CI/CD scripts for GitLab.

[riordan]

Has anyone had the chance to look into Gitlab support? Taking the path of Github & Airflow is absolutely the right path to adoption but as a Gitlab & Dagster user, I'd love to get the full SQLMesh experience too.

[tobymao]

We don’t currently have plans for gitlab, but dagster support will be a part of Tobiko Cloud. We do offer feature prioritization as a part of our Cloud / Enterprise offering. Please contact me on Slack if you’re interested in discussing.