Column lineage implementation

[Michae94]

Hey there,
I am currently trying to build a column lineage tool on my data model.
My current stack is Snowflake + dbt.

I did not found any straightforward "ready to use function" to do so, but I think I have everything needed from the library:

• the lineage function to build a lineage from a sql file for every column. All the sql files are retrieved from our dbt target folder. However you need sources + output columns as input of the lineage function to create the column lineage.
  • sources: can be retrieved using manifest.json of dbt
  • output columns in catalog.json of dbt.
• the additionals columns used, for example in join/where/qualify (using find_all(exp.Columns) + filter only on join/where/qualify)

Finally, snowflake provide already this kind of Column lineage, but it seems there are some limitations (window function), which is actually supported by sqlglot library.
I will update the post if I have something working correctly and consistently.

[tobymao]

Try using SQLMesh which has built in column level lineage and supports dbt projects.

Here's a blog post on the UI which would probably be interesting to you https://tobikodata.com/sqlmesh-ui.html.

Feel free to join our slack channel to learn more.