feat: Log own custom rules

[sadahry]

This is a:

• bug fix PR with no breaking changes
• new functionality

Link to Issue

#407

Description & motivation

Enable also to log error of our own rules by print_dbt_project_evaluator_issues

Integration Test Screenshot

./run_test.sh snowflake

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
  • Databricks
  • DuckDB
  • Trino/Starburst
• I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)

Usecase

My custom rules is like that.

{{ dbt project dir }}/models/dbt_project_evaluator/fct_public_models_without_exposures.sql

with
    public_models as (
        select *
        from {{ ref("dbt_project_evaluator", "int_all_graph_resources") }}
        where resource_type = 'model' and is_public and not is_excluded
    ),
    models_with_exposure as (
        select direct_parent_id as resource_id
        from {{ ref("dbt_project_evaluator", "base_exposure_relationships") }}
    ),
    final as (
        select resource_name as model_name, is_public, false as has_exposure
        from public_models m
        where
            not exists (
                select 1 from models_with_exposure e where m.resource_id = e.resource_id
            )
    )

select *
from final {{ dbt_project_evaluator.filter_exceptions(model.name) }}

{{ dbt project dir }}/models/dbt_project_evaluator/governance.yml

models:
  - name: fct_public_models_without_exposures
    description: This table shows each public model that does not have an exposure
    tests:
      - dbt_project_evaluator.is_empty:
          config:
            severity: "{{ env_var('DBT_PROJECT_EVALUATOR_SEVERITY', 'error') }}"
            store_failures_as: ephemeral

Log example

logged like that (using fqn because package my test alias show random_strings like dbt_project_evaluator_is_empty_{{ random_strings }})

### List of issues raised by dbt_project_evaluator ###

-- my_dbt_package.dbt_project_evaluator.dbt_project_evaluator_is_empty_fct_public_models_without_exposures_ --
| MODEL_NAME                         | IS_PUBLIC | HAS_EXPOSURE |
| ---------------------------------- | --------- | ------------ |
| workspace_sandbox_dbt_sample_model |      True |        False |


-- dbt_project_evaluator.marts.governance.is_empty_fct_public_models_without_contract_ --
| RESOURCE_NAME                      | IS_PUBLIC | IS_CONTRACT_ENFORCED |
| ---------------------------------- | --------- | -------------------- |
| workspace_sandbox_dbt_sample_model |      True |                False |


-- dbt_project_evaluator.marts.governance.is_empty_fct_undocumented_public_models_ --
| RESOURCE_NAME                      | ACCESS | IS_DESCRIBED | TOTAL_DEFINED_COLUMNS | TOTAL_DESCRIBED_COLUMNS |
| ---------------------------------- | ------ | ------------ | --------------------- | ----------------------- |
| workspace_sandbox_dbt_sample_model | public |         True |                     1 |                       0 |

[dave-connors-3]

@sadahry this looks good! do you mind adding a quick bit of documentation here to describe that if you add a custom model and use the dbt_project_evaluator.is_empty test, you can see your own custom rules in the log output?

[sadahry]

@dave-connors-3 Thank you for your review!

Sure, I'll explain and let you know once it's done.

[sadahry]

@dave-connors-3 described it. Kindly confirm it!

[sadahry]

It looks like a CI error occurred when I ran the trino test more than once

https://app.circleci.com/pipelines/github/dbt-labs/dbt-project-evaluator/813/workflows/00bbeeee-94aa-4551-b186-8320fc74d443/jobs/3738/steps

14:38:43    Database Error in model dim_model_7 (models/marts/intermediate/dim_model_7.sql)
  TrinoUserError(type=USER_ERROR, name=TABLE_ALREADY_EXISTS, message="line 6:5: Destination table '*************.dbt_project_evaluator_integration_tests_trino.dim_model_7__dbt_tmp' already exists", query_id=20240123_143830_81812_gp3ip)

[b-per]

I re-ran the test and it was just a transient one on Trino.

The one failing on Databricks is known due to some pandas dependencies failiing in the latest dbt-databricks

[sadahry]

All checks have passed (by @b-per or someone's retry)
Thank you!

[graciegoheen]

Let's ship it! Thanks for all the work here!