Discrepancy between the number of built and executed objects in dbt build with unit tests

[sadeka4]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

While executing the DBT build command, I can see contradictory outputs.
e.g., I can see this.
"188 of 208 OK created sql view model my_dataset.fct_missed_sales_score_daily_v"

However, the execution stops after the 188th object as reported above.
And DBT reports the following afterwards.

"Done. PASS=188 WARN=0 ERROR=0 SKIP=0 TOTAL=188"

Just wondering: why the execution stops after the 188th object? What are the rest of the objects?
Any help would be highly appreciated!

Thanks!

Expected Behavior

I believe the the number of objects should be the same in both cases.

I'd like to see all the 208 objects executed and reported while running the dbt build command.

Steps To Reproduce

In our DBT project, we have models, tests, unit tests, seeds and snapshots. And the command looks like this.

"dbt build --target my_core --select tag:daily tag:intra-day"

Relevant log output

No response

Environment

- OS: Unix
- Python: 3.10
- dbt: Running with dbt=1.8.5, Registered adapter: bigquery=1.9.0

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

[dbeatty10]

Thanks for reporting this @sadeka4 !

I suspect that there is bug in dbt and 208 is an overcount and 188 is actually the correct number of objects to build. See below for some simplified examples showing an overcount.

Reprex

Create these files:

models/my_model_1.sql

select 1 as id

models/my_model_2.sql

select 1 as id

models/_properties.yml

models:
  - name: my_model_1
    config:
      tags: ["daily", "weekly"]
    columns:
      - name: id
        tests:
          - unique

unit_tests:
  - name: test_1
    model: my_model_1
    given: []
    expect:
        rows:
          - {id: 1}

  - name: test_2
    model: my_model_2
    given: []
    expect:
        rows:
          - {id: 1}

Run these commands:

dbt list -q
dbt list -q | wc -l
dbt list -q --select tag:daily
dbt list -q --select tag:daily | wc -l

See that there are 5 and 3 nodes selected, respectively.

Run these commands:

dbt build
dbt build --select tag:daily

Here's the output:

$ dbt build
22:46:04  Running with dbt=1.9.1
22:46:08  Registered adapter: duckdb=1.9.1
22:46:09  Found 2 models, 1 test, 422 macros, 2 unit tests
22:46:09  
22:46:09  Concurrency: 1 threads (target='duckdb')
22:46:09  
22:46:09  1 of 9 START unit_test my_model_1::test_1 ...................................... [RUN]
22:46:09  1 of 9 PASS my_model_1::test_1 ................................................. [PASS in 0.13s]
22:46:09  2 of 9 START sql view model dbt_dbeatty.my_model_1 ............................. [RUN]
22:46:09  2 of 9 OK created sql view model dbt_dbeatty.my_model_1 ........................ [OK in 0.11s]
22:46:09  3 of 9 START unit_test my_model_2::test_2 ...................................... [RUN]
22:46:09  3 of 9 PASS my_model_2::test_2 ................................................. [PASS in 0.10s]
22:46:09  5 of 9 START sql view model dbt_dbeatty.my_model_2 ............................. [RUN]
22:46:09  5 of 9 OK created sql view model dbt_dbeatty.my_model_2 ........................ [OK in 0.07s]
22:46:09  4 of 9 START test unique_my_model_1_id ......................................... [RUN]
22:46:09  4 of 9 PASS unique_my_model_1_id ............................................... [PASS in 0.06s]
22:46:09  
22:46:09  Finished running 1 test, 2 unit tests, 2 view models in 0 hours 0 minutes and 0.67 seconds (0.67s).
22:46:10  
22:46:10  Completed successfully
22:46:10  
22:46:10  Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5

$ dbt build --select tag:daily          
22:49:20  Running with dbt=1.9.1
22:49:24  Registered adapter: duckdb=1.9.1
22:49:24  Found 2 models, 1 test, 422 macros, 2 unit tests
22:49:24  
22:49:24  Concurrency: 1 threads (target='duckdb')
22:49:24  
22:49:24  1 of 4 START unit_test my_model_1::test_1 ...................................... [RUN]
22:49:25  1 of 4 PASS my_model_1::test_1 ................................................. [PASS in 0.13s]
22:49:25  2 of 4 START sql view model dbt_dbeatty.my_model_1 ............................. [RUN]
22:49:25  2 of 4 OK created sql view model dbt_dbeatty.my_model_1 ........................ [OK in 0.11s]
22:49:25  3 of 4 START test unique_my_model_1_id ......................................... [RUN]
22:49:25  3 of 4 PASS unique_my_model_1_id ............................................... [PASS in 0.04s]
22:49:25  
22:49:25  Finished running 1 test, 1 unit test, 1 view model in 0 hours 0 minutes and 0.46 seconds (0.46s).
22:49:25  
22:49:25  Completed successfully
22:49:25  
22:49:25  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3

See that the first says "5 of 9". The 5 is what we expect and the 9 is too many!

The second says "3 of 4". Likewise, the 3 is what we expect and the 4 is one too much.

Relevant code

I think a relevant area of the code is here:

dbt-core/core/dbt/task/build.py

Line 93 in 8a8857a

self.num_nodes = self.num_nodes + len(self.selected_unit_tests)

I think that some nodes are coverted in both self.num_nodes and self.selected_unit_tests, leading to an overcount.

[sadeka4]

Thank you, @dbeatty10 . Makes sense now.

[dbeatty10]

Although it doesn't appear so on the surface, I think might be related to #10267.