Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for table specific filters different from dataset ones #12

Open
bsushmith opened this issue Mar 28, 2023 · 1 comment
Open

support for table specific filters different from dataset ones #12

bsushmith opened this issue Mar 28, 2023 · 1 comment
Assignees

Comments

@bsushmith
Copy link
Collaborator

Description
Currently, whatever filter is applied for bigquery provider - datasets type, the same filter automatically gets applied to the tables type also in the same provider config.

This is happening due to the way - tables are fetched in bigquery provider.

In bigquery provider, tables are added as part of children for dataset. As part of this filtration for bigquery datasets, tables also get filtered out, since flattening out happens later after filtration and not before.

To Reproduce

  • Create a bigquery provider which has both dataset and table resource types
  • Add a filter only for dataset resource in provider config
  • run the fetch resources job and check resources data

cc/ @utsav14nov

@rahmatrhd
Copy link
Member

to fix this, the filtration should be done after the resources are flattened, is that correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants