status_analytics : adding dbt transformation #27
Conversation
|
Need Mixpannel DBT transformation to be merged first : |
There was a problem hiding this comment.
To avoid having to many different dbt run, let's look at a way to group then, else with the other data source it's going to get messier : https://docs.getdbt.com/reference/node-selection/yaml-selectors#difference-between---select-and---selector
status_analytics.py
Outdated
| # Launch DBT transformation for Mixpanel models | ||
| dbt_transform_mixpanel = BashOperator( | ||
| task_id='dbt_run_models_mixpanel', | ||
| bash_command='dbt run --select mixpannel', | ||
| env=INTERNAL_ENV, | ||
| append_env=True | ||
| ) | ||
|
|
||
| # Launch DBT transformation for Google Play Store models | ||
| dbt_transform_playstore = BashOperator( | ||
| task_id='dbt_run_models_playstore', | ||
| bash_command='dbt run --select google_store', | ||
| env=INTERNAL_ENV, | ||
| append_env=True | ||
| ) |
There was a problem hiding this comment.
Let's look into the --selectors usage to group mixpanel and google_store, wdyt ?
There was a problem hiding this comment.
I think we can just run the models in the same command, I just did a commit for this
|
Also please add reference to the issue, like that: |
We can try in a way it is cleaner but it also gives less visibility on what we are running |
Well we don't know which exact models are run when looking at Airflow Dags, but when looking at DBT, we will know that the models are linked. |
* status_analytics : adding dbt transformation for mixpannel and google play store * status_analytics : run the dbt models in the same command * Add selector to group the dbt run of analytics models Signed-off-by: Alexis Pentori <[email protected]>
Status Analytics: Adding DBT transformation to the DAG for Mixpannel and Google play store