MWAA - CopyFromExternalStageToSnowflakeOperator - Failed with error "Invalid regular expression pattern in airflow"

[GomathiMa]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow Version:2.5.1

While using MWAA, CopyFromExternalStageToSnowflakeOperator: Failed with error "Invalid regular expression pattern in airflow"

Error
snowflake.connector.errors.ProgrammingError: 091003 (22000):

Note: I am trying to use {{ ds_nodash }} variable in pattern. This variable value is not getting replaced in the pattern while generating copy into statement. However variable getting printed correctly in echo and works well with Snowflake operator - COPY TO as sql.

What you think should happen instead

No response

How to reproduce

Use CopyFromExternalStageToSnowflakeOperator and have pattern value contains the variable {{ ds_nodash }}

Operating System

Windows

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

You have to check if the fields is in template_fields. If not you can extend the operator and add it to your template_fields. The easiest way Extend the operator:

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#concepts-jinja-templating

class MyOperator(OriginalOperator):
    templated_fields = () # <- copy fields from the original one and the one you are missing

And use MyOperator instead of the OriginalOperator

You can also likely do in your DAG:

OriginalOperator.templated_fields = () # <- here add the list of fields you want to be templated.

[Taragolis]

@GomathiMa, in additional you could propose changes into operator templated fields by submitting a PR, seems like no one attribute processed in the operator constructor so it should be easy changes, for example in the last week same changes in the other operator was approved pretty quick: #33585

[angelarosario]

Thanks for this.
For anyone else doing the second method, the correct attribute name is template_fields
OriginalOperator.template_fields = () # <- here add the list of fields you want to be templated.