Rest API - Source File Upload

[necatiarslan]

Description

It is good to have an API to upload source code files to the Airflow server to make deployments.

Use case/motivation

I am the developer of the Airflow VSCode extension, which developers use to build and test Airflow DAGs locally within VSCode. Currently, before running a DAG for testing, the source code must be deployed to the Airflow server. Developers often rely on their own deployment solutions, tailored to their specific development environments. However, if an API were provided to upload source code files directly, the VSCode extension could automatically upload any changed files and run the DAGs, streamlining the testing process. This is one of the most requested features by users of the extension.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

Uploading of DAGs was not implemented by intend especially in regards of considerations of security as far as I understand - in the past.
I understand for interactive development and debugging this feature might be useful. But as it is a structural change of how Airflow works, I recommend doing this via AIP (Airflow Improvement Process).

There are recently some plans to change DAG deployment in Ariflow 3 under the AIP-66, can you take a look and check if this is the "right" direction? --> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356&src=contextnavpagetreemode

There is a bit more complexity behind the scene. Else it might be a valid topic to be added as plans for future changes towards Ariflow 3. --> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams

[potiuk]

Yep. What @jscheffl wrote. Also converting that one ot a discussion. Also to add - submitting the source code via "productin" REST API directly is not likely to happen even in Airflow 3 - it is considered as violating the basic security model of Airflow https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html - it's quite unlikely the standard REST API will allow to submit the code - because that would mean that API user becomes a DAG Author - and these two roles are pretty distinct and have different capabilities/permissions.

But yes - AIP-66 is a way where - @necatiarslan - your contributions migh help in shaping it for Airlfow 3 in a way that might be good for your extension. I can imagine a "development" pluggable DAG backend that could expose such an API - and you could certainly develop and contribute such pluggable DAG backend and make it works as your extension would find useful. That would expose a "new" non-production API maybe that could accept dag code from outside. It could also be developed by you and released as part of your extension - it will not necessarily have to be maintained and developed in Airflow once pluggable backends are implemented. CC: @jedcunningham - for the use case for AIP-66

[necatiarslan]

Thanks, Guys for your replies.
I understand the security concerns about it. However, this is one of the features most wanted by vscode extension users and will greatly help the airflow dag development and testing. Personally, I am not happy with our current model where I can not hit the run button in the vscode.

It is better to go on this discussion under the scope of AIP-66

[potiuk]

I understand the security concerns about it. However, this is one of the features most wanted by vscode extension users and will greatly help the airflow dag development and testing. Personally, I am not happy with our current model where I can not hit the run button in the vscode.

I think we fully agree on that. Since VSCode extensions users want it badly, VSCode Extension author should implement it. AIP-66 is going to make it possible. Make use of it. This is one of the reasons we are doing it so that people like you can succeed- but it does not mean we will do the job for them. We will expose APIs so that they can spend time to implement things they need

I think we are in full alignment here. Help to make AIP-66 to be what you want and then implement integration. That's the best way forward.