Feature Request: Support Task Results as Disk Files with Automatic Cloud Storage Caching

[xiemeilong]

Describe the current behavior

not support

Describe the proposed behavior

Adding support for task results to be large disk files, and for the prefect to automatically handle caching these files in storage (like S3) for efficient sharing and access across tasks and workflows, especially in distributed environments. Because some task can only produce and consume files on disk and need to run across machines.

Example Use

@task
def produce_file():
     ....
     return S3Cache(path_on_disk)

@flow
def consume_file(path_on_disk:Path):
     ....

@flow
def process_file():
     file = produce_file()
     run_deployment(name="consume_file", parameters= {path_on_disk: file})

Additional context

No response

[zzstoatzz]

hi @xiemeilong - have you checked out results?

https://docs.prefect.io/v3/develop/results#manage-results

that would seem like what you're looking for

[xiemeilong]

No, you haven't grasped the key information I described. The files are produced and consumed by other programs in file format, and they are very large. Therefore, if I were to use the result cache directly, I would need to read the large files, and return it for Prefect serialize them again . Then, on consumption, I'd have to go through the reverse process: deserialize and write them back out.

[zzstoatzz]

hi @xiemeilong

No, you haven't grasped the key information I described

I'm going on what you've described thus far, so perhaps you could elaborate on what exactly you're trying to do, because this is an accurate description of what Prefect Results are designed for:

Adding task results to be large disk files, and for the prefect to automatically handle caching these files in storage (like S3) for efficient sharing and access across tasks and workflows, especially in distributed environments

it seems you're saying that these results are not produced by prefect tasks

The files are produced and consumed by other programs in file format, and they are very large

but then that makes this request confusing, what do you mean by "task results", and what behavior do you want from prefect?

Adding support for task results to be large disk files

[xiemeilong]

hi @xiemeilong

No, you haven't grasped the key information I described

I'm going on what you've described thus far, so perhaps you could elaborate on what exactly you're trying to do, because this is an accurate description of what Prefect Results are designed for:

Adding task results to be large disk files, and for the prefect to automatically handle caching these files in storage (like S3) for efficient sharing and access across tasks and workflows, especially in distributed environments

it seems you're saying that these results are not produced by prefect tasks

The file is produce by other program in the prefect task, for example, some shell command

The files are produced and consumed by other programs in file format, and they are very large

but then that makes this request confusing, what do you mean by "task results", and what behavior do you want from prefect?'

I need to process workflows that span multiple machines. I'm calling external commands that generate or consume files on disk within prefect tasks. I want Prefect to automatically handle the transfer of these files between tasks across machines, and I'd like this file to be cached as Prefect task results .

Adding support for task results to be large disk files