Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write raw results from flows #15434

Merged
merged 5 commits into from
Sep 26, 2024
Merged

Write raw results from flows #15434

merged 5 commits into from
Sep 26, 2024

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Sep 19, 2024

This PR updates flows to write serialized versions of their return values without persisting any metadata to disk. Previously, flows would write result files containing metadata like the serializer used to create the file's result payload. The same info already persisted to the API is used when calling .result on a flow's state.

To have the ResultStore persist only raw results, I created a NullFileSystem class that looks like a WritableFileSystem, but doesn't do anything. The ResultStore for each flow is given a NullFileSystem for metadata_storage, which results in no metadata being written.

To allow a ResultStore to handle raw results, a serializer kwarg is added to the read methods. This option allows the caller to manually provide which serializer to use to deserialize the result. The new kwarg is used internally when loading a result via a flow state's .result method.

Copy link

codspeed-hq bot commented Sep 19, 2024

CodSpeed Performance Report

Merging #15434 will not alter performance

Comparing raw-flow-results (6a1c262) with main (d82cc43)

Summary

✅ 3 untouched benchmarks

@desertaxle desertaxle marked this pull request as ready for review September 21, 2024 03:41
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea i like this approach - prevents any hard-coding that we can't walk back from, but achieves the desired result (ha) for now

@desertaxle desertaxle merged commit a15bd82 into main Sep 26, 2024
33 checks passed
@desertaxle desertaxle deleted the raw-flow-results branch September 26, 2024 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants