chunked data reading #543
Open
chunked data reading #543
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smtp_DATA() could buffer an unbounded amount of data in line_fragments until it got crlf and we're going to throw it away anyway. drop TestSMTPWithController.test_long_line_leak which is now moot
…g loop. DATA_CHUNK takes 3 parameters: data : bytes, decoded_data : Optional[str], last : bool and returns Optional[bytes] response If the hook returns a response prior to the last=True chunk, smtp_DATA will read/discard the remaining data from the client without invoking the hook again. This allows streaming the data to a file or other storage api without having to buffer the whole message in memory first. Move dotstuff and utf8 decode into data reading loop to support this. This may also improve/fix aio-libs#293 which I suspect is due to the tight loop decoding dotstuff for the whole message at once hogging the GIL in the old implementation.
otherwise do it at the end as before so we don't ~double the memory while we're reading from the client
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Add new handler hook DATA_CHUNK which is invoked from the data reading loop. This allows streaming the data to a file or other storage api without having to buffer the whole message in memory first.
Are there changes in behavior for the user?
The new hook is opt-in, there should be no behavior changes for existing users.
Related issue number
This may also improve/fix #293 which I suspect is due to the tight loop decoding dotstuff for the whole message at once hogging the GIL in the old implementation.
Checklist
{py36,py37,py38,py39}-{nocov,cov,diffcov}, qa, docs{py36,py37,py38,py39}-{nocov,cov,diffcov}{py36,py37,py38,py39}-{nocov,cov,diffcov}, pypy3-{nocov,cov}, qa, docs{py36,pypy3}-{nocov,cov,diffcov}, qapy36-{nocov,cov,diffcov}, qa, docsNEWS.rstfile