encode_content prevents requests with content=encode_multipart_data(...) from sending by AsyncClient

[wesley800]

This problem is basically an extension of #2341 . Suppose one wants to send a POST request with multipart-encoded form data (but NOT uploading files), without any cookie sessions. One way to do that is to build something like:

ac = httpx.AsyncClient()
formdata = {"user":"John","pwd":"secret!"}
header, content = httpx._content.encode_multipart_data(formdata, {}, b'----WebkitRandomBoundary')
req = httpx.Request("POST", 'https://example.org/login', content=content, headers=header)
res = await ac.send(req, auth=None, follow_redirects=False)

ac.send() would definitely check if req.stream implements AsyncByteStream (actually it's checked in ac._send_single_request()), which surprisingly fail. But it goes flawlessly when using httpx.Client() to send the request.

Although content is of type httpx._multipart.MultipartStream, and does implement AsyncByteStream. But in encode_content called by the constructure of httpx.Request, content is wrapped again into IteratorByteStream, which drops the impl to AsyncByteStream. For now it can be hacked by directly assign stream=content and manually call the "private" method req._prepare(header), which seems to not be a good practice at least to me.

I guess there's two solutions. One is to add a AsyncRequest type, as well as the corresponding encode_async_request. The other is to build a new interface SyncAndAsyncIteratorByteStream and change encode_content to check first if the data origin supports both Iterable and AsyncIterable first.

[tomchristie]

I'm going to push this into a discussion, since you're using a private API...

httpx._content.encode_multipart_data
#     ^ The underscore indicates that you're accessing a private attribute.
#       Our public API is always available directly as `httpx.<...>`

I'd tweak what you're doing like this...

header, stream = httpx._content.encode_multipart_data(formdata, {}, b'----WebkitRandomBoundary')
content = b''.join[(part for part in stream)]  # You're not actually using async IO here, so just grab the content as bytes.
req = httpx.Request("POST", 'https://example.org/login', content=content, headers=header)

The other is to build a new interface SyncAndAsyncIteratorByteStream and change encode_content to check first if the data origin supports both Iterable and AsyncIterable first.

True. There's possibly? an issue here at the public API level, if we decided that "content = ..." should support instances that are both iterators and async iterators. Not sure really.

[wesley800]

Exactly. Also I'm currently using content = stream.read() to convert the encoded formdata to bytes.

The reason I'm calling methods under private httpx._content is discribed in #2341 as I've mentioned at the beginning. I'd be more than happy if the private api call replaced.

I'm not sure about the public API definition either. But it does seems a little wierd to break on some random dual (a/sync) iter classes.

[tomchristie]

The reason I'm calling methods under private httpx._content is discribed in #2341

Okay, so there's a different question there, which is...

"How can force the encoding to multipart if I don't have any files to upload"

The answer to that is that you can't. Which is the same behaviour as requests uses.

• need to send request in multipart/form-data without files and specify mine boundary psf/requests#1423
• Generate multipart posts without a file psf/requests#1081

You can get around this with a True-evaluating empty dict. Which is odd, but does the trick.

See psf/requests#1081 (comment)

That'll work for either httpx or for requests.

We could change our behaviour there, but it'd need some discussion. Firstly, is there a sufficient motivation? (Why do you need this) And secondly what would our API be for switching between the two cases? (Perhaps treat files={} differently to files=None? Perhaps switch if an explicit Content-Type header is provided?)

[wesley800]

Thank you for the detailed information. I really missed out those "ancient" issues (and seems also does Google :) ).

I totally agree that public API needs much care before a modification. In fact i'm not referring to the API itself, instead to the logic of checking implement of SyncByteStream and AsyncByteStream, putting the former before the latter while both implement are not exclusive to each other. Cuz it's internal logic from _type, this is more a personal feeling than a real doubt to the design.