Support for installable content rendering/parsing behaviour.

[tomchristie]

Prompted by some long gentle thinking over options for "I'd like to be able to customise the JSON parsing" in #717, #1352, #1730.

I can feel that I'm still not sufficiently happy with any of the options explored there, and it's occurred to me that there's something here that plays into a more general feature.

Installable request and response parsing.

There's an awful lot of design space here that'd need careful exploration, but it's already something I'd had on the horizon as it plays into #1091.

I'm only going to offer a sketch of how what I've got in mind here, because exploring the space fully would take much longer, but I can offer the basic idea...

# A client with an installed encoder for custom JSON encoding on uploads...
client = httpx.Client(payload_handlers=...)
r = client.post("https://www.example.com/", payload=...)

We'd also want to offer support for customising the decoding of responses...

client = httpx.Client(payload_handlers=..., auto_parse=True)
r = client.get("https://www.example.com/")
print(r.payload)

This sort of thing would allow us to support customized JSON handling, as part of a much wider set of functionality, that works generally for both requests and responses, and that makes sense either client-side or server-side (The later isn't our primary aim here, but there is an important point about continuing to ensure that httpx.Request(...) and httpx.Response(...) are not overly tied into only working well for client contexts.

Anyways, bit of a preliminary conversation here, but needed to get my brain-thoughts on this out into the open a little.

[aviramha]

In my opinion, the magic of using result.payload is useless. I can just get the data and run it on my own, and covering it for json is simply calling orjson on r.content...
I'd think a better design would be to implement the following:

class BasePayload(ABC):
    CONTENT_TYPE:str = ...

    def build(self) -> bytes:
        ...


class ORJSONPayload(BasePayload):
    CONTENT_TYPE = "application/json"
    def __init__(self, json) -> None:
        self._json = json

    def build(self) -> bytes:
        return orjson.dumps(self._json)


client.post("http://www.example.com", payload=ORJSONPayload({"asd": 1}))

[bentheiii]

I like this approach, but why not then just let the content parameter accept this BasePayload class? Also, what does the ABC above have over, say, a simple protocol?

class BasePayload(Protocol):
  headers: Dict[str, str]
  content: byte

class ORJSONPayload(BasePayload):
    def __init__(self, obj) -> None:
        self.content = orjson.dumps(obj)
        self.headers= {"Content-Type": "application/json", "Content-Length": str(len(self.content)), ...}

client.post("http://www.example.com", content=ORJSONPayload({"asd": 1}))

[bentheiii]

I think it's important to note that having a client with only one payload format can be limiting. As some APIs can have different formats for different routes (for the purpose of this comment I will assume an API that has one route that accepts JSON and another that accepts MSGPACK). For this reason, I think that @aviramha has the right idea of providing the content bundled with the type per-request.

To make things easier, the client can accept a content_serializer parameter, that can adapt non-bytes content to a serialized value.

Content = Union[str, bytes, SyncByteStream, AsyncByteStream]

class BasePayload(ABC):
  headers: Dict[str, str]
  content: Content

ContentSerializer = Callable[[Any], Union[Content, BasePayload]]

def encode_content(content, content_serializer: Optional[ContentSerializer]):
  if isinstance(content, (bytes, str)):
    ...
  elif isinstance(content, BasePayload):
   # I chose a static-typing/ABC approach to payloads, a duck-typing/protocol approach is equally good
   return content.headers, content.content
  elif content_serializer:
   return encode_content(content_serializer(content), None)
  raise TypeError(...)
  
def default_content_serializer(content: Any):
  if isinstance(content, Iterable):
   ...
  elif isinstance(content, AsyncIterable):
   ...
  raise TypeError(...)

# Usage example

class ORJSONPayload(BasePayload):
    def __init__(self, obj) -> None:
        self.content = orjson.dumps(obj)
        self.headers= {"Content-Type": "application/json", "Content-Length": str(len(self.content)), ...}

class MSGPackPayload(BasePayload):
    def __init__(self, obj) -> None:
        self.content = msgpack.packb(obj)
        self.headers= {"Content-Type": "application/msgpack", "Content-Length": str(len(self.content)), ...}

client = Client(content_serializer=ORJSONPayload)
client.post(content={'a': 1})  # send a JSON with the default serializer
client.post(content=MSGPackPayload({'a': 1}))  # send a MSGPack messagewith a pre-built payload

[aviramha]

I like it.
@tomchristie WDYT?