Update WebRTC message size limit #628
Conversation
The browser bugs that made a 16KiB message size limit necessary have long been fixed. Increase the max size to 256KiB which has been verified experimentally between Firefox/Chrome and update the link to the Mozilla blog entry that discusses the limit and the relevant WebRTC issue that enables 256KiB messsages.
| Implementations MAY choose to send smaller messages, e.g. to reduce delays | ||
| sending _flagged_ messages. |
There was a problem hiding this comment.
Should we account for forwards compatibility with future implementations that aren't arbitrarily limited like browsers currently?
| Implementations MAY choose to send smaller messages, e.g. to reduce delays | |
| sending _flagged_ messages. | |
| Implementations MAY choose to send smaller messages, e.g. to reduce delays | |
| sending _flagged_ messages, and they MAY choose to accept larger messages | |
| to allow for forwards compatibility with more capable implementations in the | |
| future. |
A "be liberal in what you accept, conservative in what you emit" sort of thing.
There was a problem hiding this comment.
I'd prefer separate sentences since they're both important and about different things.
Implementations MAY choose to send smaller messages, e.g. to reduce delays
sending _flagged_ messages.
Implementations MAY choose to accept larger messages
to allow for forwards compatibility with more capable implementations in the
future.
| Encoded messages including their length prefix MUST NOT exceed 256kiB to support | ||
| all major browsers. See ["Large Data Channel Messages"](https://blog.mozilla.org/webrtc/large-data-channel-messages/) | ||
| and [WebRTC#40644524](https://issues.webrtc.org/issues/40644524). |
There was a problem hiding this comment.
Based on internal discussions, I believe we've decided to keep it at 16KiB to remain compatible with Kubo 0.30.0.
@achingbrain what is the next step here? figure out next steps without breaking interop? (e.g. how to negotiate higher size limit per connection via something like RTCSctpTransport/maxMessageSize?)
There was a problem hiding this comment.
I’m doing some experiments in the test plans repo to validate that we do in fact see a throughput increase with larger messages.
Marking this as a draft pending the outcome of that.
There was a problem hiding this comment.
See linked performance graphs here: #628 (comment)
|
I'm curious, what was the conclusion with this PR and the potential throughput increase? |
|
The result was a roughly 30% speed increase when going from 16KB to 265KB chunks. See "Throughput" at the bottom of this page: https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=a44f101803381ea2547e2eb44abc7fb6279d41f2#branch
I think this is still worth doing but we need the go/rust implementations to accept larger chunks - I think rust-libp2p will still reject them. |
|
Here's the draft pr @sukunrt created for go-libp2p: libp2p/go-libp2p#2949 Is everyone agreed that we want to increase the message size? If so, we can proceed with merging the go & rust pr |
|
@p-shahi I think we should increase the message size. WebRTC is by far the slowest libp2p transport but also one of the most useful since it's the only one that lets browsers dial each other directly, so anything that speeds it up is very welcome. |
There was a problem hiding this comment.
I think there's a way to make this backwards compatible for /webrtc-direct using the datachannel 0 that we use for handshaking.
-
Both server and client use SDPs that indicate that peer supports 256kB messages.
-
sever and client use the noise handshake early data extension to send the max message size.
-
Depending on the max message size received in the handshake we the server and client decide whether to actually send 256kB messages or 16 kB messages.
| @@ -88,9 +88,9 @@ QUIC. The libp2p WebRTC message framing is not concerned with flow-control and | |||
| thus does not need the `RESET_STREAM` frame to be send in reply to a | |||
There was a problem hiding this comment.
Can you edit the webrtc-direct file to remove this line where we recommend a max-message size of 16kB
https://github.com/libp2p/specs/blob/1253e2a1deec908037fba924fcd350ec16df13a6/webrtc/webrtc-direct.md?plain=1#L79
| Encoded messages including their length prefix MUST NOT exceed 16kiB to support | ||
| all major browsers. See ["Understanding message size | ||
| limits"](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_data_channels#understanding_message_size_limits). | ||
| Encoded messages including their length prefix MUST NOT exceed 256kiB to support |
There was a problem hiding this comment.
Why 256kB, can we support more? Can we add links for the browser specific limts here?
| Implementations MAY choose to send smaller messages, e.g. to reduce delays | ||
| sending _flagged_ messages. |
There was a problem hiding this comment.
I'd prefer separate sentences since they're both important and about different things.
Implementations MAY choose to send smaller messages, e.g. to reduce delays
sending _flagged_ messages.
Implementations MAY choose to accept larger messages
to allow for forwards compatibility with more capable implementations in the
future.
In experiments with js we've found that increasing the message size increases throughput. See: libp2p/specs#628 (comment) for details. This changes the protobuf reader for the stream to read 256kB messages. This isn't enough to support larger messages. We most likely need to change the inferred SDP of the server to use 256kB maxMessageSize, and need some backwards compatible mechanism in the handshake to opt in to large messages. See: libp2p/specs#628 for details
In experiments with js we've found that increasing the message size increases throughput. See: libp2p/specs#628 (comment) for details. This changes the protobuf reader for the stream to read 256kB messages. This also forces a change to the connection SCTP read buffer to be increased to about 2.5 MB, to support 1 message being buffered for 10 streams. This isn't enough to support larger messages. We most likely need to change the inferred SDP of the server to use 256kB maxMessageSize, and need some backwards compatible mechanism in the handshake to opt in to large messages. See: libp2p/specs#628 for details
The browser bugs that made a 16KiB message size limit necessary have long been fixed.
Increase the max size to 256KiB which has been verified experimentally between Firefox/Chrome and update the link to the Mozilla blog entry that discusses the limit and the relevant WebRTC issue that enables 256KiB messages.
Implementation status