[MQTT-SN] No way to give back pressure

[TorstenRobitzki]

The current design of the MQTT-SN library just covers the case, that the library has to wait for input. The case, where the library has to wait for bandwidth on the outgoing link is (nearly) not covered.

If the library finds the output link being saturated, it will wait and retry. This leads then to either very poor performance (when that poll interval is large) or very much power consumption (if that poll interval is small).

My current workaround for that is a function (similar to mqtt_sn_input()) that can be called, when there is confidence, that a next attempt to transmit a message might not fail. Would such a workaround be accepted as a fix? Better ideas, as how to fix the issue with the current design?

/**
 * @brief Check the transport for capacity on the outgoing side.
 *
 * Call this function if you have a good reason to believe that there is enough capacity
 * on the outgoing side of the link, so that pending outgoing messages could be send.
 *
 * @param client            The MQTT-SN client to check for incoming data.
 *
 * @return -EINVAL if the client is invalid or the transport contains no means to send data.
 *         Otherwise, 0 is returned.
 */
int mqtt_sn_output(struct mqtt_sn_client *client)
{
    if (!client || !client->transport || !client->transport->msg_send) {
        return -EINVAL;
    }

    k_work_reschedule(&client->process_work, K_NO_WAIT);

    return 0;
}

[TorstenRobitzki]

An other issue is, that the library ignores the result of the encode_and_send function in process_pubs() and thus ignores the fact, that no outgoing bandwidth is available. If, outgoing bandwidth becomes available while iterating over publishes for the same topic, a message that was published later can be send out earlier.

[BeckmaR]

I'm not entirely sure if I understand your point. First sentence is clear but what do you mean by

a message that was published later can be send out earlier.

I'm stumbling over the "was published later" part.

[canatella]

@BeckmaR Sorry to bring you in this conversation, but as you were really helpful with the wait for suback issue, I thought it might be interesting to get your insights on this. Thanks!

[BeckmaR]

Thanks, I was not aware of this

[BeckmaR]

I get your point. From my experience though, almost all libraries in the embedded domain act like this. I have worked with Zephyr's Bluetooth stack before as well as the LwM2M stack in the Nordic Extensions - same problem, you can send data ad infinitum, until suddenly you get an error return code because the internal buffers are full. This is what it always boils down to - you can't send more data than your transport can handle, at least on average. You can handle temporary fluctuations by adding buffers, which is the way this library is designed, and at some point they will be full.

All in all:

• You can't send more data than your transport can handle
• If you try, your buffers will run full
• If it only happens for a short time, you can increase the buffers
• If your buffers are often full, you are sending too much data

I'm not entirely sure that adding a function for a corner case that should not happen is a good idea, after all.

[BeckmaR]

Some more thoughts;

If the library finds the output link being saturated, it will wait and retry.

That is by the way not even true. If sending fails, the packet is dropped. MQTT-SN is designed to work with lossy transports. It will wait and retry if and only if a response does not arrive, but the operation would require one, independent of the reason for the missing response. Your output link being saturated is just one of a trillion possible reasons why a packet could go missing. For example, NB-IoT networks happily drop UDP packages when the radio link or cell is congested, as I know from personal experience.

So, even if you are very carefully waiting for bandwidth on your output, there is no guarantee at all that the package you just sent will ever arrive. A saturated output is rather a symptom of a problem then the problem itself, and working around it is usually just throws you into the next problem. So, usually it's better to just drop the package and let the protocol's retry mechanisms do its thing.

Retry timers and buffer sizes always have to be carefully adjusted to your specific link, which is why TCP has mechanisms to do that automatically, all the time, and people have spent considerable amounts of time to get that to working correctly.

[TorstenRobitzki]

Well, at least for the Zephyr l2cap layer, it is possible to get a callback called to monitor the available credits and thus, the available buffers on the receiving side.

We plan to transmit a large file (some kind of) over BLE (L2CAP) and USB (CDC). So, there is not really a "too much data", but an available bandwidth, that we would like to use as much as possible to keep download times short.

And yes, I know, that is not really the type of application, where MQTT shines ;-)

Anyway, thank you very much for looking into this!

[BeckmaR]

So, there is not really a "too much data", but an available bandwidth

Well, isn't it always the ratio of both? 😆

Can you explain your use case a bit more? Are you trying to do both MQTT-SN and something else over the same link, and the something else sometimes congests your link, sometimes not? And you would like to react to that dynamically?

[TorstenRobitzki]

There is a difference between not having enough bandwidth for the data that is generated and some fixed amount of data that has to be transferred. In the second case, the bandwidth is never "large enough" and it really depends on the useable bandwidth.

The device, is recording data and has to either transfer the data in real time or that recordings could be transferred later. For the first use case, we also want to cover the case, where a connection is dropped and once, that connection is reestablished, we want to transmit the buffered data (not buffered in the MQTT library, but on an external flash memory) as fast as possible. So both cases are quite similar and boil down to being able to saturate the transport layer as much as possible.