Websocket has no route out if connection is started before interface is up or is reset

[LilMouse]

Hello,

INFO

I am using libwebsockets version 4.3.3 and the minimal ws client example.
It is cross compiled for a linux (buildroot) system with -DLWS_WITH_NETLINK=1

The issue

When the ppp0 (GSM) connection comes up (by running ifup ppp0), after the connection is started it has no route.

I understand why the message "has no route out" is given before the interface is up, but I would assume it can connect when the ppp0 interface is up and there is a route.
I see rops_handle_POLLIN_netlink: DELADDR when the interface is going up but I would have expected an ADDADDR.

The only thing I changed in the minimal example is the IP address & path because this SIM can only connect to a specific IP range.

If the ppp0 connection is up before starting the minimal example it connects without an issue.

LOG

[2024/08/05 09:56:59:7186] U: LWS minimal ws client
[2024/08/05 09:56:59:7213] N: lws_create_context: LWS: 4.3.3-2021.11.1.1, NET CLI SRV H1 H2 WS ConMon IPV6-on
[2024/08/05 09:56:59:7236] N: __lws_lc_tag:  ++ [wsi|0|pipe] (1)
[2024/08/05 09:56:59:7256] N: __lws_lc_tag:  ++ [vh|0|netlink] (1)
[2024/08/05 09:56:59:7571] N: __lws_lc_tag:  ++ [vh|1|default||-1] (2)
[2024/08/05 09:56:59:7620] N: __lws_lc_tag:  ++ [wsicli|0|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:56:59:7647] N: __lws_lc_untag:  -- [wsicli|0|WS/h1/default/192.168.255.254] (0) 2.751ms
[2024/08/05 09:57:00:7775] N: __lws_lc_tag:  ++ [wsicli|1|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:00:7793] N: [wsicli|1|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:00:7812] N: __lws_lc_untag:  -- [wsicli|1|WS/h1/default/192.168.255.254] (0) 3.644ms
[2024/08/05 09:57:03:1549] N: __lws_lc_tag:  ++ [wsicli|2|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:03:1552] N: [wsicli|2|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:03:1555] N: __lws_lc_untag:  -- [wsicli|2|WS/h1/default/192.168.255.254] (0) 546μs
[2024/08/05 09:57:06:4964] N: __lws_lc_tag:  ++ [wsicli|3|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:06:4967] N: [wsicli|3|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:06:4969] N: __lws_lc_untag:  -- [wsicli|3|WS/h1/default/192.168.255.254] (0) 465μs

// executed 'ifup ppp0'

[2024/08/05 09:57:09:9942] N: rops_handle_POLLIN_netlink: DELADDR
[2024/08/05 09:57:11:1215] N: __lws_lc_tag:  ++ [wsicli|4|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:11:1218] N: [wsicli|4|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:11:1220] N: __lws_lc_untag:  -- [wsicli|4|WS/h1/default/192.168.255.254] (0) 463μs
[2024/08/05 09:57:17:0533] N: __lws_lc_tag:  ++ [wsicli|5|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:17:0536] N: [wsicli|5|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:17:0538] N: __lws_lc_untag:  -- [wsicli|5|WS/h1/default/192.168.255.254] (0) 522μs
[2024/08/05 09:57:21:4879] N: rops_handle_POLLIN_netlink: DELADDR
[2024/08/05 09:57:23:7755] N: __lws_lc_tag:  ++ [wsicli|6|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:23:7757] N: [wsicli|6|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:23:7759] N: __lws_lc_untag:  -- [wsicli|6|WS/h1/default/192.168.255.254] (0) 453μs

// executed 'ifdown ppp0'
[2024/08/05 09:57:29:5560] N: rops_handle_POLLIN_netlink: DELADDR
[2024/08/05 09:57:31:3284] N: __lws_lc_tag:  ++ [wsicli|7|WS/h1/default/192.168.255.254] (1)
[2024/08/05 09:57:31:3287] N: [wsicli|7|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/05 09:57:31:3289] N: __lws_lc_untag:  -- [wsicli|7|WS/h1/default/192.168.255.254] (0) 454μs
^C[2024/08/05 09:57:32:2112] N: __lws_lc_untag:  -- [wsi|0|pipe] (0) 32.487s
[2024/08/05 09:57:32:2123] N: __lws_lc_untag:  -- [vh|0|netlink] (1) 32.486s
[2024/08/05 09:57:32:2145] N: __lws_lc_untag:  -- [vh|1|default||-1] (0) 32.457s

But if I check the default route is there.

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         *               0.0.0.0         U     0      0        0 ppp0
10.64.64.64     *               255.255.255.255 UH    0      0        0 ppp0

it seems to be similar to issue #2311 but the links there to the setup/examples are broken so I can't see the fix.

EDIT

I also tested it with -DLWS_WITH_NETLINK=0, that seems to work. Could this be a netlink bug or is this working as intended?

[lws-team]

I would definitely start by trying lws main branch

[LilMouse]

Thanks for the suggestion.
It took a bit longer to get the main branch into buildroot so it took a bit longer than expected.

But it has the same result:

[2024/08/06 14:00:41:0237] N: lws_create_context: LWS: 4.3.99-2021.11.1.1, NET CLI SRV H1 H2 WS SS-JSON-POL ConMon IPV6-off
[2024/08/06 14:00:41:0247] N: __lws_lc_tag:  ++ [wsi|0|pipe] (1)
[2024/08/06 14:00:41:0256] N: __lws_lc_tag:  ++ [vh|0|netlink] (1)
[2024/08/06 14:00:41:0263] N: __lws_lc_tag:  ++ [vh|1|default||-1] (2)

/// ifup ppp0

[2024/08/06 14:01:18:3695] N: __lws_lc_tag:  ++ [wsicli|1|WS/h1/default/192.168.255.254] (1)
[2024/08/06 14:01:18:3700] N: [wsicli|1|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out
[2024/08/06 14:01:18:3717] N: __lws_lc_untag:  -- [wsicli|1|WS/h1/default/192.168.255.254] (0) 2.085ms
[2024/08/06 14:01:18:3728] W: lws_smd_message_pending: timing out queued message 0x76000ad8
[2024/08/06 14:01:18:3738] N: rops_handle_POLLIN_netlink: DELADDR
[2024/08/06 14:01:35:7694] N: __lws_lc_tag:  ++ [wsicli|2|WS/h1/default/192.168.255.254] (1)
[2024/08/06 14:01:35:7697] N: [wsicli|2|WS/h1/default/192.168.255.254]: lws_sort_dns: 192.168.255.254 has no route out

PS: the "2021.11.1.1" in the version is probably from the buildroot I am using. (which is 2021.11.1.1)
But I based the libwebsocket package on the current main branch

[lws-team]

The version number not changing is a bit suspicious... this is created when running cmake on a git checkout, it implies either redoing cmake was not triggered by changing the sources wholesale, which is very doubtful, or that you unpacked main on top of the original checkout without any git operations, it's possible but unusual.

Since nothing changed it sounds more likely that your app is linking against the original built library unchanged. You could check it by adding a "sanity" log in the create context code itself to prove if you are running main or not.

Netlink support works for most cases but it is heavily dependent on what the userland part is doing with the network device and its routes, it's basically reporting primitive operations triggered by NetworkManager or conman or whatever. GSM added or not is not something I am able to test.