ipv6 disabled on kernel cmdline disrupts Tox = most tests fail

[emdee-is]

ipv6 disabled on kernel cmdline disrupts Tox and most tests fail

Boot a linux with ipv6.disable=1 and run the testsuite

17% tests passed, 43 tests failed out of 5

All the tests fail all at the same place.

 new_networking_ex:     failed to get a socket?! 97, Address family not supported by protocol

Note that this is on a build with USE_IPV6 "Use IPv6 in tests" OFF in the CMakeLists.txt which just disables tests - no change to the library I think.

As most of the tests fail if you boot a linux with ipv6.disable=1 and run the testsuite, booting a vm with ipv6.disable=1 should be a test.

[emdee-is]

19% tests passed, 42 tests failed out of 52

The errors in the current build all fail at:

ERROR network.c:1169   new_networking_ex: failed to get a  socket?! 97, Address family not supported    by protocol 

Does anyone have a Vagrant file for spinning up machines to test Tox with different command lines?

[emdee-is]

A year later and 2/2 more tests failed:

19% tests passed, 44 tests failed out of 54

[iphydf]

Interesting. I'll look into adding a qemu VM with Linux and ipv6 disabled. The code can deal with ipv6 not being provisioned, but maybe not with it being absent entirely.

[emdee-is]

A qemu VM with Linux and ipv6 disabled would be really nice - it's black and white for me: I reboot the same system and run the same code the tests run clean.

Thing is, it means I'm working in a greyzone: I've doing ctypes wrapping of the library into Python which will SEGV if you get it wrong. And I'm running that under a Qt GUI which is reported to SEGV on exit if it feels like it.

I'll appreciate the test coverage.

[Green-Sky]

I disabled ipv6 on my nixos using networking.enableIPv6 = false; and ran ctest.

Test project /home/green/workspace/tox/c-toxcore/build
      Start  1: TCP
      Start  2: announce
      Start  3: conference
      Start  4: conference_double_invite
      Start  5: conference_invite_merge
      Start  6: conference_peer_nick
 1/52 Test  #2: announce .........................   Passed    0.00 sec
      Start  7: conference_simple
 2/52 Test  #4: conference_double_invite .........   Passed    0.37 sec
      Start  8: conference_two
 3/52 Test  #8: conference_two ...................   Passed    0.00 sec
      Start  9: crypto
 4/52 Test  #6: conference_peer_nick .............   Passed    0.58 sec
      Start 10: dht_getnodes_api
 5/52 Test  #9: crypto ...........................   Passed    0.36 sec
      Start 11: encryptsave
 6/52 Test #11: encryptsave ......................   Passed    1.11 sec
      Start 12: file_transfer
 7/52 Test  #5: conference_invite_merge ..........   Passed    2.46 sec
      Start 13: file_saving
 8/52 Test #13: file_saving ......................   Passed    0.11 sec
      Start 14: forwarding
 9/52 Test #10: dht_getnodes_api .................   Passed    2.16 sec
      Start 15: friend_connection
10/52 Test #15: friend_connection ................   Passed    0.36 sec
      Start 16: friend_request
11/52 Test  #3: conference .......................   Passed   10.01 sec
      Start 17: friend_request_spam
12/52 Test  #7: conference_simple ................   Passed   10.86 sec
      Start 18: group_general
13/52 Test #17: friend_request_spam ..............   Passed    1.88 sec
      Start 19: group_invite
14/52 Test #14: forwarding .......................   Passed    9.38 sec
      Start 20: group_message
15/52 Test #18: group_general ....................   Passed    1.15 sec
      Start 21: group_moderation
16/52 Test #19: group_invite .....................   Passed    2.80 sec
      Start 22: group_save
17/52 Test #21: group_moderation .................   Passed    2.86 sec
      Start 23: group_state
18/52 Test #22: group_save .......................   Passed    0.50 sec
      Start 24: group_sync
19/52 Test #16: friend_request ...................   Passed   12.09 sec
      Start 25: group_tcp
20/52 Test #25: group_tcp ........................   Passed    0.00 sec
      Start 26: group_topic
21/52 Test  #1: TCP ..............................   Passed   16.03 sec
      Start 27: invalid_tcp_proxy
22/52 Test #26: group_topic ......................   Passed    1.48 sec
      Start 28: invalid_udp_proxy
23/52 Test #23: group_state ......................   Passed    1.89 sec
      Start 29: lan_discovery
24/52 Test #20: group_message ....................   Passed    7.69 sec
      Start 30: lossless_packet
25/52 Test #30: lossless_packet ..................   Passed    0.35 sec
      Start 31: lossy_packet
26/52 Test #31: lossy_packet .....................   Passed    0.36 sec
      Start 32: network
27/52 Test #32: network ..........................Subprocess aborted***Exception:   0.08 sec
      Start 33: onion
28/52 Test #24: group_sync .......................   Passed    5.59 sec
      Start 34: overflow_recvq
29/52 Test #12: file_transfer ....................   Passed   23.68 sec
      Start 35: overflow_sendq
30/52 Test #34: overflow_recvq ...................   Passed    5.40 sec
      Start 36: reconnect
31/52 Test #35: overflow_sendq ...................   Passed    0.72 sec
      Start 37: save_friend
32/52 Test #36: reconnect ........................   Passed    3.16 sec
      Start 38: save_load
33/52 Test #33: onion ............................   Passed   11.70 sec
      Start 39: send_message
34/52 Test #39: send_message .....................   Passed    0.72 sec
      Start 40: set_name
35/52 Test #37: save_friend ......................   Passed   10.29 sec
      Start 41: set_status_message
36/52 Test #38: save_load ........................   Passed    9.69 sec
      Start 42: tox_dispatch
37/52 Test #40: set_name .........................   Passed   11.29 sec
      Start 43: tox_events
38/52 Test #43: tox_events .......................   Passed    0.60 sec
      Start 44: tox_many
39/52 Test #27: invalid_tcp_proxy ................   Passed   30.08 sec
      Start 45: tox_many_tcp
40/52 Test #28: invalid_udp_proxy ................   Passed   30.15 sec
      Start 46: tox_strncasecmp
41/52 Test #46: tox_strncasecmp ..................   Passed    0.00 sec
      Start 47: typing
42/52 Test #47: typing ...........................   Passed    0.37 sec
      Start 48: version
43/52 Test #48: version ..........................   Passed    0.00 sec
      Start 49: save_compatibility
44/52 Test #49: save_compatibility ...............   Passed    0.00 sec
      Start 50: conference_av
45/52 Test #41: set_status_message ...............   Passed   11.29 sec
      Start 51: toxav_basic
46/52 Test #42: tox_dispatch .....................   Passed   10.30 sec
      Start 52: toxav_many
47/52 Test #52: toxav_many .......................   Passed    5.25 sec
48/52 Test #44: tox_many .........................   Passed   11.49 sec
49/52 Test #50: conference_av ....................   Passed   10.47 sec
50/52 Test #51: toxav_basic ......................   Passed   15.58 sec
51/52 Test #45: tox_many_tcp .....................   Passed   18.74 sec
52/52 Test #29: lan_discovery ....................***Timeout 120.01 sec

96% tests passed, 2 tests failed out of 52

Total Test time (real) = 136.77 sec

The following tests FAILED:
	 29 - lan_discovery (Timeout)
	 32 - network (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /home/green/workspace/tox/c-toxcore/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

local_discovery timing out seems to be normal. but that network fail is worrying.

running it by itself:

$ auto_tests/auto_network_test
/home/green/workspace/tox/c-toxcore/auto_tests/network_test.c:58: failed `res': Resolver failed: 0, Success
Aborted (core dumped)

[emdee-is]

thanks - my tests give more dramatic results; bear in mind i don't know C or ctest: does subprocess aborted mean a SEGv?

What does auto_tests/auto_network_test do? It does nothing for me and returns 0:

Here's my run on a 3 day old pull:

Test project /var/local/src/c-toxcore/_build
      Start  1: TCP
 1/54 Test  #1: TCP ..............................   Passed   16.16 sec
      Start  2: announce
 2/54 Test  #2: announce .........................   Passed    0.11 sec
      Start  3: conference
 3/54 Test  #3: conference .......................Subprocess aborted***Exception:   0.06 sec
      Start  4: conference_double_invite
 4/54 Test  #4: conference_double_invite .........Subprocess aborted***Exception:   0.06 sec
      Start  5: conference_invite_merge
 5/54 Test  #5: conference_invite_merge ..........Subprocess aborted***Exception:   0.04 sec
      Start  6: conference_peer_nick
 6/54 Test  #6: conference_peer_nick .............Subprocess aborted***Exception:   0.05 sec
      Start  7: conference_simple
 7/54 Test  #7: conference_simple ................Subprocess aborted***Exception:   0.05 sec
      Start  8: conference_two
 8/54 Test  #8: conference_two ...................Subprocess aborted***Exception:   0.03 sec
      Start  9: crypto
 9/54 Test  #9: crypto ...........................   Passed    0.67 sec
      Start 10: dht_getnodes_api
10/54 Test #10: dht_getnodes_api .................Subprocess aborted***Exception:   0.04 sec
      Start 11: encryptsave
11/54 Test #11: encryptsave ......................Subprocess aborted***Exception:   0.43 sec
      Start 12: file_transfer
12/54 Test #12: file_transfer ....................Subprocess aborted***Exception:   0.04 sec
      Start 13: file_saving
13/54 Test #13: file_saving ......................Subprocess aborted***Exception:   0.05 sec
      Start 14: forwarding
14/54 Test #14: forwarding .......................Subprocess aborted***Exception:   0.07 sec
      Start 15: friend_connection
15/54 Test #15: friend_connection ................Subprocess aborted***Exception:   0.04 sec
      Start 16: friend_request
16/54 Test #16: friend_request ...................Subprocess aborted***Exception:   0.06 sec
      Start 17: friend_request_spam
17/54 Test #17: friend_request_spam ..............Subprocess aborted***Exception:   0.03 sec
      Start 18: group_general
18/54 Test #18: group_general ....................Subprocess aborted***Exception:   0.05 sec
      Start 19: group_invite
19/54 Test #19: group_invite .....................Subprocess aborted***Exception:   0.04 sec
      Start 20: group_message
20/54 Test #20: group_message ....................Subprocess aborted***Exception:   0.04 sec
      Start 21: group_moderation
21/54 Test #21: group_moderation .................Subprocess aborted***Exception:   0.03 sec
      Start 22: group_save
22/54 Test #22: group_save .......................Subprocess aborted***Exception:   0.07 sec
      Start 23: group_state
23/54 Test #23: group_state ......................Subprocess aborted***Exception:   0.04 sec
      Start 24: group_sync
24/54 Test #24: group_sync .......................Subprocess aborted***Exception:   0.03 sec
      Start 25: group_tcp
25/54 Test #25: group_tcp ........................   Passed    0.00 sec
      Start 26: group_topic
26/54 Test #26: group_topic ......................Subprocess aborted***Exception:   0.03 sec
      Start 27: invalid_tcp_proxy
27/54 Test #27: invalid_tcp_proxy ................   Passed   30.09 sec
      Start 28: invalid_udp_proxy
28/54 Test #28: invalid_udp_proxy ................   Passed   30.14 sec
      Start 29: lan_discovery
29/54 Test #29: lan_discovery ....................Subprocess aborted***Exception:   0.07 sec
      Start 30: lossless_packet
30/54 Test #30: lossless_packet ..................Subprocess aborted***Exception:   0.04 sec
      Start 31: lossy_packet
31/54 Test #31: lossy_packet .....................Subprocess aborted***Exception:   0.04 sec
      Start 32: network
32/54 Test #32: network ..........................   Passed    0.04 sec
      Start 33: onion
33/54 Test #33: onion ............................   Passed   13.72 sec
      Start 34: overflow_recvq
34/54 Test #34: overflow_recvq ...................Subprocess aborted***Exception:   0.12 sec
      Start 35: overflow_sendq
35/54 Test #35: overflow_sendq ...................Subprocess aborted***Exception:   0.04 sec
      Start 36: reconnect
36/54 Test #36: reconnect ........................Subprocess aborted***Exception:   0.03 sec
      Start 37: save_friend
37/54 Test #37: save_friend ......................Subprocess aborted***Exception:   0.03 sec
      Start 38: save_load
38/54 Test #38: save_load ........................Subprocess aborted***Exception:   0.06 sec
      Start 39: send_message
39/54 Test #39: send_message .....................Subprocess aborted***Exception:   0.05 sec
      Start 40: set_name
40/54 Test #40: set_name .........................Subprocess aborted***Exception:   0.05 sec
      Start 41: set_status_message
41/54 Test #41: set_status_message ...............Subprocess aborted***Exception:   0.04 sec
      Start 42: tox_dispatch
42/54 Test #42: tox_dispatch .....................Subprocess aborted***Exception:   0.06 sec
      Start 43: tox_events
43/54 Test #43: tox_events .......................Subprocess aborted***Exception:   0.05 sec
      Start 44: tox_many
44/54 Test #44: tox_many .........................Subprocess aborted***Exception:   0.04 sec
      Start 45: tox_many_tcp
45/54 Test #45: tox_many_tcp .....................Subprocess aborted***Exception:   0.04 sec
      Start 46: tox_strncasecmp
46/54 Test #46: tox_strncasecmp ..................   Passed    0.02 sec
      Start 47: typing
47/54 Test #47: typing ...........................Subprocess aborted***Exception:   0.05 sec
      Start 48: version
48/54 Test #48: version ..........................   Passed    0.01 sec
      Start 49: save_compatibility
49/54 Test #49: save_compatibility ...............Subprocess aborted***Exception:   0.06 sec
      Start 50: bootstrap
50/54 Test #50: bootstrap ........................Subprocess aborted***Exception:   0.05 sec
      Start 51: tcp_relay
51/54 Test #51: tcp_relay ........................***Timeout 120.00 sec
      Start 52: conference_av
52/54 Test #52: conference_av ....................Subprocess aborted***Exception:   0.07 sec
      Start 53: toxav_basic
53/54 Test #53: toxav_basic ......................Subprocess aborted***Exception:   0.12 sec
      Start 54: toxav_many
54/54 Test #54: toxav_many .......................Subprocess aborted***Exception:   0.07 sec

19% tests passed, 44 tests failed out of 54

Total Test time (real) = 213.62 sec

The following tests FAILED:
	  3 - conference (Subprocess aborted)
	  4 - conference_double_invite (Subprocess aborted)
	  5 - conference_invite_merge (Subprocess aborted)
	  6 - conference_peer_nick (Subprocess aborted)
	  7 - conference_simple (Subprocess aborted)
	  8 - conference_two (Subprocess aborted)
	 10 - dht_getnodes_api (Subprocess aborted)
	 11 - encryptsave (Subprocess aborted)
	 12 - file_transfer (Subprocess aborted)
	 13 - file_saving (Subprocess aborted)
	 14 - forwarding (Subprocess aborted)
	 15 - friend_connection (Subprocess aborted)
	 16 - friend_request (Subprocess aborted)
	 17 - friend_request_spam (Subprocess aborted)
	 18 - group_general (Subprocess aborted)
	 19 - group_invite (Subprocess aborted)
	 20 - group_message (Subprocess aborted)
	 21 - group_moderation (Subprocess aborted)
	 22 - group_save (Subprocess aborted)
	 23 - group_state (Subprocess aborted)
	 24 - group_sync (Subprocess aborted)
	 26 - group_topic (Subprocess aborted)
	 29 - lan_discovery (Subprocess aborted)
	 30 - lossless_packet (Subprocess aborted)
	 31 - lossy_packet (Subprocess aborted)
	 34 - overflow_recvq (Subprocess aborted)
	 35 - overflow_sendq (Subprocess aborted)
	 36 - reconnect (Subprocess aborted)
	 37 - save_friend (Subprocess aborted)
	 38 - save_load (Subprocess aborted)
	 39 - send_message (Subprocess aborted)
	 40 - set_name (Subprocess aborted)
	 41 - set_status_message (Subprocess aborted)
	 42 - tox_dispatch (Subprocess aborted)
	 43 - tox_events (Subprocess aborted)
	 44 - tox_many (Subprocess aborted)
	 45 - tox_many_tcp (Subprocess aborted)
	 47 - typing (Subprocess aborted)
	 49 - save_compatibility (Subprocess aborted)
	 50 - bootstrap (Subprocess aborted)
	 51 - tcp_relay (Timeout)
	 52 - conference_av (Subprocess aborted)
	 53 - toxav_basic (Subprocess aborted)
	 54 - toxav_many (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /var/local/src/c-toxcore/_build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[Green-Sky]

@emdee-is 's failure:

[#0] ERROR network.c:1171	new_networking_ex:	failed to get a socket?! 97, Address family not supported by protocol

[emdee-is]

@Green-Sky I just pulled down the current repo: running behind tor:

93% tests passed, 4 tests failed out of 56

Total Test time (real) = 969.58 sec

The following tests FAILED:
3 - conference (Timeout)
51 - bootstrap (Timeout)
52 - tcp_relay (Timeout)
56 - proxy (Failed)
Errors while running CTest

I can increase the timeout_which needs to be big behind tor.
On rerunning, #3 conference Passed 51.10 sec, but the other 3 still failed.

Where does ctest get the names/IPs of the bootstrap nodes to connect to? #2467 If they are bad the tests will fail.

[emdee-is]

I think the failure in proxy test is expected if you are behind a firewall - it assumes it has open access to clearnet to run a proxy.

[emdee-is]

Signing this off as working - many thanks. I still get

The following tests FAILED:
         51 - bootstrap (Timeout)
         52 - tcp_relay (Timeout)
         56 - proxy (Failed)

but that's for other reasons: #2469

Please note that bootstrap and tcp_relay tests are the only tests that use the network. All the other ctests ate not testing real network-connected behaviour.
To confirm, turn off your wifi and run ctest,