Worker threads crash while connecting two different machines.

[kaowul]

Is it possible for this project to consider integration with mp-quic?

[max-niederman]

There's a configuration example for a simple two peer network in peer1.toml and peer2.toml.

As for integration with MP-QUIC, I'm not sure what you have in mind exactly. Centipede operates over message-oriented protocols like raw UDP rather than stream-oriented protocols like QUIC.

[kaowul]

I used two Ubuntu 22.04.3 LTS machines to build and conduct ping tests successfully. When using iperf3 for network speed testing, I encountered the following error upon starting iperf -c centipede.

./target/debug/centipede peer2.toml
2024-04-24T16:03:57.878Z INFO centipede > spawned 2 worker threads
2024-04-24T16:03:58.079Z INFO centipede_control > received initiation acknowledgement from Gb1SnzVaODF8LHKoBD+dJtS/e/9GYEhnLX0sOk8OIbA=
× worker thread 1 failed
├─▶ failed to write to UDP socket
╰─▶ Resource temporarily unavailable (os error 11)

2024-04-24T16:04:09.207Z INFO centipede > shutting down due to error
2024-04-24T16:04:09.210Z INFO centipede > received shutdown signal, waiting for workers to finish...

[kaowul]

I used two VPS instances, and direct iperf speed tests were successful. Further testing revealed that when running "iperf3 -c 10.0.10.1 -u -b 500M", an error occurred, while "iperf3 -c 10.0.10.1 -u -b 300M" resulted in a successful connection to centipede. There were no significant abnormalities observed in CPU usage.

[max-niederman]

Hmm, this is very weird. I've beenable to reproduce the same error on some local machines. It doesn't seem to happen between two netnses on the same machine, which is how I've been testing so far.

I'll keep looking into it. I suspect this could have something to do with device queues filling up.

[max-niederman]

Should be fixed in the latest commit. @kaowul would you mind testing this to verify?

[kaowul]

iperf3 -c 10.0.10.1
Connecting to host 10.0.10.1, port 5201
[ 5] local 10.0.10.2 port 52038 connected to 10.0.10.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 42.8 MBytes 359 Mbits/sec 42 177 KBytes
[ 5] 1.00-2.00 sec 41.6 MBytes 349 Mbits/sec 10 192 KBytes
[ 5] 2.00-3.00 sec 41.4 MBytes 348 Mbits/sec 14 205 KBytes

This was my recent test, which confirms that the link speed between my two VPNs is around 350Mbits/sec. However, if I conduct a test using iperf3 -c 10.0.10.1 -u -b 400M, after a few seconds, the following error occurs:

./target/debug/centipede /root/peer2.toml
2024-04-25T06:58:43.580Z INFO centipede > spawned 2 worker threads
2024-04-25T06:58:43.742Z INFO centipede_control > received initiation acknowledgement from Gb1SnzVaODF8LHKoBD+dJtS/e/9GYEhnLX0sOk8OIbA=
× worker thread 0 failed
├─▶ failed to poll for events
╰─▶ Resource temporarily unavailable (os error 11)

2024-04-25T06:59:49.876Z INFO centipede > shutting down due to error
2024-04-25T06:59:49.885Z INFO centipede > received shutdown signal, waiting for workers to finish...

[kaowul]

This time when I ran iperf3 -c 10.0.10.2 -u -b 600m, an error occurred. It seems that this error occurs when the data sent exceeds the receiving capacity of the destination.