Device in Heartbeat Only mode but VPN connected

[kb2ma]

Like the title states, a user is able to confirm that a VPN connection exists, but the balena dashboard reports the device is in Heartbeat Only mode. So the cloud thinks there is not a VPN connection.

In the log below, we see what looks like a race condition between VPN instances. The numbers in parentheses refer to the log lines below.

• (1) the device is connected on 2022-12-04
• (3,4) the device connects to VPN instance 127577
• (2,5) the device connects and disconnects to VPN instance 127568
• (6) VPN instance 127568 recognizes the device as connected

Logs / Screenshots

(1) 2022-12-04 16:09:43 [854936.832432] vpn[718]: info: [vpn-127564.7] connection established with client_id=1506309 uuid=6ea096ec955ef0fff8eeedbe64316564
(2) 2022-12-05 05:18:17 [902248.477595] vpn[548]: debug: [vpn-127568] successfully updated state for device: uuid=6ea096ec955ef0fff8eeedbe64316564 connected=true
(3) 2022-12-05 05:18:32 [ 1150.154516] vpn[714]: info: [vpn-127577.2] connection established with client_id=3134 uuid=6ea096ec955ef0fff8eeedbe64316564
(4) 2022-12-05 05:18:33 [ 1150.471219] vpn[548]: debug: [vpn-127577] successfully updated state for device: uuid=6ea096ec955ef0fff8eeedbe64316564 connected=true
(5) 2022-12-05 05:18:33 [902264.764050] vpn[548]: debug: [vpn-127568] successfully updated state for device: uuid=6ea096ec955ef0fff8eeedbe64316564 connected=false
(6) 2022-12-05 05:18:47 [902278.515212] vpn[716]: info: [vpn-127568.4] connection established with client_id=1613412 uuid=6ea096ec955ef0fff8eeedbe64316564```

[majorz]

On the client side the device got back to online state by itself after inactivity timeout occurred, which confirms the theory:

Dec 08 20:30:02 xv190329 openvpn[1162]: Thu Dec  8 20:30:02 2022 [vpn.balena-cloud.com] Inactivity timeout (--ping-restart), restarting

On the client side I think this could be related to the very common TUNSETIFF error that restarts the VPN client, so it is being connected twice to two different instances in short periods of time - balena-os/meta-balena#1779. Very likely if we solve that client issue we will lower the number of those cases by a lot.

[jellyfish-bot]

[lmbarros] This has attached https://jel.ly.fish/c95ef1df-db68-4f0c-84bb-0c99d5d12068