Feature request: traffic based healthcheck

[dfadev]

What's the feature 🧐

Make the healthcheck optionally monitor packet counts to determine if a connection is still healthy.

Extra information and references

On a saturated connection, the healthcheck can fail but the interface is still passing packets.

gluetun8  | 2023-09-10T06:46:16-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T07:22:16-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T08:47:20-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:14:43-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:42:06-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:48:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:01:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:08:39-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:16:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:30:09-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:39:52-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:49:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:00:05-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:05:05-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:12:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:18:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:25:06-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:30:48-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:42:29-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:43:34-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:03:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:09:09-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:50:17-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:02:53-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:12:44-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:17:49-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:22:30-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:35:59-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:47:51-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:57:49-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:09:35-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:17:11-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:21:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:32:24-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:37:58-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:47:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:53:46-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:59:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:08:55-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:13:47-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:21:47-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:22:52-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:38:54-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:47:37-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:48:42-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:02:17-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:07:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:12:52-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:14:41-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:18:59-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:20:05-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:21:25-04:00 INFO [healthcheck] program has been unhealthy for 1m10s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:26:14-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:32:32-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:39:38-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:47:43-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:55:35-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:01:37-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:07:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:08:08-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:14:04-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:21:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:26:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:27:18-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:39:10-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:48:48-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:59:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T19:05:01-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)

[qdm12]

It's not because packets get sent through the interface that the connection is working right? Or are you talking about receiving packets? What's your use case to monitor packets versus the current tcp dialing to 1.1.1.1:443?

[dfadev]

It's for torrenting over links with high latency and lots of bandwidth management, like cellular. When there are a high number of peers the healthcheck begins to fail even though traffic is still being exchanged.

For a quick POC, I added this patch:

diff --git a/internal/healthcheck/health.go b/internal/healthcheck/health.go
index 2cd61426..8aa1b3ac 100644
--- a/internal/healthcheck/health.go
+++ b/internal/healthcheck/health.go
@@ -5,6 +5,7 @@ import (
        "errors"
        "fmt"
        "net"
+       "os"
        "time"
 )
 
@@ -63,6 +64,10 @@ func (s *Server) runHealthcheckLoop(ctx context.Context, done chan<- struct{}) {
 }
 
 func (s *Server) healthCheck(ctx context.Context) (err error) {
+       if os.Getenv("DISABLE_HEALTH_TCP") == "true" {
+               return nil
+       }
+
        // TODO use mullvad API if current provider is Mullvad
 
        address, err := makeAddressToDial(s.config.TargetAddress)

And use this for the healthcheck in docker-compose:

  test: ["CMD-SHELL", "packet_count=$$(cat /sys/class/net/tun0/statistics/rx_packets); sleep 2; new_packet_count=$$(cat /sys/class/net/tun0/statistics/rx_packets); if [ ! -z $$new_packet_count ] && [ $$new_packet_count -le $$packet_count ]; then exit 1; fi"]

With DISABLE_HEALTH_TCP set to true, and the custom healthcheck command set, I can push the connection to about 7500 peers without the VPN restarting at all. Without, the TCP check starts to recycle the VPN when the peer count reaches about 2500.

Restarting the VPN is not ideal because it takes time to handshake that many connections and unnecessary because some peer connections are still able to exchange traffic.

[qdm12]

This will eventually be added thanks to #2411 which parses iptables results, and gets packets/bytes sent for each chain rule. We can use that to monitor traffic.