Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Proxmox not displaying ZFS disk info #494

Open
truncsphere opened this issue Jan 25, 2025 · 7 comments
Open

[Bug]: Proxmox not displaying ZFS disk info #494

truncsphere opened this issue Jan 25, 2025 · 7 comments
Labels
bug Something isn't working

Comments

@truncsphere
Copy link

truncsphere commented Jan 25, 2025

Description

Added one of my Proxmox servers and have been trying to get it to populate disk usage stats. I know currently I/O is not supported.
I've added the mountpoints of the disk which are /rpool/data (root partition is on this disk as well) and /pm2-2tb-zfs . However they both show 0.00 MB used.

Image

Expected Behavior

Show disk usage for each disk added. Added debug log below, however it does not give me info about Disk Partitions

Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 INFO Detected network interface name=tap104i0 sent=5286209484 recv=683013816
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG Getting stats
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG Temperature sensors="[{\"sensorKey\":\"nvme_composite\",\"temperature\":34.85,\"sensorHigh\":81.85,\"sensorCritical>
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG sysinfo data="{Hostname:proxmox2 KernelVersion:6.8.12-4-pve Cores:6 Threads:12 CpuModel:12th Gen Intel(R) Core(TM) >
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG System stats data="{Stats:{Cpu:0 MaxCpu:0 Mem:62.57 MemUsed:28.32 MemPct:45.26 MemBuffCache:0.63 MemZfsArc:5.05 Swa>
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG Error getting docker stats err="Get \"http://localhost/containers/json\": dial unix /var/run/docker.sock: connect: >
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG Extra filesystems data="map[data:0xc0002280e0 pm2-2tb-zfs:0xc000228150]"
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 DEBUG Stats data="{Stats:{Cpu:0 MaxCpu:0 Mem:62.57 MemUsed:28.32 MemPct:45.26 MemBuffCache:0.63 MemZfsArc:5.05 Swap:0 Swa>
Jan 25 14:35:05 proxmox2 beszel-agent[1852242]: 2025/01/25 14:35:05 INFO Starting SSH server address=:45876

Steps to Reproduce

  1. Use zfs list to get mountpoints for both disks (I've also tried adding the Disk and Partition from lsblk with no success)
  2. Added both mountpoints to beszel-agent.service
[Unit]
Description=Beszel Agent Service
After=network-online.target
Wants=network-online.target

[Service]
Environment="PORT=45876"
Environment="KEY=******"
Environment="EXTRA_FILESYSTEMS=/rpool/data,/pm2-2tb-zfs"
Restart=always
RestartSec=5
ExecStart=/root/beszel-agent
Environment="LOG_LEVEL=debug"

[Install]
WantedBy=multi-user.target

OS / Architecture

Proxmox

Beszel version

0.9.1

Installation method

Docker

@truncsphere truncsphere added the bug Something isn't working label Jan 25, 2025
@henrygd
Copy link
Owner

henrygd commented Jan 25, 2025

Can you please restart the agent and verify that it's not logging the partitions?

It should happen before the Detected network interface line, so I think your logs are truncated.

Try just executing it directly with temp env vars like this:

PORT=44555 KEY="..." LOG_LEVEL=debug /root/beszel-agent

If it still doesn't log partitions and disk I/O then something very strange is happening.

@truncsphere
Copy link
Author

truncsphere commented Jan 25, 2025

Here is output for Disk partitions

2025/01/25 16:19:58 DEBUG Disk partitions="[{\"device\":\"rpool/ROOT/pve-1\",\"mountpoint\":\"/\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"rpool\",\"mountpoint\":\"/rpool\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"rpool/var-lib-vz\",\"mountpoint\":\"/var/lib/vz\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"pm2-2tb-zfs\",\"mountpoint\":\"/pm2-2tb-zfs\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"rpool/data\",\"mountpoint\":\"/rpool/data\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"rpool/ROOT\",\"mountpoint\":\"/rpool/ROOT\",\"fstype\":\"zfs\",\"opts\":[\"rw\",\"relatime\"]}]"
2025/01/25 16:19:58 DEBUG Disk I/O diskstats="map[nvme0n1:{\"readCount\":3808928,\"mergedReadCount\":18913,\"writeCount\":98225238,\"mergedWriteCount\":11160,\"readBytes\":169580981760,\"writeBytes\":1689654737408,\"readTime\":2829370,\"writeTime\":35756889,\"iopsInProgress\":0,\"ioTime\":26729455,\"weightedIO\":47661671,\"name\":\"nvme0n1\",\"serialNumber\":\"Samsung_SSD_980_PRO_1TB_S5P2NS0X210722Y_1\",\"label\":\"\"} nvme0n1p1:{\"readCount\":140,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":4481024,\"writeBytes\":0,\"readTime\":11,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":10,\"weightedIO\":11,\"name\":\"nvme0n1p1\",\"serialNumber\":\"Samsung_SSD_980_PRO_1TB_S5P2NS0X210722Y_1\",\"label\":\"\"} nvme0n1p2:{\"readCount\":6838,\"mergedReadCount\":18913,\"writeCount\":3189,\"mergedWriteCount\":11160,\"readBytes\":41286144,\"writeBytes\":1480509952,\"readTime\":594,\"writeTime\":21960,\"iopsInProgress\":0,\"ioTime\":880,\"weightedIO\":22555,\"name\":\"nvme0n1p2\",\"serialNumber\":\"Samsung_SSD_980_PRO_1TB_S5P2NS0X210722Y_1\",\"label\":\"\"} nvme0n1p3:{\"readCount\":3801401,\"mergedReadCount\":0,\"writeCount\":98222049,\"mergedWriteCount\":0,\"readBytes\":169520276480,\"writeBytes\":1688174227456,\"readTime\":2828689,\"writeTime\":35734928,\"iopsInProgress\":0,\"ioTime\":50930775,\"weightedIO\":38667087,\"name\":\"nvme0n1p3\",\"serialNumber\":\"Samsung_SSD_980_PRO_1TB_S5P2NS0X210722Y_1\",\"label\":\"rpool\"} sdg:{\"readCount\":23074038,\"mergedReadCount\":22010,\"writeCount\":84660574,\"mergedWriteCount\":681085,\"readBytes\":1134983081984,\"writeBytes\":2812728999936,\"readTime\":17458896,\"writeTime\":43328505,\"iopsInProgress\":0,\"ioTime\":26011768,\"weightedIO\":69961123,\"name\":\"sdg\",\"serialNumber\":\"Samsung_SSD_870_EVO_2TB_S6PNNS0W500286A\",\"label\":\"\"} sdg1:{\"readCount\":23073766,\"mergedReadCount\":22010,\"writeCount\":84660574,\"mergedWriteCount\":681085,\"readBytes\":1134968033280,\"writeBytes\":2812728999936,\"readTime\":17458721,\"writeTime\":43328505,\"iopsInProgress\":0,\"ioTime\":61372248,\"weightedIO\":60787227,\"name\":\"sdg1\",\"serialNumber\":\"Samsung_SSD_870_EVO_2TB_S6PNNS0W500286A\",\"label\":\"pm2-2tb-zfs\"} sdg9:{\"readCount\":128,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":6995968,\"writeBytes\":0,\"readTime\":89,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":99,\"weightedIO\":89,\"name\":\"sdg9\",\"serialNumber\":\"Samsung_SSD_870_EVO_2TB_S6PNNS0W500286A\",\"label\":\"\"} zd0:{\"readCount\":849790,\"mergedReadCount\":0,\"writeCount\":7789238,\"mergedWriteCount\":0,\"readBytes\":247140929024,\"writeBytes\":134499459584,\"readTime\":319298,\"writeTime\":2410093,\"iopsInProgress\":0,\"ioTime\":17872290,\"weightedIO\":2729391,\"name\":\"zd0\",\"serialNumber\":\"\",\"label\":\"\"} zd0p1:{\"readCount\":30,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1069056,\"writeBytes\":0,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd0p1\",\"serialNumber\":\"\",\"label\":\"\"} zd0p2:{\"readCount\":28,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1064960,\"writeBytes\":0,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd0p2\",\"serialNumber\":\"\",\"label\":\"\"} zd0p3:{\"readCount\":8,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":32768,\"writeBytes\":0,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd0p3\",\"serialNumber\":\"\",\"label\":\"\"} zd16:{\"readCount\":49,\"mergedReadCount\":0,\"writeCount\":6,\"mergedWriteCount\":0,\"readBytes\":708608,\"writeBytes\":24576,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd16\",\"serialNumber\":\"\",\"label\":\"\"} zd32:{\"readCount\":18,\"mergedReadCount\":0,\"writeCount\":63,\"mergedWriteCount\":0,\"readBytes\":327680,\"writeBytes\":258048,\"readTime\":2,\"writeTime\":37,\"iopsInProgress\":0,\"ioTime\":3017041422,\"weightedIO\":39,\"name\":\"zd32\",\"serialNumber\":\"\",\"label\":\"\"} zd48:{\"readCount\":1265366,\"mergedReadCount\":0,\"writeCount\":3704567,\"mergedWriteCount\":0,\"readBytes\":168471405056,\"writeBytes\":56296189952,\"readTime\":1386322,\"writeTime\":547871,\"iopsInProgress\":0,\"ioTime\":31467014,\"weightedIO\":1934193,\"name\":\"zd48\",\"serialNumber\":\"\",\"label\":\"\"} zd48p1:{\"readCount\":11,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":45056,\"writeBytes\":0,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"zd48p1\",\"serialNumber\":\"\",\"label\":\"\"} zd48p2:{\"readCount\":23,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":299008,\"writeBytes\":0,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"zd48p2\",\"serialNumber\":\"\",\"label\":\"boot-pool\"} zd64:{\"readCount\":4439335,\"mergedReadCount\":0,\"writeCount\":2042581,\"mergedWriteCount\":0,\"readBytes\":2244445471744,\"writeBytes\":530652394496,\"readTime\":2954108,\"writeTime\":211172,\"iopsInProgress\":0,\"ioTime\":1356016,\"weightedIO\":3174550,\"name\":\"zd64\",\"serialNumber\":\"\",\"label\":\"\"} zd64p1:{\"readCount\":24,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":302080,\"writeBytes\":0,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd64p1\",\"serialNumber\":\"\",\"label\":\"\"} zd64p2:{\"readCount\":29,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1069056,\"writeBytes\":0,\"readTime\":3,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":3,\"weightedIO\":3,\"name\":\"zd64p2\",\"serialNumber\":\"\",\"label\":\"New_Volume\"} zd80:{\"readCount\":10423463,\"mergedReadCount\":0,\"writeCount\":16397370,\"mergedWriteCount\":0,\"readBytes\":1224257753600,\"writeBytes\":304074342400,\"readTime\":2870828,\"writeTime\":835589,\"iopsInProgress\":0,\"ioTime\":3094388,\"weightedIO\":3715042,\"name\":\"zd80\",\"serialNumber\":\"\",\"label\":\"\"} zd80p1:{\"readCount\":29,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1069056,\"writeBytes\":0,\"readTime\":4,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":3,\"weightedIO\":4,\"name\":\"zd80p1\",\"serialNumber\":\"\",\"label\":\"System_Reserved\"} zd80p2:{\"readCount\":29,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1083392,\"writeBytes\":0,\"readTime\":5,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":5,\"weightedIO\":5,\"name\":\"zd80p2\",\"serialNumber\":\"\",\"label\":\"\"} zd80p3:{\"readCount\":29,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1069056,\"writeBytes\":0,\"readTime\":6,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":4,\"weightedIO\":6,\"name\":\"zd80p3\",\"serialNumber\":\"\",\"label\":\"\"} zd80p4:{\"readCount\":29,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":1073152,\"writeBytes\":0,\"readTime\":1,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":1,\"weightedIO\":1,\"name\":\"zd80p4\",\"serialNumber\":\"\",\"label\":\"\"}]"
2025/01/25 16:19:58 INFO Root disk mountpoint=/ io=zd64

@henrygd
Copy link
Owner

henrygd commented Jan 26, 2025

Yeah I'm not sure why that's not working.

Does Error getting disk stats ever get logged while the program is running? (You wouldn't need debug on to see this.)

I'd be curious to see what the output of this is. I can give you a binary if you don't have Go installed or haven't used it before.

package main

import (
	"fmt"

	"github.com/shirou/gopsutil/v4/disk"
)

func main() {
	rpool, err := disk.Usage("/rpool/data")
	if err != nil {
		panic(err)
	}
	fmt.Printf("All: %v, Used: %v, Free: %v, UsedPercent: %f%%\n", rpool.Total, rpool.Used, rpool.Free, rpool.UsedPercent)

	pm2, err := disk.Usage("/pm2-2tb-zfs")
	if err != nil {
		panic(err)
	}
	fmt.Printf("All: %v, Used: %v, Free: %v, UsedPercent: %f%%\n", pm2.Total, pm2.Used, pm2.Free, pm2.UsedPercent)

	vz, err := disk.Usage("/var/lib/vz")
	if err != nil {
		panic(err)
	}
	fmt.Printf("All: %v, Used: %v, Free: %v, UsedPercent: %f%%\n", vz.Total, vz.Used, vz.Free, vz.UsedPercent)
}

@henrygd
Copy link
Owner

henrygd commented Jan 26, 2025

Also check if you get any Invalid disk I/O. Resetting. warnings please.

@truncsphere
Copy link
Author

The only thing I see when not running debug is the below.
Also Proxmox does not have GO installed. I'd be OK with running a binary for more detailed analysis.

root@proxmox2:~# systemctl status beszel-agent.service
● beszel-agent.service - Beszel Agent Service
     Loaded: loaded (/etc/systemd/system/beszel-agent.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-01-26 13:39:54 MST; 3s ago
   Main PID: 405104 (beszel-agent)
      Tasks: 6 (limit: 76799)
     Memory: 5.9M
        CPU: 124ms
     CGroup: /system.slice/beszel-agent.service
             └─405104 /root/beszel-agent

Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=enp4s0f0 sent=300 recv=30425
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=enp4s0f1 sent=30949799978 recv=115873521054
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=bond0 sent=30949800278 recv=115873551479
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=vmbr0 sent=11605073895 recv=913731592
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=tap109i0 sent=103331551086 recv=73247906637
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=fwpr109p0 sent=103329949595 recv=73247906637
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=fwln109i0 sent=73247906637 recv=103329949595
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=tap104i0 sent=256639586 recv=32559413
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Detected network interface name=tap107i0 sent=179352895031 recv=116381942012
Jan 26 13:39:54 proxmox2 beszel-agent[405104]: 2025/01/26 13:39:54 INFO Starting SSH server address=:45876

@henrygd
Copy link
Owner

henrygd commented Jan 26, 2025

Thanks, please run this and let me know what it says.

# download binary
wget https://static.beszel.dev/issues/494/check-disk-usage_amd64

# make sure it's executable
chmod +x ./check-disk-usage_amd64

# check usage of specific directories
./check-disk-usage_amd64 /rpool/data /pm2-2tb-zfs /var/lib/vz

This is the code for reference:

package main

import (
	"fmt"
	"os"

	"github.com/shirou/gopsutil/v4/disk"
)

func main() {
	if len(os.Args) < 2 {
		fmt.Printf("Usage: %s <directory1> <directory2> ...\n", os.Args[0])
		return
	}

	dirs := os.Args[1:]

	for _, dir := range dirs {
		usg, err := disk.Usage(dir)
		fmt.Println("Checking disk usage for", dir)
		if err != nil {
			fmt.Println(err)
		} else {
			fmt.Printf("All: %v, Used: %v, Free: %v, UsedPercent: %f%%\n", usg.Total, usg.Used, usg.Free, usg.UsedPercent)
		}
	}
}

@truncsphere
Copy link
Author

Output below

Checking disk usage for /rpool/data
All: 930356789248, Used: 131072, Free: 930356658176, UsedPercent: 0.000014%
Checking disk usage for /pm2-2tb-zfs
All: 1880819630080, Used: 131072, Free: 1880819499008, UsedPercent: 0.000007%
Checking disk usage for /var/lib/vz
All: 937562734592, Used: 7206076416, Free: 930356658176, UsedPercent: 0.768597%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants