Metrics Reference

This reference lists metric names, units, labels, sources, and measurement semantics. See Kernel-Wide Insight for capabilities and troubleshooting scenarios.

How to Read This Reference

Common metrics are grouped by CPU, memory, network, I/O, system events, and GPU for querying and troubleshooting. Actual output depends on kernel interfaces, device drivers, enabled collectors, and filters. The node’s /metrics output is authoritative for the complete set. See Node Configuration for enablement.

• Names: tables omit the huatuo_bamai_ prefix. For example, cpu_util_total is exposed as huatuo_bamai_cpu_util_total.
• Labels: all metrics include host and region. Container metrics also include container_host, container_hostnamespace, container_level, container_name, and container_type. Tables list only additional labels; — means none.
• Counts and units: counts measure events or objects, not latency. Use rate() or increase() for cumulative Counters. Some cumulative kernel statistics are exported as Gauges; interpret their field semantics rather than inferring from type or name alone.
• Examples: values illustrate output format, not diagnostic thresholds. A missing metric does not mean zero.

# TYPE huatuo_bamai_runqlat_latency counter
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
# TYPE huatuo_bamai_runqlat_container_latency counter
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226

This example shows host and container scheduling-latency bucket counts. The zone ranges are listed below.

Build Information

huatuo_bamai_build_info and huatuo_apiserver_build_info are Gauges with a constant value of 1, describing the running bamai and apiserver builds respectively. Each is exposed by its application’s /metrics endpoint and uses the same build information as that application’s /version endpoint. They carry the existing host and region labels (empty for apiserver) and these build labels:

When no release version is supplied, version uses (devel). git_commit uses the injected identifier first, then the VCS revision from Go build information, and falls back to (devel) when neither is available. Build labels apply only to this metric; business metric labels remain unchanged. After scraping, join build information with business metrics using the same job and instance. Cross-cluster queries must also include the labels that distinguish targets between clusters.

CPU

Scheduling Latency

Scheduling latency is the wait from becoming runnable to actually receiving CPU time. These eBPF metrics are cumulative Counters bucketed by zone: 0 = [0, 10) ms, 1 = [10, 20) ms, 2 = [20, 50) ms, and 3 = >=50 ms.

Softirq Latency

Measures the wait from softirq raise to processing on each CPU. Currently only NET_RX and NET_TX are emitted. This is a cumulative Counter with zone buckets: 0 = [0, 10) μs, 1 = [10, 100) μs, 2 = [100, 1000) μs, and 3 = >=1 ms.

Utilization

Host utilization is normalized by online CPU cores; container utilization by available CPU capacity. Values range from 0 to 100%.

Available Cores

Container CPU capacity is the smaller of quota-derived cores and the effective CPU set size. Without a quota, it uses the effective CPU set size; if unavailable, it falls back to online host cores.

Throttling and Contention

Throttling statistics come from cgroup CPU interfaces and include only fields provided by the kernel. The wait ratio also requires kernel wait_sum and kernel.sched_schedstats=1.

The interval wait ratio is wait-time delta / (wait-time delta + CPU-usage delta) × 100%. It measures waiting as a share of schedulable time, not the share spent throttled.

References:

• CFS bandwidth statistics
• cgroup v2 CPU interface

Burst Usage

Counts periods and time using burst CPU capacity. Emitted only when the kernel provides the corresponding fields.

Load

Hosts report 1-, 5-, and 15-minute load averages. Containers report current runnable and uninterruptible task counts, which use a different time basis.

Memory

Reclaim

Memory reclaim and compaction can block tasks. These metrics record host cumulative durations and container direct-reclaim event counts.

memory_others_container_directstall_time, memory_others_container_asyncreclaim_time, and memory_others_container_local_direct_reclaim_time depend on DiDi custom-kernel interfaces memory.directstall_stat, memory.asynreclaim_stat, and memory.local_direct_reclaim_time.

Mainline and common distribution kernels without these interfaces do not emit the metrics; no additional module is needed. Use memory_reclaim_container_directstall for container direct-reclaim counts on standard kernels.

Memory State

Container fields come from memory.stat; host fields from /proc/vmstat. Both are subject to MetricCollector.Vmstat filters. Field availability and meanings depend on kernel and cgroup versions.

Container memory

Host memory

The names below include memory_vmstat_. Underlying values are exported unchanged rather than uniformly converted to bytes. pgpgin and pgpgout use the kernel’s KiB units; see the Linux vmstat implementation.

Page state and LRU

Dirty pages and writeback

Page faults and swapping

Reclaim and scanning

Transparent Huge Pages (THP)

NUMA

References:

• cgroup v2
• cgroup v1 memory controller
• Transparent Huge Pages

Memory Events

These fields come from memory.events and count reclaim, memory-limit, and OOM events. Mainline kernels provide it in cgroup v2; DiDi custom kernels may also provide it in cgroup v1. Nothing is emitted if the interface is absent. Fields count different conditions and must not be added as independent events.

Free Memory Blocks

Shows free block counts by NUMA node, memory zone, and order. order=n means each block contains 2^n contiguous pages. The value counts blocks, not pages.

Container reclaim and compaction stall times

These cumulative Gauge values measure task time in global direct reclaim and compaction. Host totals include container work. Missing or ambiguous container records are omitted; container collection failures preserve host output. Loading requires the container probes; a load or attachment failure disables this collector. Container recreation starts a new container total without resetting the host total. BPF reload resets both.

Host-wide memcg reclaim count

This Gauge counts host-wide memcg reclaim events, excluding kswapd; it is not a global reclaim count or duration. Host totals include container activity and must not be added to container counts. BPF reload resets the total.

Network

TCP Memory

Reports TCP memory usage and its proportion of the limit.

Connection Tracking

Host nf_conntrack table usage and capacity come from /proc/sys/net/netfilter/nf_conntrack_count and /proc/sys/net/netfilter/nf_conntrack_max. The repository configuration disables net_conntrack through BlackList by default.

ARP Neighbors

Counts ARP neighbor entries in host and container network namespaces.

Transmit Queues

Qdisc metrics describe backlog, transmitted traffic, drops, and requeues, not queueing latency directly. Only root Qdiscs of matching devices are collected; noqueue is skipped. The repository configuration disables netdev_qdisc through BlackList by default.

Hardware Drops

Counts receive-side hardware drops for supported NIC drivers. It prefers rx_missed_errors; otherwise it attempts to subtract eBPF software-drop counts from rx_dropped. The repository configuration disables netdev_hw through BlackList by default.

RDMA Port Statistics

netdev_rdma_link_<counter> exposes per-port kernel counters; available names depend on the driver. Additional labels are device, nodeguid, index, num_ports, and port.

The new port label creates new time series, including on single-port devices. Keep it for port-level queries. For a selected counter, use sum by (region, host, device) (<metric>) for device totals, or sum by (region, host, device) (rate(<metric>[5m])) for device rates; replace <metric> with the full metric name. Grouping only by device mixes identically named devices across hosts.

NIC Statistics

Collects NIC statistics through proc or netlink. Common metrics follow Linux /proc/net/dev semantics:

• receive_dropped：rx_dropped + rx_missed_errors。
• receive_frame_errors：rx_length_errors + rx_over_errors + rx_crc_errors + rx_frame_errors。
• transmit_carrier_errors：tx_carrier_errors + tx_aborted_errors + tx_window_errors + tx_heartbeat_errors。

Netlink also provides detailed error metrics and receive_nohandler. Detailed errors may already be included in the aggregates above and must not be added again. Tables show host names; container names insert container_ after netdev_.

TCP Statistics

Reads network-namespace /proc/<pid>/net/netstat and snmp, subject to MetricCollector.Netstat filters. Tables show host names; container names insert container_ after netstat_.

Only selected TcpExt fields are listed. See Linux kernel documentation for other fields and version differences.

References:

• TCP/SNMP counters

Socket Usage

Reads network-namespace /proc/<pid>/net/sockstat. Tables show host names; container names insert container_ after sockstat_. mem and mem_bytes are emitted only for the host.

I/O

Configuration

iotracing, iolatency, blk_throtl, iocost, and io_health are enabled by default. Add a feature name to the global BlackList and restart Huatuo to disable it. Disabling iotracing stops both its disk metrics and automatic diagnosis. Disabling io_health stops health events, evidence collection, and its metrics independently of iotracing.

Disk performance

The following metrics describe disk performance. They use the huatuo_bamai_iotracing_ prefix and are interval Gauges with a disk label:

Metrics cover whole disks and supported logical devices, excluding partitions and pseudo devices such as loop, ram, zram, and fd.

Latency and IO size distributions

iolatency measures read/write IO latency and size distributions for SCSI/SAS, NVMe, and virtio disks and containers. Only disks with iostats and request-time accounting enabled at startup are included. Disabling these settings during operation is unsupported. If software blk-crypto fallback is initialized at startup, the entire iolatency collector remains disabled.

SCSI issued sizes use the kernel’s 16-bit sector count and wrap at 32 MiB. When requests merge different blkcgs, container issued-size attribution is approximate; host accounting is unaffected.

Block device additions, removals, or recreation restart iolatency, blk_throtl, and iocost collection. Data for all disks may be omitted during the transition.

Bucket values count IOs in the collection interval at or below the le upper bound; +Inf is the total. Latency bounds are in seconds and size bounds in bytes. Metrics are interval Gauges.

# TYPE huatuo_bamai_iolatency_q2d_seconds_bucket gauge
huatuo_bamai_iolatency_q2d_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="0.001"} 3
huatuo_bamai_iolatency_q2d_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_d2c_seconds_bucket gauge
huatuo_bamai_iolatency_d2c_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_q2g_seconds_bucket gauge
huatuo_bamai_iolatency_q2g_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_container_q2g_seconds_bucket gauge
huatuo_bamai_iolatency_container_q2g_seconds_bucket{container_host="etcd-hostname",container_hostnamespace="kube-system",container_level="burstable",container_name="etcd",container_type="normal",disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 2

The 17 latency bounds in seconds are 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1, 2, and +Inf.

Grafana

The IO overview shows latency distributions and IO counts by collection interval. Click a bar to view latency and size histograms for that window. The panels use Business Charts (volkovlabs-echarts-panel).

Migrating former metrics

Update custom dashboards and alerts after upgrading. The table omits the huatuo_bamai_ prefix:

disk now contains names such as sda instead of device numbers such as 8:0. The new operation label distinguishes reads and writes. Former zone numbers are replaced by cumulative le bounds in seconds, listed above; they cannot be mapped by index. For a range (a, b], subtract the le=a value from the le=b value.

Former latency metrics counted only IOs of at least 20 ms. New metrics cover all read/write IO on supported disks, and D2C starts at the last dispatch. Adjust alerts for the new stages and bounds rather than only renaming metrics.

Latency buckets change from cumulative Counters to interval Gauges. Do not apply rate() or increase() to them. Query directly for one interval; for example, sum read D2C samples for a disk over the last five minutes:

sum_over_time(huatuo_bamai_iolatency_d2c_seconds_bucket{disk="sda",operation="read",le="+Inf"}[5m])

Hardware

# HELP huatuo_bamai_io_health_blkdisk_freeze the disk freeze event count
# TYPE huatuo_bamai_io_health_blkdisk_freeze counter
huatuo_bamai_io_health_blkdisk_freeze{disk="sda",host="hostname",region="dev"} 0

Select this metric under Overview → Selected Metric on the host dashboard.

blk-throttle waits

blk_throtl counts IOs delayed by blk-throttle and their mean queue wait. Each IO is counted when it leaves the queue; its wait starts at its first queue entry.

Metrics distinguish disk and operation (read or write). scope=host is the host total; scope=other is its portion without container attribution. Container series use container labels instead of scope. All four metrics are interval Gauges.

Container discovery is optional; host collection does not require kubelet. Failed lookups or invalid labels send the current data to other; recovered associations do not replay past intervals. After temporary collection failure, the first successful collection rebuilds the baseline and the next resumes output, without filling the failed interval. Errors that stop collection require troubleshooting and a Huatuo restart.

After cgroup release, uncollected tail wait deltas are included in Host and Other without container attribution. Results can undercount if one series on one CPU completes at least 67,108,864 waits, or accumulates about 2.749 million seconds of waiting, between successful collections.

IOCOST waits

iocost measures waits caused by insufficient IOCOST budget. Waiting starts when the IO enters the wait queue and ends at its final wakeup; the full wait belongs to the collection interval in which it ends.

These interval Gauges use disk (sda, nvme0n1, etc.) and operation (read or write). scope=host includes all containers; scope=other is the portion without container attribution.

Container discovery is optional; host collection does not require kubelet. Failed lookups or invalid labels send the current data to other; recovered associations do not replay past intervals. After temporary collection failure, the first successful collection rebuilds the baseline and the next resumes output, without filling the failed interval. Errors that stop collection require troubleshooting and a Huatuo restart.

After cgroup release, uncollected tail wait deltas are included in Host and Other without container attribution. Results can undercount if one series on one CPU completes at least 67,108,864 waits, or accumulates about 2.749 million seconds of waiting, between successful collections. For example, 10,240 waits for one series, each lasting 270 seconds and settled on that CPU in the same interval, exceed the wait-time capacity.

IOCOST requires a non-PREEMPT_RT Linux kernel with IOCOST functions, BPF/BTF, and kprobes.

Device health

io_health records these events:

If an NVMe state name is unavailable, new_state is unknown and new_state_raw retains the numeric state.

Block errors, NVMe timeouts, SCSI errors, and MD members entering faulty, blocked, write_error, or removed trigger evidence collection. A uniquely resolved NVMe target uses read-only nvme-cli commands; SCSI uses read-only smartctl commands. Ambiguous targets retain the event without scanning other devices. MD monitoring does not require mdadm. Evidence commands use the resolved physical disk or NVMe controller; the event’s device can retain an MD member’s partition name. Evidence commands for the same device are suppressed while running and for a 60-second cooldown; raw events remain independent.

The huatuo_bamai_io_health_ Counters track events since process startup:

MD and NVMe controller state changes are stored as events, without a current state Gauge. Counters update before events are saved; save failures are counted by event_persistence_failures_total{reason="save_error"}.

Accepted health events are stored as individual documents. Unavailable or inapplicable optional fields are omitted:

NVMe evidence includes critical_warning, media_errors_total, and up to eight error-log entries with status code, NSID, and LBA. SCSI evidence includes overall health, ASC/ASCQ, temperature, defect counts, read/write/verify error counts, and pending-defect counts with up to eight sample LBAs.

Disk Statistics

diskio reads /proc/diskstats and /proc/stat for per-device request counts, throughput, cumulative durations, and host CPU iowait.

Its disk-name label changes from device to disk; update selectors and legends in custom queries. Metric names and types are unchanged.

The repository configuration disables diskio through BlackList by default. Remove it and restart huatuo-bamai to enable it.

Request counts, bytes, and I/O durations are cumulative Counters. Use rate() to calculate IOPS, throughput, and duration growth rates. Mean request duration is the duration growth rate divided by request rate; exclude intervals with zero requests. io_in_progress and disk_iowait_percent are Gauges.

XFS

xfs collects per-device host statistics from sysfs with the additional label device. xfs_log_free_bytes is a Gauge; metrics ending in _total are cumulative Counters.

xfs_log_free_bytes is omitted when free log space is unavailable. Restart huatuo-bamai after adding an XFS mount.

System Events

Soft Lockups

Records soft lockup events detected by the kernel.

Hung Tasks

Records hung-task events detected by the kernel.

attribute hungtask events and counts to containers

Container counts are included in the host Counter and are independent of trace backoff. Events with unresolved container attribution still contribute to the host count. Container exit removes its count; agent restart resets counts. Repeated reports of the same blocked task count as separate events.

GPU

MetaX

The repository configuration disables metax_gpu through BlackList by default. Meet the device and SDK dependencies before enabling it.

MThreads

MThreads GPU collection requires the MTML management library. Startup searches for libmtml.so.2, then libmtml.so.

Remove mthreads_gpu from BlackList and restart huatuo-bamai to enable it. EnableHealth, EnablePCIe, and EnableMTLink in MetricCollector.Mthreads control the corresponding metric groups.