Reference

• 1: Metrics Reference
• 2: AutoTracing Fields
• 3: Abnormal Event Fields
• 4: Hardware Event Fields
• 5: Profiling API
• 6: profiler CLI Parameters

1 - Metrics Reference

This reference lists metric names, units, labels, sources, and measurement semantics. See Kernel-Wide Insight for capabilities and troubleshooting scenarios.

How to Read This Reference

Common metrics are grouped by CPU, memory, network, I/O, system events, and GPU for querying and troubleshooting. Actual output depends on kernel interfaces, device drivers, enabled collectors, and filters. The node’s /metrics output is authoritative for the complete set. See Node Configuration for enablement.

• Names: tables omit the huatuo_bamai_ prefix. For example, cpu_util_total is exposed as huatuo_bamai_cpu_util_total.
• Labels: all metrics include host and region. Container metrics also include container_host, container_hostnamespace, container_level, container_name, and container_type. Tables list only additional labels; — means none.
• Counts and units: counts measure events or objects, not latency. Use rate() or increase() for cumulative Counters. Some cumulative kernel statistics are exported as Gauges; interpret their field semantics rather than inferring from type or name alone.
• Examples: values illustrate output format, not diagnostic thresholds. A missing metric does not mean zero.

# TYPE huatuo_bamai_runqlat_latency counter
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
# TYPE huatuo_bamai_runqlat_container_latency counter
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226

This example shows host and container scheduling-latency bucket counts. The zone ranges are listed below.

Build Information

huatuo_bamai_build_info and huatuo_apiserver_build_info are Gauges with a constant value of 1, describing the running bamai and apiserver builds respectively. Each is exposed by its application’s /metrics endpoint and uses the same build information as that application’s /version endpoint. They carry the existing host and region labels (empty for apiserver) and these build labels:

When no release version is supplied, version uses (devel). git_commit uses the injected identifier first, then the VCS revision from Go build information, and falls back to (devel) when neither is available. Build labels apply only to this metric; business metric labels remain unchanged. After scraping, join build information with business metrics using the same job and instance. Cross-cluster queries must also include the labels that distinguish targets between clusters.

CPU

Scheduling Latency

Scheduling latency is the wait from becoming runnable to actually receiving CPU time. These eBPF metrics are cumulative Counters bucketed by zone: 0 = [0, 10) ms, 1 = [10, 20) ms, 2 = [20, 50) ms, and 3 = >=50 ms.

Softirq Latency

Measures the wait from softirq raise to processing on each CPU. Currently only NET_RX and NET_TX are emitted. This is a cumulative Counter with zone buckets: 0 = [0, 10) μs, 1 = [10, 100) μs, 2 = [100, 1000) μs, and 3 = >=1 ms.

Utilization

Host utilization is normalized by online CPU cores; container utilization by available CPU capacity. Values range from 0 to 100%.

Available Cores

Container CPU capacity is the smaller of quota-derived cores and the effective CPU set size. Without a quota, it uses the effective CPU set size; if unavailable, it falls back to online host cores.

Throttling and Contention

Throttling statistics come from cgroup CPU interfaces and include only fields provided by the kernel. The wait ratio also requires kernel wait_sum and kernel.sched_schedstats=1.

The interval wait ratio is wait-time delta / (wait-time delta + CPU-usage delta) × 100%. It measures waiting as a share of schedulable time, not the share spent throttled.

References:

• CFS bandwidth statistics
• cgroup v2 CPU interface

Burst Usage

Counts periods and time using burst CPU capacity. Emitted only when the kernel provides the corresponding fields.

Load

Hosts report 1-, 5-, and 15-minute load averages. Containers report current runnable and uninterruptible task counts, which use a different time basis.

Memory

Reclaim

Memory reclaim and compaction can block tasks. These metrics record host cumulative durations and container direct-reclaim event counts.

memory_others_container_directstall_time, memory_others_container_asyncreclaim_time, and memory_others_container_local_direct_reclaim_time depend on DiDi custom-kernel interfaces memory.directstall_stat, memory.asynreclaim_stat, and memory.local_direct_reclaim_time.

Mainline and common distribution kernels without these interfaces do not emit the metrics; no additional module is needed. Use memory_reclaim_container_directstall for container direct-reclaim counts on standard kernels.

Memory State

Container fields come from memory.stat; host fields from /proc/vmstat. Both are subject to MetricCollector.Vmstat filters. Field availability and meanings depend on kernel and cgroup versions.

Container memory

Host memory

The names below include memory_vmstat_. Underlying values are exported unchanged rather than uniformly converted to bytes. pgpgin and pgpgout use the kernel’s KiB units; see the Linux vmstat implementation.

Page state and LRU

Dirty pages and writeback

Page faults and swapping

Reclaim and scanning

Transparent Huge Pages (THP)

NUMA

References:

• cgroup v2
• cgroup v1 memory controller
• Transparent Huge Pages

Memory Events

These fields come from memory.events and count reclaim, memory-limit, and OOM events. Mainline kernels provide it in cgroup v2; DiDi custom kernels may also provide it in cgroup v1. Nothing is emitted if the interface is absent. Fields count different conditions and must not be added as independent events.

Free Memory Blocks

Shows free block counts by NUMA node, memory zone, and order. order=n means each block contains 2^n contiguous pages. The value counts blocks, not pages.

Container reclaim and compaction stall times

These cumulative Gauge values measure task time in global direct reclaim and compaction. Host totals include container work. Missing or ambiguous container records are omitted; container collection failures preserve host output. Loading requires the container probes; a load or attachment failure disables this collector. Container recreation starts a new container total without resetting the host total. BPF reload resets both.

Host-wide memcg reclaim count

This Gauge counts host-wide memcg reclaim events, excluding kswapd; it is not a global reclaim count or duration. Host totals include container activity and must not be added to container counts. BPF reload resets the total.

Network

TCP Memory

Reports TCP memory usage and its proportion of the limit.

Connection Tracking

Host nf_conntrack table usage and capacity come from /proc/sys/net/netfilter/nf_conntrack_count and /proc/sys/net/netfilter/nf_conntrack_max. The repository configuration disables net_conntrack through BlackList by default.

ARP Neighbors

Counts ARP neighbor entries in host and container network namespaces.

Transmit Queues

Qdisc metrics describe backlog, transmitted traffic, drops, and requeues, not queueing latency directly. Only root Qdiscs of matching devices are collected; noqueue is skipped. The repository configuration disables netdev_qdisc through BlackList by default.

Hardware Drops

Counts receive-side hardware drops for supported NIC drivers. It prefers rx_missed_errors; otherwise it attempts to subtract eBPF software-drop counts from rx_dropped. The repository configuration disables netdev_hw through BlackList by default.

RDMA Port Statistics

netdev_rdma_link_<counter> exposes per-port kernel counters; available names depend on the driver. Additional labels are device, nodeguid, index, num_ports, and port.

The new port label creates new time series, including on single-port devices. Keep it for port-level queries. For a selected counter, use sum by (region, host, device) (<metric>) for device totals, or sum by (region, host, device) (rate(<metric>[5m])) for device rates; replace <metric> with the full metric name. Grouping only by device mixes identically named devices across hosts.

NIC Statistics

Collects NIC statistics through proc or netlink. Common metrics follow Linux /proc/net/dev semantics:

• receive_dropped：rx_dropped + rx_missed_errors。
• receive_frame_errors：rx_length_errors + rx_over_errors + rx_crc_errors + rx_frame_errors。
• transmit_carrier_errors：tx_carrier_errors + tx_aborted_errors + tx_window_errors + tx_heartbeat_errors。

Netlink also provides detailed error metrics and receive_nohandler. Detailed errors may already be included in the aggregates above and must not be added again. Tables show host names; container names insert container_ after netdev_.

TCP Statistics

Reads network-namespace /proc/<pid>/net/netstat and snmp, subject to MetricCollector.Netstat filters. Tables show host names; container names insert container_ after netstat_.

Only selected TcpExt fields are listed. See Linux kernel documentation for other fields and version differences.

References:

• TCP/SNMP counters

Socket Usage

Reads network-namespace /proc/<pid>/net/sockstat. Tables show host names; container names insert container_ after sockstat_. mem and mem_bytes are emitted only for the host.

I/O

Configuration

iotracing, iolatency, blk_throtl, iocost, and io_health are enabled by default. Add a feature name to the global BlackList and restart Huatuo to disable it. Disabling iotracing stops both its disk metrics and automatic diagnosis. Disabling io_health stops health events, evidence collection, and its metrics independently of iotracing.

Disk performance

The following metrics describe disk performance. They use the huatuo_bamai_iotracing_ prefix and are interval Gauges with a disk label:

Metrics cover whole disks and supported logical devices, excluding partitions and pseudo devices such as loop, ram, zram, and fd.

Latency and IO size distributions

iolatency measures read/write IO latency and size distributions for SCSI/SAS, NVMe, and virtio disks and containers. Only disks with iostats and request-time accounting enabled at startup are included. Disabling these settings during operation is unsupported. If software blk-crypto fallback is initialized at startup, the entire iolatency collector remains disabled.

SCSI issued sizes use the kernel’s 16-bit sector count and wrap at 32 MiB. When requests merge different blkcgs, container issued-size attribution is approximate; host accounting is unaffected.

Block device additions, removals, or recreation restart iolatency, blk_throtl, and iocost collection. Data for all disks may be omitted during the transition.

Bucket values count IOs in the collection interval at or below the le upper bound; +Inf is the total. Latency bounds are in seconds and size bounds in bytes. Metrics are interval Gauges.

# TYPE huatuo_bamai_iolatency_q2d_seconds_bucket gauge
huatuo_bamai_iolatency_q2d_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="0.001"} 3
huatuo_bamai_iolatency_q2d_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_d2c_seconds_bucket gauge
huatuo_bamai_iolatency_d2c_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_q2g_seconds_bucket gauge
huatuo_bamai_iolatency_q2g_seconds_bucket{disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 5
# TYPE huatuo_bamai_iolatency_container_q2g_seconds_bucket gauge
huatuo_bamai_iolatency_container_q2g_seconds_bucket{container_host="etcd-hostname",container_hostnamespace="kube-system",container_level="burstable",container_name="etcd",container_type="normal",disk="sda",operation="read",host="hostname",region="dev",le="+Inf"} 2

The 17 latency bounds in seconds are 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1, 2, and +Inf.

Grafana

The IO overview shows latency distributions and IO counts by collection interval. Click a bar to view latency and size histograms for that window. The panels use Business Charts (volkovlabs-echarts-panel).

Migrating former metrics

Update custom dashboards and alerts after upgrading. The table omits the huatuo_bamai_ prefix:

disk now contains names such as sda instead of device numbers such as 8:0. The new operation label distinguishes reads and writes. Former zone numbers are replaced by cumulative le bounds in seconds, listed above; they cannot be mapped by index. For a range (a, b], subtract the le=a value from the le=b value.

Former latency metrics counted only IOs of at least 20 ms. New metrics cover all read/write IO on supported disks, and D2C starts at the last dispatch. Adjust alerts for the new stages and bounds rather than only renaming metrics.

Latency buckets change from cumulative Counters to interval Gauges. Do not apply rate() or increase() to them. Query directly for one interval; for example, sum read D2C samples for a disk over the last five minutes:

sum_over_time(huatuo_bamai_iolatency_d2c_seconds_bucket{disk="sda",operation="read",le="+Inf"}[5m])

Hardware

# HELP huatuo_bamai_io_health_blkdisk_freeze the disk freeze event count
# TYPE huatuo_bamai_io_health_blkdisk_freeze counter
huatuo_bamai_io_health_blkdisk_freeze{disk="sda",host="hostname",region="dev"} 0

Select this metric under Overview → Selected Metric on the host dashboard.

blk-throttle waits

blk_throtl counts IOs delayed by blk-throttle and their mean queue wait. Each IO is counted when it leaves the queue; its wait starts at its first queue entry.

Metrics distinguish disk and operation (read or write). scope=host is the host total; scope=other is its portion without container attribution. Container series use container labels instead of scope. All four metrics are interval Gauges.

Container discovery is optional; host collection does not require kubelet. Failed lookups or invalid labels send the current data to other; recovered associations do not replay past intervals. After temporary collection failure, the first successful collection rebuilds the baseline and the next resumes output, without filling the failed interval. Errors that stop collection require troubleshooting and a Huatuo restart.

After cgroup release, uncollected tail wait deltas are included in Host and Other without container attribution. Results can undercount if one series on one CPU completes at least 67,108,864 waits, or accumulates about 2.749 million seconds of waiting, between successful collections.

IOCOST waits

iocost measures waits caused by insufficient IOCOST budget. Waiting starts when the IO enters the wait queue and ends at its final wakeup; the full wait belongs to the collection interval in which it ends.

These interval Gauges use disk (sda, nvme0n1, etc.) and operation (read or write). scope=host includes all containers; scope=other is the portion without container attribution.

Container discovery is optional; host collection does not require kubelet. Failed lookups or invalid labels send the current data to other; recovered associations do not replay past intervals. After temporary collection failure, the first successful collection rebuilds the baseline and the next resumes output, without filling the failed interval. Errors that stop collection require troubleshooting and a Huatuo restart.

After cgroup release, uncollected tail wait deltas are included in Host and Other without container attribution. Results can undercount if one series on one CPU completes at least 67,108,864 waits, or accumulates about 2.749 million seconds of waiting, between successful collections. For example, 10,240 waits for one series, each lasting 270 seconds and settled on that CPU in the same interval, exceed the wait-time capacity.

IOCOST requires a non-PREEMPT_RT Linux kernel with IOCOST functions, BPF/BTF, and kprobes.

Device health

io_health records these events:

If an NVMe state name is unavailable, new_state is unknown and new_state_raw retains the numeric state.

Block errors, NVMe timeouts, SCSI errors, and MD members entering faulty, blocked, write_error, or removed trigger evidence collection. A uniquely resolved NVMe target uses read-only nvme-cli commands; SCSI uses read-only smartctl commands. Ambiguous targets retain the event without scanning other devices. MD monitoring does not require mdadm. Evidence commands use the resolved physical disk or NVMe controller; the event’s device can retain an MD member’s partition name. Evidence commands for the same device are suppressed while running and for a 60-second cooldown; raw events remain independent.

The huatuo_bamai_io_health_ Counters track events since process startup:

MD and NVMe controller state changes are stored as events, without a current state Gauge. Counters update before events are saved; save failures are counted by event_persistence_failures_total{reason="save_error"}.

Accepted health events are stored as individual documents. Unavailable or inapplicable optional fields are omitted:

NVMe evidence includes critical_warning, media_errors_total, and up to eight error-log entries with status code, NSID, and LBA. SCSI evidence includes overall health, ASC/ASCQ, temperature, defect counts, read/write/verify error counts, and pending-defect counts with up to eight sample LBAs.

Disk Statistics

diskio reads /proc/diskstats and /proc/stat for per-device request counts, throughput, cumulative durations, and host CPU iowait.

Its disk-name label changes from device to disk; update selectors and legends in custom queries. Metric names and types are unchanged.

The repository configuration disables diskio through BlackList by default. Remove it and restart huatuo-bamai to enable it.

Request counts, bytes, and I/O durations are cumulative Counters. Use rate() to calculate IOPS, throughput, and duration growth rates. Mean request duration is the duration growth rate divided by request rate; exclude intervals with zero requests. io_in_progress and disk_iowait_percent are Gauges.

XFS

xfs collects per-device host statistics from sysfs with the additional label device. xfs_log_free_bytes is a Gauge; metrics ending in _total are cumulative Counters.

xfs_log_free_bytes is omitted when free log space is unavailable. Restart huatuo-bamai after adding an XFS mount.

System Events

Soft Lockups

Records soft lockup events detected by the kernel.

Hung Tasks

Records hung-task events detected by the kernel.

attribute hungtask events and counts to containers

Container counts are included in the host Counter and are independent of trace backoff. Events with unresolved container attribution still contribute to the host count. Container exit removes its count; agent restart resets counts. Repeated reports of the same blocked task count as separate events.

GPU

MetaX

The repository configuration disables metax_gpu through BlackList by default. Meet the device and SDK dependencies before enabling it.

MThreads

MThreads GPU collection requires the MTML management library. Startup searches for libmtml.so.2, then libmtml.so.

Remove mthreads_gpu from BlackList and restart huatuo-bamai to enable it. EnableHealth, EnablePCIe, and EnableMTLink in MetricCollector.Mthreads control the corresponding metric groups.

2 - AutoTracing Fields

This reference covers AutoTracing record fields, units, and examples. See AutoTracing for use cases and Node Configuration for settings. Thresholds below are built-in defaults that configuration files can override.

Results

Automatic tracing events are written to Elasticsearch or local files according to enabled storage settings. Filter by tracer_type=autotracing, tracer_name, host or container identifiers, and time range. The JSON examples illustrate field structure, not measured results.

CPU examples use perf. With profiler, events retain trigger metrics; profiles are stored separately in Elasticsearch with the same tracer_id. See Node Configuration for storage requirements and viewing instructions.

Common Fields

Records contain these common fields. Container fields are provided only when a container is associated.

Host CPU

Tracer: cpusys. Select perf (default) or profiler with AutoTracing.CPUSys.Tool.

Periodically reads /proc/stat to detect host system, user (user + nice), and executing-total (user + nice + system + irq + softirq) CPU bursts, including container work. Any group triggers profiling when both utilization and its increase strictly exceed its thresholds. Default utilization/increase thresholds are system 45%/20 percentage points, user 75%/45 percentage points, and total 90%/55 percentage points.

All three groups share a capture and a default 30-minute cooldown. Thresholds are configured under CPUSys independently of CPUIdle; see the configuration reference. Records retain system fields and add matching user_percent* and total_percent* fields; zero-valued additional fields are omitted.

Example with some common fields omitted:

{
    "tracer_name": "cpusys",
    "tracer_data": {
        "system_percent": 52,
        "system_percent_threshold": 45,
        "system_percent_delta": 25,
        "system_percent_delta_threshold": 20,
        "flamedata": [
            {"level": 0, "value": 1000, "self": 0, "label": "all"},
            {"level": 1, "value": 350, "self": 350, "label": "do_syscall_64"}
        ]
    }
}

Container CPU

Tracer: cpuidle. Select perf (default) or profiler with AutoTracing.CPUIdle.Tool.

Periodically reads container cgroup CPU statistics. Container-level CPU sampling starts when user, system, or total utilization exceeds both its utilization threshold and its increase threshold between consecutive samples.

The default cooldown is 30 minutes per container. AutoTracing.CPUIdle.Filter controls target scope. Built-in thresholds:

Example with some common fields omitted:

{
    "tracer_name": "cpuidle",
    "tracer_data": {
        "user_percent": 80,
        "user_percent_threshold": 75,
        "user_percent_delta": 48,
        "user_percent_delta_threshold": 45,
        "system_percent": 12,
        "system_percent_threshold": 45,
        "system_percent_delta": 5,
        "system_percent_delta_threshold": 20,
        "total_percent": 92,
        "total_percent_threshold": 90,
        "total_percent_delta": 53,
        "total_percent_delta_threshold": 55,
        "flamedata": [
            {"level": 0, "value": 1000, "self": 0, "label": "all"},
            {"level": 1, "value": 800, "self": 800, "label": "java/com.example.App.main"}
        ]
    }
}

Container D-State Load

Tracer: dload.

Reads container process states through netlink and cgroups to calculate an exponential moving average (EMA) of uninterruptible (D-state) process load. It triggers when the EMA exceeds the threshold (default: 5), with a default 30-minute cooldown per container.

Results include kernel stacks of container and host D-state processes. AutoTracing.IssuesList labels known issues.

Example with some common fields omitted:

{
    "tracer_name": "dload",
    "tracer_data": {
        "threshold": 5,
        "nr_sleeping": 120,
        "nr_running": 4,
        "nr_stopped": 0,
        "nr_uninterruptible": 8,
        "nr_iowait": 3,
        "load_avg": 7.23,
        "dload_avg": 6.81,
        "known_issue": "",
        "stack": "task:java            state:D stack:    0 pid: 12345 tgid: 12345 ...\n  io_schedule+0x18/0x40\n  ext4_file_write_iter+0x..."
    }
}

Disk I/O

Tracer: iotracing.

Reads /proc/diskstats every 5 seconds. A trigger condition must hold for two consecutive intervals. Non-NVMe devices check utilization; NVMe devices require both utilization and either read or write throughput to exceed thresholds. Each device also checks mean read and write wait times independently. Results include process and file I/O rates and I/O scheduling wait stacks.

When running iotracing directly, --enable-pprof (disabled by default) serves profiles at http://127.0.0.1:6000/debug/pprof/; if the address is already in use, the command fails to start. These profiles describe only the iotracing process’s Go runtime, not huatuo-bamai or the application processes performing disk I/O. Automatic tracing does not enable this option.

Example with some common fields omitted:

{
    "tracer_name": "iotracing",
    "tracer_data": {
        "reason_snapshot": {
            "type": "ioutil",
            "device": "sda",
            "major_num": 8,
            "minor_num": 0,
            "iostatus": {
                "read_bps": 125829120,
                "read_iops": 450,
                "read_await": 12,
                "write_bps": 2202009600,
                "write_iops": 890,
                "write_await": 145,
                "io_util": 95,
                "queue_size": 32
            }
        },
        "process_file_io_stats": [
            {
                "pid": 12345,
                "comm": "java",
                "container_hostname": "app-pod-xxx",
                "total_fs_read_bps": 0,
                "total_fs_write_bps": 52428800,
                "total_disk_read_bps": 0,
                "total_disk_write_bps": 49152000,
                "total_files": [
                    {
                        "path": "/data/logs/app.log",
                        "dev_name": "sda",
                        "fs_write_bps": 52428800,
                        "disk_write_bps": 49152000
                    }
                ],
                "total_file_count": 1
            }
        ],
        "io_schedule_timeout_stacks": [
            {
                "pid": 12345,
                "tid": 12346,
                "cpu": 2,
                "comm": "java",
                "container_hostname": "app-pod-xxx",
                "schedule_latency_us": 250000,
                "stack": [
                    "io_schedule+0x18/0x40",
                    "ext4_file_write_iter+0x2a0/0x4c0"
                ]
            }
        ]
    }
}

Host Anonymous Memory

Tracer: memburst.

By default, samples host anonymous memory every 10 seconds and evaluates after filling a 60-sample sliding window. It triggers when current anonymous memory reaches twice the oldest window value and at least 70% of total host memory. The default cooldown is 30 minutes.

Results list the top N processes by descending RSS (default: 10) and attempt runtime snapshots serially for the first min(N, 3) processes. Go, HotSpot, and CPython use their respective providers. Unsupported runtimes report unavailable and retain process memory counters when available. A failed capture does not mean zero memory usage.

Each process has a two-second capture budget. Each snapshot retains at most 10 memory object entries and 512 KiB of JSON. Timeouts are cooperative and cannot interrupt synchronous system calls, so capture may take longer than two seconds.

The complete tracer_data is limited to 2 MiB, with at most 1,000 persisted ranking entries. When limits are exceeded, the lowest-ranked snapshots are removed first, then ranking tails if needed, and snapshot_reason is set. Cancellation and fallback records follow the same limits.

Example (some common fields omitted):

{
    "tracer_name": "memburst",
    "tracer_data": {
        "top_memory_usage": [
            {
                "PID": 3456,
                "ProcessName": "go-service",
                "MemSize": 268435456
            }
        ],
        "process_snapshots": [
            {
                "pid": 3456,
                "process_name": "go-service",
                "language": "go",
                "snapshot": {
                    "runtime_version": "go1.24.0",
                    "status": "complete",
                    "duration_ms": 25,
                    "entries": [
                        {
                            "kind": "inuse_space_objects",
                            "name": "main.allocate",
                            "bytes": 1048576,
                            "objects": 16,
                            "stack": [
                                "main.allocate fixture.go:42"
                            ]
                        }
                    ]
                },
                "process_memory": {
                    "status": "complete",
                    "rss_bytes": 268435456,
                    "rss_anon_bytes": 260046848
                }
            }
        ]
    }
}

snapshot fields:

process_memory contains language-independent process memory counters. Its status and optional status_reason describe availability. All counters below are optional; missing values mean unavailable, not zero:

3 - Abnormal Event Fields

This reference covers abnormal event structures, fields, and limitations. See Abnormal Event Diagnosis for capabilities and Node Configuration for collection settings.

Event Records

Results are written to Elasticsearch or local files according to enabled storage settings. Filter by tracer_type=event, event name, host or container, and time range. The JSON examples illustrate structure and omit some common fields. Threshold descriptions use built-in defaults.

Container fields are provided only when a container is associated. Event-specific data is in tracer_data.

CPU Scheduling

Scheduling Tick Intervals

Observer: sched_tick.

Measures the total interval between consecutive scheduling ticks. When the interval reaches the threshold, it records the current kernel stack and process information. This can reveal prolonged IRQ disabling, CPU stalls, or virtualization scheduling delays, but does not alone prove that softirqs were disabled. comm and pid identify the task interrupted by the reporting tick, not the task responsible for the delay.

Use cases

• Kernel or driver code disables local interrupts for a long time, or hardirq/NMI work occupies the CPU for too long.
• The host preempts a VM’s vCPU for an extended period, including high steal time and scheduling stalls.
• Low-level anomalies such as SMI, firmware stalls, or delayed or lost clockevent delivery.

Limitations

Normal tick suppression after successfully entering NO_HZ is filtered out. High CPU load or a softirq backlog alone does not imply tick delay.

Set the threshold above the machine’s normal tick interval. The event stack captures only the first tick after the delay ends; use IRQ, steal-time, and hardware metrics to investigate the cause.

Example with some common fields omitted:

{
    "uploaded_timestamp": "2025-06-11T16:05:16.251152703+08:00",
    "hostname": "***",
    "tracer_data": {
        "tick_interval_ns": 237328905,
        "tick_interval_threshold_ns": 10000000,
        "comm": "***-agent",
        "pid": 688073,
        "cpu": 1,
        "stack": "scheduler_tick/..."
    },
    "observed_timestamp": "2025-06-11T16:05:16.251+08:00",
    "tracer_type": "event",
    "region": "***",
    "tracer_name": "sched_tick"
}

Soft Lockups

Observer: softlockup.

Records the process, CPU, and NMI backtraces when the kernel reports a soft lockup. These details alone cannot identify the responsible process. Kernel watchdog settings determine trigger timing.

The reporting interval for repeated events increases from 10 minutes to at most 3 hours. Events are still counted during this interval.

Example with some common fields omitted:

{
    "tracer_data": {
        "cpu": 15,
        "pid": 12345,
        "comm": "kworker/15:0",
        "cpus_stack": "2025-06-10 14:30:22 sysrq: Show backtrace of all active CPUs\nNMI backtrace for cpu 15\n..."
    }
}

Hung Tasks

Observer: hungtask.

Matched hungtask events carry a container ID; unresolved events retain host output. Stack snapshots remain system-wide. See container counts.

Detects system hung-task events and collects kernel stacks of D-state (uninterruptible sleep) processes and CPU backtraces. The repeat reporting interval increases from 10 minutes to at most 3 hours, while events continue to be counted. The observer is not registered if the kernel does not provide hung_task_timeout_secs. Setting that value to 0 disables kernel hung-task detection.

Example with some common fields omitted:

{
    "tracer_data": {
        "tid": 2567042,
        "comm": "kworker/u48:2",
        "cpus_stack": "2025-06-10 09:57:14 sysrq: Show backtrace of all active CPUs\nNMI backtrace for cpu 33\n...",
        "blocked_processes_stack": "task:java            state:D stack:    0 pid: 12345 ..."
    }
}

Memory

OOM Kills

Observer: memory_oom_kill.

Records the process that triggered the OOM Killer (trigger), the killed process (victim), and available container, cgroup, and host memory information. The triggering process is not necessarily the main memory consumer; analyze usage and memory limits together.

Example with some fields omitted:

{
    "tracer_name": "memory_oom_kill",
    "tracer_data": {
        "trigger": {
            "memory_cgroup_css_addr": "0xff4b8d8be3818000",
            "container_id": "example-container",
            "container_hostname": "app-pod",
            "pid": 3218804,
            "comm": "java"
        },
        "victim": {
            "memory_cgroup_css_addr": "0xff4b8d8be3818000",
            "container_id": "example-container",
            "container_hostname": "app-pod",
            "pid": 3218745,
            "comm": "java",
            "cgroup": {
                "path": "kubepods.slice/...",
                "current": 2143289344,
                "max": 2147483648,
                "stat": {
                    "anon": 1509949440,
                    "file": 83886080
                },
                "events": {
                    "oom": 1,
                    "oom_kill": 1
                }
            }
        },
        "memory_snapshot": {
            "host_meminfo": {
                "MemAvailable": 3355443200,
                "Cached": 1073741824,
                "Slab": 268435456
            }
        }
    }
}

cgroup appears inside trigger or victim only when container attribution and snapshot reads succeed. The host snapshot may also be absent. These values are read in user space and are not strictly synchronized with the OOM event.

Direct Reclaim

Observer: memory_reclaim_events.

Records the duration of an individual direct memory reclaim by a thread. When try_to_free_pages returns after exceeding the threshold (default: 900 milliseconds), it reports process, thread, and container details. Events without a resolved container are retained in the host stream. Blacklists use the registration name memory_reclaim_events; stored records use tracer_name=memory_reclaim.

Example with some common fields omitted:

{
    "tracer_name": "memory_reclaim",
    "tracer_data": {
        "pid": 1896137,
        "tid": 1896138,
        "comm": "java",
        "reclaim_duration_ns": 1412702917
    }
}

An empty container ID can mean a host task or unresolved container attribution. Resolved container events are unaffected and are not duplicated in the host stream.

Network

Protocol Stack Drops

Observer: dropwatch.

Detects packet drops in the kernel network stack and reports kernel stacks, network five-tuples, TCP state, and other details. An optional stack filter can suppress noise patterns verified on the local system. type is reserved for TCP drop-type compatibility and is currently unset.

Example with some common fields omitted:

{
    "observed_timestamp": "2026-07-23T02:14:40.304775546Z",
    "tracer_data": {
        "drop_reason": "SKB_DROP_REASON_NOT_SPECIFIED",
        "source": "events",
        "comm": "kubelet",
        "pid": 1687046,
        "net_namespace_cookie": 123456789,
        "net_namespace_inum": 402653184,
        "netdev_queue_mapping": 3,
        "netdev_linkstatus": ["linkStatusUp"],
        "netdev_name": "eth0",
        "netdev_ifindex": 2,
        "packet_eth_proto": "0x0800",
        "packet_len_bytes": 1460,
        "layers": {
            "label": "IPv4/TCP",
            "ipv4": {
                "saddr": "10.79.68.62",
                "daddr": "10.134.72.4",
                "protocol": "TCP"
            },
            "tcp": {
                "sport": 8080,
                "dport": 49000,
                "seq": 1009085774,
                "ack_seq": 689410995,
                "sk_state": "ESTABLISHED"
            }
        },
        "stack": "kfree_skb/..."
    }
}

Receive Latency

Observer: net_rx_latency.

Records TCP packet latency from the driver receive point to __netif_receive_skb, tcp_v4_rcv, and the user-copy point. All three durations start at the driver, not at the preceding observation point. Default thresholds are 5, 10, and 115 milliseconds, respectively. Reaching a threshold records endpoints, TCP state, sequence numbers, and latency.

The host network namespace is excluded by default; only container networking is observed.

Example with some common fields omitted:

{
    "observed_timestamp": "2025-06-11T00:00:00Z",
    "tracer_data": {
        "comm": "nginx",
        "pid": 2921092,
        "latency_stage": "RX_STAGE_USERCOPY",
        "latency_ms": 95973,
        "latency_threshold_ms": 115,
        "tcp_state": "ESTABLISHED",
        "tcp_saddr": "10.156.248.76",
        "tcp_daddr": "10.134.72.4",
        "tcp_sport": 9213,
        "tcp_dport": 49000,
        "tcp_seq": 1009085774,
        "tcp_ack_seq": 689410995,
        "net_namespace_cookie": 123456789,
        "net_namespace_inum": 402653184,
        "packet_len_bytes": 26064
    }
}

NIC State

Observer: netdev_events.

Subscribes to kernel netlink RTM_NEWLINK messages to detect NIC link-flag changes such as AdminDown or CarrierDown. It reports interface name, link state, MAC address, and driver details. At startup, it scans devices in EventTracing.Netdev.DeviceList to establish a baseline, then reports only state changes.

Example with some common fields omitted:

{
    "tracer_data": {
        "ifname": "eth1",
        "index": 3,
        "linkstatus": "linkStatusAdminDown, linkStatusCarrierDown",
        "mac": "5c:6f:69:34:dc:72",
        "start": false,
        "driver": "ixgbe",
        "driver_version": "5.1.0-k",
        "firmware_version": "3.25 0x80000421 1.2163.0"
    }
}

LACP State

Observer: netdev_bonding_lacp.

Enabled only when the system has an IEEE 802.3ad bonding interface. On LACP (Link Aggregation Control Protocol) state changes, it reads and records all bonding interfaces under /proc/net/bonding/, including mode, MII state, Actor/Partner negotiation parameters, and slave link states.

Stored records use tracer_name=lacp; configuration blacklists use the registration name netdev_bonding_lacp.

Example with some common fields omitted:

{
    "tracer_name": "lacp",
    "tracer_data": {
        "content": "/proc/net/bonding/bond0\nEthernet Channel Bonding Driver: v4.18.0...\nBonding Mode: IEEE 802.3ad Dynamic link aggregation\nMII Status: down\n..."
    }
}

Transmit Queue Timeouts

Observer: netdev_txqueue_timeout.

Detects NIC TX queue timeouts and records the affected queue index, device name, and driver name to investigate transmit-side device, driver, or link anomalies.

Example with some common fields omitted:

{
    "tracer_data": {
        "queue_index": 3,
        "device_name": "eth0",
        "driver_name": "ixgbe"
    }
}

Compatibility

OOM Name Migration

The OOM kill tracer is now memory_oom_kill, previously memory_oom and earlier oom. Update blacklists, event filters, and alert queries together.

The metric prefix is now huatuo_bamai_memory_oom_kill_, replacing huatuo_bamai_memory_oom_ and the earlier huatuo_bamai_oom_. Update metric dashboards and recording rules accordingly.

Historical events retain their original names. Cross-version queries must match all three; the built-in event dashboard already covers them.

4 - Hardware Event Fields

This reference covers RAS metrics, event structures, severity, and source fields. See Hardware Fault Diagnosis for coverage and enablement.

EventTracing.Ras.MceThrBackoff controls the save interval for threshold-interrupt events. Events are still counted during cooldown, so counter increases need not equal the number of saved records.

Results

Error Counts

huatuo_bamai_ras_hw_err_total counts received RAS events by type: mce, edac, acpi, aer, thr, and arm_ghes. The presence of a label does not prove that the current platform supports its probe.

huatuo_bamai_ras_hw_err_total{hostname="node-01",region="region-01",type="edac"} 12

These are example values. The unit is events, not the sum of EDAC err_count values. See Kernel-Wide Insight for NIC hardware drops and PFC metrics, and Abnormal Event Diagnosis for NIC state changes.

Event Records

Query with tracer_name=ras and tracer_type=event. This EDAC example omits some fields:

{
    "hostname": "node-01",
    "observed_timestamp": "2026-05-11T08:58:47.329Z",
    "kernel_observed_timestamp": "2026-05-11T08:58:47.328548319Z",
    "tracer_name": "ras",
    "tracer_type": "event",
    "tracer_data": {
        "dev": "MEM",
        "event": "EDAC",
        "type": "Corrected",
        "info": "{\"err_count\":1,\"err_type\":\"Corrected\",\"err_msg\":\"memory read error\",\"label\":\"CPU_SrcID#0_Ha#0_Chan#0_DIMM#0\",\"mc_index\":0,\"addr\":7860269056,\"grain\":128}"
    }
}

Severity

Severity classifies the event. It does not establish successful recovery, absence of application impact, or a need to reboot.

If MCE sets both DEFERRED and UC, UncorrectedDeferred takes precedence.

Event Details

The following fields are in the parsed tracer_data.info.

MCE

Register and bank meanings depend on the CPU model. Use the corresponding processor manual for diagnosis.

MCE Threshold Interrupts

EDAC

ACPI GHES

Non-standard sections (NVDIMM, CXL, and vendor payloads) can exceed the copied window. raw_data contains only the bytes available after the section offset: the length is capped by the remaining window, so with a valid record length unused bytes at the end are not reported as payload. The window also holds the record’s fixed fields and any data before the section, so it is not a per-record raw_data capacity. data_len still reports the length the kernel put in the record. Both values come from the kernel record and are bounded before use, so a malformed event cannot expose bytes outside the copied window.

PCIe AER

Error names describe the reported condition and do not prove device damage. The current parser recognizes the following status values.

Correctable Errors

Uncorrectable Errors

ARM GHES

ARM_GHES records use severity unknown. See Coverage for platform limitations.

5 - Profiling API

This reference covers profiling task requests, responses, and states. For a first collection, see the Profiling Guide.

huatuo-apiserver creates, supervises, and stops profiling tasks through /v1/profiling. Each HTTP creation request is an independent execution. Profiling tasks are persisted as Jobs in huatuo-apiserver; the node keeps only in-memory operations and profiler processes.

Task control does not depend on Elasticsearch. Raw-result queries and dashboard links require it. huatuo-bamai and huatuo-apiserver must use the same index.

Request Conventions

huatuo-apiserver listens on :12740 by default. These examples set the address and Bearer token through environment variables:

API_BASE="http://127.0.0.1:12740"
API_TOKEN="REPLACE_WITH_RANDOM_HEX"

Every request must supply the configured Bearer token in Authorization:

Authorization: Bearer REPLACE_WITH_RANDOM_HEX

Non-admin users need permissions for /v1/profiling and /v1/profiling/**. Permissions may include an HTTP method prefix, such as GET /v1/profiling/**. Successful responses contain only data:

{
  "data": {}
}

Error responses contain a stable error code and a readable message:

{
  "error": {
    "code": "result_unavailable",
    "message": "Job state does not provide a complete Profiling result"
  }
}

Clients should branch on error.code, not parse error.message.

Query Capabilities

Before creating a task, query supported profiling types, languages, CPU and memory modes, and runtime parameters:

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiling/capabilities"

data.items is a static capability table shipped with each version. Entries contain:

CPU profiling for c, c++, and go supports oncpu and offcpu; java and python support only oncpu. Memory profiling supports:

Create a Task

JSON parameters for POST /v1/profiling:

Every request creates a new task, even with identical content. Exceeding per-node or process-wide active task quotas returns HTTP 429.

Create a host Go CPU profiling task:

curl -sS -i \
  -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "cpu",
    "language": "go",
    "mode": "oncpu",
    "scope": "host",
    "duration_seconds": 60,
    "hostname": "node-01"
  }' \
  "${API_BASE}/v1/profiling"

Create a Java live-object profiling task in a container:

curl -sS -i \
  -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "memory",
    "language": "java",
    "mode": "object_usage",
    "scope": "container",
    "duration_seconds": 60,
    "container_id": "9f4c2f1a8b7d",
    "hostname": "node-01"
  }' \
  "${API_BASE}/v1/profiling"

Success returns 201 Created, a Location header pointing to the new task, and the task ID needed for later queries:

{
  "data": {
    "request_id": "<profile-job-id>",
    "hostname": "node-01",
    "duration_seconds": 60,
    "scope": "host",
    "type": "cpu",
    "language": "go",
    "mode": "oncpu",
    "status": "pending",
    "created_at": "2026-08-24T10:00:00Z",
    "updated_at": "2026-08-24T10:00:00Z"
  }
}

JOB_ID="<profile-job-id>"

Query Tasks

Task list

GET /v1/profiling supports these query parameters:

Query the 20 most recent profiling tasks:

curl -sS -G \
  -H "Authorization: Bearer ${API_TOKEN}" \
  --data-urlencode "limit=20" \
  --data-urlencode "offset=0" \
  "${API_BASE}/v1/profiling"

The server returns only profiling tasks, ordered by created_at descending. data.items is the task array; data.limit and data.offset are the effective pagination parameters; data.has_more indicates another page. Non-admin users see only their own tasks.

Task details

GET /v1/profiling/:request_id returns one task:

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiling/${JOB_ID}"

Task information is in data:

Task states:

terminal.outcome is completed, failed, stopped, or unknown. For failed, inspect terminal.reason and terminal.message. stopped means an explicit stop and possibly incomplete results. unknown means huatuo-apiserver cannot confirm the final operation state, although durably published results may still be available.

operation_lost and execution_timed_out are terminal.reason codes, not task states. Other reasons include startup or stop timeouts, unavailable nodes, execution-capacity limits, startup or runtime failure, invalid_node_request for an invalid node request constructed by huatuo-apiserver, and protocol errors.

Get Results

GET /v1/profiling/:request_id/raw returns raw windows for tasks with terminal.outcome of completed or unknown only while the durable publication marker exists. A missing or expired marker returns result_not_found. Active states return result_not_ready; failed and stopped return result_unavailable.

Responses may be large; save them directly to a file:

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -o profile-raw.json \
  "${API_BASE}/v1/profiling/${JOB_ID}/raw?limit=100&offset=0"

Windows are in data.items; data.limit, data.offset, and data.has_more describe pagination. Each record includes uploaded_timestamp, started_timestamp, profile_type, and pprof-compatible profile data. A durably published empty result succeeds with an empty items array. Explicit limit=0 is invalid; only omission uses the default. limit defaults to 20 and is capped at 100. Encoded Profile data above 64 MiB returns 413 result_too_large; reduce limit and retry.

Flame graphs read and merge batches of 100 windows, with a total limit of 100000 per request. At one window per second for an hour, all 3600 windows participate. Exceeding 100000 returns an error asking for a narrower time range or filters. Query or merge failures never return partial flame graphs. Raw Profile lists remain scoped to a task; their page limit does not cap aggregation for a complete flame graph.

Stop a Task

Stopping is an asynchronous intent accepted for tasks in pending, running, or stopping:

curl -sS \
  -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiling/${JOB_ID}/stop"

Success returns 200 OK with the current task snapshot. Terminal tasks return 409 Conflict. Because results may be incomplete, stopped tasks do not support result queries.

Unified flame graph queries

Grafana uses the Pyroscope-compatible Protobuf endpoints under /v1/profiling/flamegraph/. SelectMergeStacktraces selectors require scope="host" or scope="container". region is the Agent’s configured region; hostname is only the collection node. Container queries require container_id or one task id. Host queries cannot include container filters. For example: {region="example",hostname="node-01",scope="container",container_id="…",id="…"}.

Each label accepts one equality condition. On this endpoint the wildcard values *, all, and the empty string only drop the restriction for that label, so {scope="host",hostname="*",id="job-1"} still selects one task; scope is still required. Repeating a label with the same value is accepted, while contradictory duplicates such as {hostname="host-a",hostname="host-b"} are rejected with 400 and the error reports both values, because one query cannot select two hosts. A matcher that uses any operator other than = is rejected as well, including hostname!="*".

The shared dashboard UID is continuous-profiling; variables are ordered as Region, node, scope, container, profile type, and task. The timeline and flame graph use the same task filter. With All containers selected, the task still identifies one target. Physical usage always requires one task and selects only the latest snapshot. The task summary uses the Infinity data source in pyroscope.yaml, with the same API token as the Pyroscope data source. result_url includes the stored windows’ Region and actual profile type. Results without windows do not provide a dashboard link.

6 - profiler CLI Parameters

This reference lists profiler parameters, defaults, and applicability. See CLI Collection for prerequisites and examples, and Output Formats to choose a format.

Both --output-path and --output-format must be explicit. The declared default for --output-format is collapsed, but validation does not allow the flag to be omitted.

General

These pprof endpoints describe the profiler process’s own Go runtime, not the --pid target or Java/Python collector subprocesses. The listener is now restricted to loopback rather than all interfaces; an occupied port fails startup.

Collection

When the hardware PMU event or sampling is unsupported, the command exits with code 1 and identifies the error on stderr with Error: hardware_pmu_unavailable:. Permission and other startup errors do not use this marker.

Logging