Key Feature on HUATUO Documentation

Metrics

Sun, 11 Jan 2026 00:00:00 +0000

Metrics supported in the current version:

CPU

Scheduling

The following metrics allow observation of process scheduling latency, i.e., the time from when a process becomes runnable (placed in the run queue) until it actually starts executing on the CPU.

# HELP huatuo_bamai_runqlat_container_latency cpu run queue latency for the containers
# TYPE huatuo_bamai_runqlat_container_latency gauge
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="3"} 0

# HELP huatuo_bamai_runqlat_latency cpu run queue latency for the host
# TYPE huatuo_bamai_runqlat_latency gauge
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="3"} 0

Metric	Description	Unit	Target	Source	Labels
runqlat_container_latency	scheduling latency histogram buckets: zone0: 0–10 ms zone1: 10–20 ms zone2: 20–50 ms zone3: 50+ ms	count	Container	eBPF	container_host, container_hostnamespace, container_level, container_name, container_type, host, region, zone
runqlat_latency	scheduling latency histogram buckets: zone0, 0~10ms zone1, 10-20ms zone2, 20-50ms zone3, 50+ms	count	Host	eBPF	host, region, zone

SoftIRQ

SoftIRQ response latency on different CPUs (currently only NET_RX and NET_TX are collected).

Events

Sun, 11 Jan 2026 00:00:00 +0000

The HUATUO platform uses eBPF technology to detect various abnormal events in the Linux kernel in real time, helping users quickly locate issues related to the system, applications, and hardware.

Supported Events

Event Name	Core Function	Typical Scenarios
softirq	Detects excessively long softirq disable time in the kernel, outputs call stack and process information	Resolves system stalls, network latency, and scheduling delays
softlockup	Detects softlockup events and provides target process and kernel stack information	Locates and resolves system softlockup issues
hungtask	Detects hungtask events, outputs all D-state processes and their stack information	Captures transient mass D-state process scenarios and preserves fault scenes
oom	Detects OOM events in the host or containers	Focuses on memory exhaustion issues and provides detailed fault snapshots
memory_reclaim_events	Detects direct memory reclaim events, records reclaim duration, process and container information	Resolves business stalls caused by memory pressure
ras	Detects hardware faults in CPU, Memory, PCIe, etc.	Timely awareness of hardware failures to reduce business impact
dropwatch	Detects packet drops in the kernel network protocol stack, outputs call stack and network context	Resolves business jitters and latency caused by protocol stack packet drops
net_rx_latency	Detects latency events in the protocol stack receive path (driver → protocol → user space)	Resolves business timeouts and jitters caused by receive latency
netdev_events	Detects network device link status changes	Detects physical link failures on network cards
netdev_bonding_lacp	Detects bonding LACP protocol status changes	Identifies fault boundaries between physical machines and switches
netdev_txqueue_timeout	Detects network card transmit queue timeout events	Locates hardware failures in network card transmit queues

Event Details

Common Fields

hostname: Physical machine hostname
region: Availability zone where the physical machine is located
uploaded_time: Data upload time
container_id: Container ID if the event is associated with a container
container_hostname: Container hostname if the event is associated with a container
container_host_namespace: Kubernetes namespace of the container if the event is associated with a container
container_type: Container type, e.g., normal for regular containers, sidecar for sidecar containers, etc.
container_qos: Container QoS level
tracer_name: Event name
tracer_id: Tracing ID for this event
tracer_time: Time when tracing was triggered
tracer_type: Trigger type — manual or automatic
tracer_data: Tracer-specific private data

1. softirq

Description
Detects when the kernel disables interrupts for too long. Records the kernel call stack during the disable period, current process information, and other key data to help analyze interrupt-related latency issues.

Autotracing

Sun, 11 Jan 2026 00:00:00 +0000

Overview

AutoTracing is an intelligent diagnostic feature of the Huatuo kernel monitoring system.

When the system experiences specific performance anomalies or sudden resource spikes, AutoTracing is automatically triggered. It captures detailed on-site information in real time, including flame graphs, process context, call stacks, and resource status. This helps operations and development teams quickly locate and analyze issues without manual intervention.

This feature is built on eBPF technology, offering low overhead and high real-time performance. It is suitable for anomaly diagnosis in both physical machines and container environments.

Hardware Errors

Fri, 06 Mar 2026 00:00:00 +0000

Architecture

The huatuo supports detection of various hardware faults, including:

CPU, L1/L2/L3 Cache, TLB
Memory, ECC
PCIe
Network Interface Card Link
PFC / RDMA
ACPI
GPU MetaX

Overall Architecture of HUATUO

The huatuo is built on Linux kernel MCE (Machine Check Exception) and RAS (Reliability, Availability, and Serviceability) mechanisms. It uses eBPF to capture critical hardware events and retrieve device information. The Linux kernel RAS framework has been continuously evolving since kernel 2.6, gradually adding more tracepoints. This lightweight, event-driven approach covers most high-frequency hardware fault scenarios. In addition, HUATUO supports monitoring of PFC/RDMA congestion as well as physical link status of network interfaces.