Documentation on HUATUO Documentation

Collection Framework

Sun, 11 Jan 2026 00:00:00 +0000

HuaTuo framework provides three data collection modes: autotracing, event, and metrics, covering different monitoring scenarios, helping users gain comprehensive insights into system performance.

Collection Mode Comparison

Mode	Type	Trigger Condition	Data Output	Use Case
Autotracing	Event-driven	Triggered on system anomalies	ES + Local Storage, Prometheus (optional)	Non-routine operations, triggered on anomalies
Event	Event-driven	Continuously running, triggered on preset thresholds	ES + Local Storage, Prometheus (optional)	Continuous operations, directly dump context
Metrics	Metric collection	Passive collection	Prometheus format	Monitoring system metrics

Autotracing

Type: Event-driven (tracing).
Function: Automatically tracks system anomalies and dump context when anomalies occur.
Features:
- When a system anomaly occurs, autotracing is triggered automatically to dump relevant context.
- Data is stored to ES in real-time and stored locally for subsequent analysis and troubleshooting. It can also be monitored in Prometheus format for statistics and alerts.
- Suitable for scenarios with high performance overhead, such as triggering captures when metrics exceed a threshold or rise too quickly.
Integrated Features: CPU anomaly tracking (cpu idle), D-state tracking (dload), container contention (waitrate), memory burst allocation (memburst), disk anomaly tracking (iotracer).

Event

Type: Event-driven (tracing).
Function: Continuously operates within the system context, directly dump context when preset thresholds are met.
Features:
- Unlike autotracing, event continuously operates within the system context, rather than being triggered by anomalies.
- Data is also stored to ES and locally, and can be monitored in Prometheus format.
- Suitable for continuous monitoring and real-time analysis, enabling timely detection of abnormal behaviors. The performance impact of event collection is negligible.
Integrated Features: Soft interrupt anomalies (softirq), memory allocation anomalies (oom), soft lockups (softlockup), D-state processes (hungtask), memory reclamation (memreclaim), packet droped abnormal (dropwatch), network ingress latency (net_rx_latency).

Metrics

Type: Metric collection.
Function: Collects performance metrics from subsystems.
Features:
- Metric data can be sourced from regular procfs collection or derived from tracing (autotracing, event) data.
- Outputs in Prometheus format for easy integration into Prometheus monitoring systems.
- Unlike tracing data, metrics primarily focus on system performance metrics such as CPU usage, memory usage, and network traffic, etc.
- Suitable for monitoring system performance metrics, supporting real-time analysis and long-term trend observation.
Integrated Features: CPU (sys, usr, util, load, nr_running, etc.), memory (vmstat, memory_stat, directreclaim, asyncreclaim, etc.), IO (d2c, q2c, freeze, flush, etc.), network (arp, socket mem, qdisc, netstat, netdev, sockstat, etc.).

Multiple Purpose of Tracing Mode

Both autotracing and event belong to the tracing collection mode, offering the following dual purposes:

Docker

Sun, 11 Jan 2026 00:00:00 +0000

Run Only the Collector

Start the Container

docker run --privileged --cgroupns=host --network=host -v /sys:/sys -v /proc:/proc -v /run:/run huatuo/huatuo-bamai:latest

⚠️ This uses the default configuration file inside the container. The internal default configuration does not connect to Elasticsearch. For a complete setup, mount your own huatuo-bamai.conf using -v, and update the config according to your environment (kubelet access, Elasticsearch settings, local log storage path, etc.).

Docker

Sun, 11 Jan 2026 00:00:00 +0000

Run Only the Collector

Start the Container

docker run --privileged --cgroupns=host --network=host -v /sys:/sys -v /proc:/proc -v /run:/run huatuo/huatuo-bamai:latest

⚠️ This uses the default configuration file inside the container. The internal default configuration does not connect to Elasticsearch. For a complete setup, mount your own huatuo-bamai.conf using -v, and update the config according to your environment (kubelet access, Elasticsearch settings, local log storage path, etc.).

Framework

Sun, 11 Jan 2026 00:00:00 +0000

Collection Mode Comparison

Mode	Type	Trigger Condition	Data Output	Use Case
Autotracing	Event-driven	Triggered on system anomalies	ES + Local Storage, Prometheus (optional)	Non-routine operations, triggered on anomalies
Event	Event-driven	Continuously running, triggered on preset thresholds	ES + Local Storage, Prometheus (optional)	Continuous operations, directly dump context
Metrics	Metric collection	Passive collection	Prometheus format	Monitoring system metrics

Autotracing

Type: Event-driven (tracing).
Function: Automatically tracks system anomalies and dump context when anomalies occur.
Features:
- When a system anomaly occurs, autotracing is triggered automatically to dump relevant context.
- Data is stored to ES in real-time and stored locally for subsequent analysis and troubleshooting. It can also be monitored in Prometheus format for statistics and alerts.
- Suitable for scenarios with high performance overhead, such as triggering captures when metrics exceed a threshold or rise too quickly.
Integrated Features: CPU anomaly tracking (cpu idle), D-state tracking (dload), container contention (waitrate), memory burst allocation (memburst), disk anomaly tracking (iotracer).

Event

Type: Event-driven (tracing).
Function: Continuously operates within the system context, directly dump context when preset thresholds are met.
Features:
- Unlike autotracing, event continuously operates within the system context, rather than being triggered by anomalies.
- Data is also stored to ES and locally, and can be monitored in Prometheus format.
- Suitable for continuous monitoring and real-time analysis, enabling timely detection of abnormal behaviors. The performance impact of event collection is negligible.
Integrated Features: Soft interrupt anomalies (softirq), memory allocation anomalies (oom), soft lockups (softlockup), D-state processes (hungtask), memory reclamation (memreclaim), packet droped abnormal (dropwatch), network ingress latency (net_rx_latency).

Metrics

Type: Metric collection.
Function: Collects performance metrics from subsystems.
Features:
- Metric data can be sourced from regular procfs collection or derived from tracing (autotracing, event) data.
- Outputs in Prometheus format for easy integration into Prometheus monitoring systems.
- Unlike tracing data, metrics primarily focus on system performance metrics such as CPU usage, memory usage, and network traffic, etc.
- Suitable for monitoring system performance metrics, supporting real-time analysis and long-term trend observation.
Integrated Features: CPU (sys, usr, util, load, nr_running, etc.), memory (vmstat, memory_stat, directreclaim, asyncreclaim, etc.), IO (d2c, q2c, freeze, flush, etc.), network (arp, socket mem, qdisc, netstat, netdev, sockstat, etc.).

Multiple Purpose of Tracing Mode

Both autotracing and event belong to the tracing collection mode, offering the following dual purposes:

Getting started

Sun, 11 Jan 2026 00:00:00 +0000

To help users quickly experience and deploy HUATUO, this document is divided into three sections: Quick Experience，Quick Start，Compilation & Deployment.

1. Quick Experience

This section helps you quickly explore the frontend capabilities. You can directly access demo station, such as viewing exception event overviews, exception event context information, metric curves, etc. (Account: huatuo passwd: huatuo1024).

Getting started

Sun, 11 Jan 2026 00:00:00 +0000

To help users quickly experience and deploy HUATUO, this document is divided into three sections: Quick Experience，Quick Start，Compilation & Deployment.

1. Quick Experience

Metrics

Sun, 11 Jan 2026 00:00:00 +0000

Metrics supported in the current version:

CPU

Scheduling

The following metrics allow observation of process scheduling latency, i.e., the time from when a process becomes runnable (placed in the run queue) until it actually starts executing on the CPU.

# HELP huatuo_bamai_runqlat_container_latency cpu run queue latency for the containers
# TYPE huatuo_bamai_runqlat_container_latency gauge
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="3"} 0

# HELP huatuo_bamai_runqlat_latency cpu run queue latency for the host
# TYPE huatuo_bamai_runqlat_latency gauge
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="3"} 0

Metric	Description	Unit	Target	Source	Labels
runqlat_container_latency	scheduling latency histogram buckets: zone0: 0–10 ms zone1: 10–20 ms zone2: 20–50 ms zone3: 50+ ms	count	Container	eBPF	container_host, container_hostnamespace, container_level, container_name, container_type, host, region, zone
runqlat_latency	scheduling latency histogram buckets: zone0, 0~10ms zone1, 10-20ms zone2, 20-50ms zone3, 50+ms	count	Host	eBPF	host, region, zone

SoftIRQ

SoftIRQ response latency on different CPUs (currently only NET_RX and NET_TX are collected).

Metrics

Sun, 11 Jan 2026 00:00:00 +0000

Metrics supported in the current version:

CPU

Scheduling

The following metrics allow observation of process scheduling latency, i.e., the time from when a process becomes runnable (placed in the run queue) until it actually starts executing on the CPU.

# HELP huatuo_bamai_runqlat_container_latency cpu run queue latency for the containers
# TYPE huatuo_bamai_runqlat_container_latency gauge
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="3"} 0

# HELP huatuo_bamai_runqlat_latency cpu run queue latency for the host
# TYPE huatuo_bamai_runqlat_latency gauge
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="1"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="2"} 0
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="3"} 0

Metric	Description	Unit	Target	Source	Labels
runqlat_container_latency	scheduling latency histogram buckets: zone0: 0–10 ms zone1: 10–20 ms zone2: 20–50 ms zone3: 50+ ms	count	Container	eBPF	container_host, container_hostnamespace, container_level, container_name, container_type, host, region, zone
runqlat_latency	scheduling latency histogram buckets: zone0, 0~10ms zone1, 10-20ms zone2, 20-50ms zone3, 50+ms	count	Host	eBPF	host, region, zone

SoftIRQ

SoftIRQ response latency on different CPUs (currently only NET_RX and NET_TX are collected).

Add Metrics

Sun, 11 Jan 2026 00:00:00 +0000

Overview

The Metrics type is used to collect system performance and other indicator data. It can output in Prometheus format, serving as a data provider through the /metrics (curl localhost:<port>/metrics) .

Type：Metrics collection
Function：Collects performance metrics from various subsystems
Characteristics：
- Metrics are primarily used to collect system performance metrics such as CPU usage, memory usage, network statistics, etc. They are suitable for monitoring system performance and support real-time analysis and long-term trend observation.
- Metrics can come from regular procfs/sysfs collection or be generated from tracing types (autotracing, event).
- Outputs in Prometheus format for seamless integration into the Prometheus observability ecosystem.
Already Integrated：

Add Metrics

Sun, 11 Jan 2026 00:00:00 +0000

Overview

Type：Metrics collection
Function：Collects performance metrics from various subsystems
Characteristics：
- Metrics are primarily used to collect system performance metrics such as CPU usage, memory usage, network statistics, etc. They are suitable for monitoring system performance and support real-time analysis and long-term trend observation.
- Metrics can come from regular procfs/sysfs collection or be generated from tracing types (autotracing, event).
- Outputs in Prometheus format for seamless integration into the Prometheus observability ecosystem.
Already Integrated：

Daemonset

Sun, 11 Jan 2026 00:00:00 +0000

HUATUO provides the simplest DaemonSet deployment option to minimize setup complexity. Deploying the HUATUO collector via DaemonSet involves the following steps:

1. Download the Collector Configuration File

curl -L -o huatuo-bamai.conf https://github.com/ccfos/huatuo/raw/main/huatuo-bamai.conf

Modify this configuration file according to your environment, such as kubelet connection settings and Elasticsearch settings.

2. Create a ConfigMap

kubectl create configmap huatuo-bamai-config --from-file=./huatuo-bamai.conf

3. Deploy the Collector

kubectl apply -f huatuo-daemonset.minimal.yaml

Contents of huatuo-daemonset.minimal.yaml:

Daemonset

Sun, 11 Jan 2026 00:00:00 +0000

HUATUO provides the simplest DaemonSet deployment option to minimize setup complexity. Deploying the HUATUO collector via DaemonSet involves the following steps:

1. Download the Collector Configuration File

curl -L -o huatuo-bamai.conf https://github.com/ccfos/huatuo/raw/main/huatuo-bamai.conf

Modify this configuration file according to your environment, such as kubelet connection settings and Elasticsearch settings.

2. Create a ConfigMap

kubectl create configmap huatuo-bamai-config --from-file=./huatuo-bamai.conf

3. Deploy the Collector

kubectl apply -f huatuo-daemonset.minimal.yaml

Contents of huatuo-daemonset.minimal.yaml:

Events

Sun, 11 Jan 2026 00:00:00 +0000

The HUATUO platform uses eBPF technology to detect various abnormal events in the Linux kernel in real time, helping users quickly locate issues related to the system, applications, and hardware.

Supported Events

Event Name	Core Function	Typical Scenarios
softirq	Detects excessively long softirq disable time in the kernel, outputs call stack and process information	Resolves system stalls, network latency, and scheduling delays
softlockup	Detects softlockup events and provides target process and kernel stack information	Locates and resolves system softlockup issues
hungtask	Detects hungtask events, outputs all D-state processes and their stack information	Captures transient mass D-state process scenarios and preserves fault scenes
oom	Detects OOM events in the host or containers	Focuses on memory exhaustion issues and provides detailed fault snapshots
memory_reclaim_events	Detects direct memory reclaim events, records reclaim duration, process and container information	Resolves business stalls caused by memory pressure
ras	Detects hardware faults in CPU, Memory, PCIe, etc.	Timely awareness of hardware failures to reduce business impact
dropwatch	Detects packet drops in the kernel network protocol stack, outputs call stack and network context	Resolves business jitters and latency caused by protocol stack packet drops
net_rx_latency	Detects latency events in the protocol stack receive path (driver → protocol → user space)	Resolves business timeouts and jitters caused by receive latency
netdev_events	Detects network device link status changes	Detects physical link failures on network cards
netdev_bonding_lacp	Detects bonding LACP protocol status changes	Identifies fault boundaries between physical machines and switches
netdev_txqueue_timeout	Detects network card transmit queue timeout events	Locates hardware failures in network card transmit queues

Event Details

Common Fields

hostname: Physical machine hostname
region: Availability zone where the physical machine is located
uploaded_time: Data upload time
container_id: Container ID if the event is associated with a container
container_hostname: Container hostname if the event is associated with a container
container_host_namespace: Kubernetes namespace of the container if the event is associated with a container
container_type: Container type, e.g., normal for regular containers, sidecar for sidecar containers, etc.
container_qos: Container QoS level
tracer_name: Event name
tracer_id: Tracing ID for this event
tracer_time: Time when tracing was triggered
tracer_type: Trigger type — manual or automatic
tracer_data: Tracer-specific private data

1. softirq

Description
Detects when the kernel disables interrupts for too long. Records the kernel call stack during the disable period, current process information, and other key data to help analyze interrupt-related latency issues.

Events

Sun, 11 Jan 2026 00:00:00 +0000

The HUATUO platform uses eBPF technology to detect various abnormal events in the Linux kernel in real time, helping users quickly locate issues related to the system, applications, and hardware.

Supported Events

Event Name	Core Function	Typical Scenarios
softirq	Detects excessively long softirq disable time in the kernel, outputs call stack and process information	Resolves system stalls, network latency, and scheduling delays
softlockup	Detects softlockup events and provides target process and kernel stack information	Locates and resolves system softlockup issues
hungtask	Detects hungtask events, outputs all D-state processes and their stack information	Captures transient mass D-state process scenarios and preserves fault scenes
oom	Detects OOM events in the host or containers	Focuses on memory exhaustion issues and provides detailed fault snapshots
memory_reclaim_events	Detects direct memory reclaim events, records reclaim duration, process and container information	Resolves business stalls caused by memory pressure
ras	Detects hardware faults in CPU, Memory, PCIe, etc.	Timely awareness of hardware failures to reduce business impact
dropwatch	Detects packet drops in the kernel network protocol stack, outputs call stack and network context	Resolves business jitters and latency caused by protocol stack packet drops
net_rx_latency	Detects latency events in the protocol stack receive path (driver → protocol → user space)	Resolves business timeouts and jitters caused by receive latency
netdev_events	Detects network device link status changes	Detects physical link failures on network cards
netdev_bonding_lacp	Detects bonding LACP protocol status changes	Identifies fault boundaries between physical machines and switches
netdev_txqueue_timeout	Detects network card transmit queue timeout events	Locates hardware failures in network card transmit queues

Event Details

Common Fields

hostname: Physical machine hostname
region: Availability zone where the physical machine is located
uploaded_time: Data upload time
container_id: Container ID if the event is associated with a container
container_hostname: Container hostname if the event is associated with a container
container_host_namespace: Kubernetes namespace of the container if the event is associated with a container
container_type: Container type, e.g., normal for regular containers, sidecar for sidecar containers, etc.
container_qos: Container QoS level
tracer_name: Event name
tracer_id: Tracing ID for this event
tracer_time: Time when tracing was triggered
tracer_type: Trigger type — manual or automatic
tracer_data: Tracer-specific private data

1. softirq

Add Event

Sun, 11 Jan 2026 00:00:00 +0000

Overview

Type: Exception event-driven（tracing/event）
Function：Continuously runs in the system and captures context information when preset thresholds are reached
Characteristics:
- Unlike autotracing, event runs continuously rather than being triggered only when exceptions occur.
- Event data is stored locally in real-time and also sent to remote ES. You can also generate Prometheus metrics for observation.
- Suitable for continuous monitoring and real-time analysis, enabling timely detection of abnormal behaviors in the system. The performance impact of event type collection is negligible.
Already Integrated: Soft interrupt abnormalities（softirq）、abnormal memory allocation（oom）、soft lockups（softlockup）、D-state processes（hungtask）、memory reclaim（memreclaim）、abnormal packet loss（dropwatch）、network inbound latency (net_rx_latency), etc.

How to Add Event Metrics

Simply implement the ITracingEvent interface and complete registration to add events to the system.

Add Event

Sun, 11 Jan 2026 00:00:00 +0000

Overview

Type: Exception event-driven（tracing/event）
Function：Continuously runs in the system and captures context information when preset thresholds are reached
Characteristics:
- Unlike autotracing, event runs continuously rather than being triggered only when exceptions occur.
- Event data is stored locally in real-time and also sent to remote ES. You can also generate Prometheus metrics for observation.
- Suitable for continuous monitoring and real-time analysis, enabling timely detection of abnormal behaviors in the system. The performance impact of event type collection is negligible.
Already Integrated: Soft interrupt abnormalities（softirq）、abnormal memory allocation（oom）、soft lockups（softlockup）、D-state processes（hungtask）、memory reclaim（memreclaim）、abnormal packet loss（dropwatch）、network inbound latency (net_rx_latency), etc.

How to Add Event Metrics

Simply implement the ITracingEvent interface and complete registration to add events to the system.

Autotracing

Sun, 11 Jan 2026 00:00:00 +0000

Overview

AutoTracing is an intelligent diagnostic feature of the Huatuo kernel monitoring system.

When the system experiences specific performance anomalies or sudden resource spikes, AutoTracing is automatically triggered. It captures detailed on-site information in real time, including flame graphs, process context, call stacks, and resource status. This helps operations and development teams quickly locate and analyze issues without manual intervention.

This feature is built on eBPF technology, offering low overhead and high real-time performance. It is suitable for anomaly diagnosis in both physical machines and container environments.

Autotracing

Sun, 11 Jan 2026 00:00:00 +0000

Overview

AutoTracing is an intelligent diagnostic feature of the Huatuo kernel monitoring system.

This feature is built on eBPF technology, offering low overhead and high real-time performance. It is suitable for anomaly diagnosis in both physical machines and container environments.

Compile

Sun, 11 Jan 2026 00:00:00 +0000

1. Build with the Official Image

To isolate the developer’s local environment and simplify the build process, we provide a containerized build method. You can directly use docker build to produce an image containing the core collector huatuo-bamai, BPF objects, tools, and more. Run the following in the project root directory:

docker build --network host -t huatuo/huatuo-bamai:latest .

2. Build a Custom Image

Dockerfile.dev:

FROM golang:1.23.0-alpine AS base
# Speed up Alpine package installation if needed
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk add --no-cache \
                make \
                clang15 \
                libbpf-dev \
                bpftool \
                curl \
                git

ENV PATH=$PATH:/usr/lib/llvm15/bin

# build huatuo components
FROM base AS build
ARG BUILD_PATH=${BUILD_PATH:-/go/huatuo-bamai}
ARG RUN_PATH=${RUN_PATH:-/home/huatuo-bamai}
WORKDIR ${BUILD_PATH}

2.1 Build the Dev Image

docker build --network host -t huatuo/huatuo-bamai-dev:latest -f ./Dockerfile.dev .

2.2 Run the Dev Container

docker run -it --privileged --cgroupns=host --network=host \
  -v /path/to/huatuo:/go/huatuo-bamai \
  huatuo/huatuo-bamai-dev:latest sh

2.3 Compile Inside the Container

Run:

Compile

Sun, 11 Jan 2026 00:00:00 +0000

1. Build with the Official Image

docker build --network host -t huatuo/huatuo-bamai:latest .

2. Build a Custom Image

Dockerfile.dev:

FROM golang:1.23.0-alpine AS base
# Speed up Alpine package installation if needed
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk add --no-cache \
                make \
                clang15 \
                libbpf-dev \
                bpftool \
                curl \
                git

ENV PATH=$PATH:/usr/lib/llvm15/bin

# build huatuo components
FROM base AS build
ARG BUILD_PATH=${BUILD_PATH:-/go/huatuo-bamai}
ARG RUN_PATH=${RUN_PATH:-/home/huatuo-bamai}
WORKDIR ${BUILD_PATH}

2.1 Build the Dev Image

docker build --network host -t huatuo/huatuo-bamai-dev:latest -f ./Dockerfile.dev .

2.2 Run the Dev Container

docker run -it --privileged --cgroupns=host --network=host \
  -v /path/to/huatuo:/go/huatuo-bamai \
  huatuo/huatuo-bamai-dev:latest sh

2.3 Compile Inside the Container

Run:

Systemd

Sun, 11 Jan 2026 00:00:00 +0000

Install by RPM

OpenCloudOS currently provides the v2.1.0 RPM package; the master is for reference only.

Tencent OpenCloudOS provides an official HUATUO package:
https://mirrors.opencloudos.tech/epol/9/Everything/x86_64/os/Packages/huatuo-bamai-2.1.0-2.oc9.x86_64.rpm
This allows HUATUO to be quickly installed and enabled on OpenCloudOS.

x86_64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/x86_64/os/Packages/huatuo-bamai-2.1.0-2.oc9.x86_64.rpm

arm64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/aarch64/os/Packages/huatuo-bamai-2.1.0-2.oc9.aarch64.rpm

Install HUATUO on OC8

sudo rpm -ivh huatuo-bamai*.rpm

Other RPM-based operating systems can install HUATUO the same way.
As usual, you must update the config file according to your environment (e.g., kubelet connection, Elasticsearch settings).

Systemd

Sun, 11 Jan 2026 00:00:00 +0000

Install by RPM

OpenCloudOS currently provides the v2.1.0 RPM package; the master is for reference only.

x86_64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/x86_64/os/Packages/huatuo-bamai-2.1.0-2.oc9.x86_64.rpm

arm64 architecture

wget https://mirrors.opencloudos.tech/epol/9/Everything/aarch64/os/Packages/huatuo-bamai-2.1.0-2.oc9.aarch64.rpm

Install HUATUO on OC8

sudo rpm -ivh huatuo-bamai*.rpm

Other RPM-based operating systems can install HUATUO the same way.
As usual, you must update the config file according to your environment (e.g., kubelet connection, Elasticsearch settings).

Configuration Guide

Sun, 29 Mar 2026 00:00:00 +0000

1. Document Overview

huatuo-bamai is the core collector of HUATUO (a BPF-based metrics and anomaly inspector). Its configuration file defines the data collection scope, probe enablement strategy, metric output format, anomaly detection rules, and logging behavior.

The configuration file uses TOML format and includes multiple sections such as global blacklist, logging, runtime resource limits, storage configuration, and AutoTracing. Each configuration item comes with detailed comments explaining its purpose, default value, and important notes. This document provides a clear and detailed English explanation for every configuration item to help users understand and safely customize the settings.

Configuration Guide

Sun, 29 Mar 2026 00:00:00 +0000

1. Document Overview

Hardware Errors

Fri, 06 Mar 2026 00:00:00 +0000

Architecture

The huatuo supports detection of various hardware faults, including:

CPU, L1/L2/L3 Cache, TLB
Memory, ECC
PCIe
Network Interface Card Link
PFC / RDMA
ACPI
GPU MetaX

Overall Architecture of HUATUO

The huatuo is built on Linux kernel MCE (Machine Check Exception) and RAS (Reliability, Availability, and Serviceability) mechanisms. It uses eBPF to capture critical hardware events and retrieve device information. The Linux kernel RAS framework has been continuously evolving since kernel 2.6, gradually adding more tracepoints. This lightweight, event-driven approach covers most high-frequency hardware fault scenarios. In addition, HUATUO supports monitoring of PFC/RDMA congestion as well as physical link status of network interfaces.

Hardware Errors

Fri, 06 Mar 2026 00:00:00 +0000

Architecture

The huatuo supports detection of various hardware faults, including:

CPU, L1/L2/L3 Cache, TLB
Memory, ECC
PCIe
Network Interface Card Link
PFC / RDMA
ACPI
GPU MetaX

Overall Architecture of HUATUO

Add Autotracing

Sun, 11 Jan 2026 00:00:00 +0000

Overview

Type：Exception event-driven（tracing/autotracing）
Function：Automatically tracks system abnormal states and triggers context information capture when exceptions occur
Characteristics：
- When system abnormalities occur, autotracing automatically triggers and captures relevant context information
- Event data is stored locally in real-time and also sent to remote ES, while you can also generate Prometheus metrics for observation
- Suitable for significant performance overhead， such as triggering capture when detecting metrics rising above certain thresholds or rising too rapidly
Already Integrated：abnormal usage tracking (cpu idle), D-state tracking (dload), container internal/external contention (waitrate), sudden memory allocation (memburst), disk abnormal tracking (iotracer)

How to Add Autotracing

AutoTracing only requires implementing the ITracingEvent interface and completing registration to add events to the system.

Add Autotracing

Sun, 11 Jan 2026 00:00:00 +0000

Overview

Type：Exception event-driven（tracing/autotracing）
Function：Automatically tracks system abnormal states and triggers context information capture when exceptions occur
Characteristics：
- When system abnormalities occur, autotracing automatically triggers and captures relevant context information
- Event data is stored locally in real-time and also sent to remote ES, while you can also generate Prometheus metrics for observation
- Suitable for significant performance overhead， such as triggering capture when detecting metrics rising above certain thresholds or rising too rapidly
Already Integrated：abnormal usage tracking (cpu idle), D-state tracking (dload), container internal/external contention (waitrate), sudden memory allocation (memburst), disk abnormal tracking (iotracer)

How to Add Autotracing

AutoTracing only requires implementing the ITracingEvent interface and completing registration to add events to the system.

Integration Test

Wed, 04 Mar 2026 00:00:00 +0000

This integration test validates that huatuo-bamai can start correctly with mocked /proc and /sys filesystems and expose the expected Prometheus metrics.

The test runs the real huatuo-bamai binary and verifies the /metricsendpoint output without relying on the host kernel or hardware.

What the Script Does

The integration test performs the following steps:

Generates a temporary bamai.conf
Starts huatuo-bamai with mocked procfs and sysfs
Waits for the Prometheus /metrics endpoint to become available
Fetches all metrics from /metrics
Verifies that all expected metrics exist
Stops the service and cleans up resources

If any expected metric is missing, the test fails.

Integration Test

Wed, 04 Mar 2026 00:00:00 +0000

This integration test validates that huatuo-bamai can start correctly with mocked /proc and /sys filesystems and expose the expected Prometheus metrics.

The test runs the real huatuo-bamai binary and verifies the /metricsendpoint output without relying on the host kernel or hardware.

What the Script Does

The integration test performs the following steps:

Generates a temporary bamai.conf
Starts huatuo-bamai with mocked procfs and sysfs
Waits for the Prometheus /metrics endpoint to become available
Fetches all metrics from /metrics
Verifies that all expected metrics exist
Stops the service and cleans up resources

If any expected metric is missing, the test fails.

Change Log

Sun, 29 Mar 2026 00:00:00 +0000

Change Log

Sun, 29 Mar 2026 00:00:00 +0000

Features

Added iotracing AutoTracing capability
Added general hardware failure detection (CPU, Memory, PCIe)
Added MetaX GPU failure detection
Added physical link detection support
Added support for Amazon EKS deployment
Added support for Aliyun ACK deployment
Added dropwatch namespace cookie support
Added container throttled_time metric
Added compatibility with kubelet systemd cgroup driver
Added automatic detection of kubelet cgroup driver type
Added, optimized, and standardized the huatuo-bamai configuration file
Added Github CI/CD automated testing
Added unit testing, integration testing, and end-to-end testing
Enhanced golangci-lint static code analysis rules
Added daemonset YAML deployment files
Added new Metric API interfaces
Added compatibility support for 5.15.x kernel series

Bug Fixes & Improvements

Optimized local storage format
Optimized and refactored code in multiple modules
Improved and enriched documentation at https://huatuo.tech/docs/