Reference on HUATUO Documentation

Metrics Reference

Sun, 11 Jan 2026 00:00:00 +0000

This reference lists metric names, units, labels, sources, and measurement semantics. See Kernel-Wide Insight for capabilities and troubleshooting scenarios.

How to Read This Reference

Common metrics are grouped by CPU, memory, network, I/O, system events, and GPU for querying and troubleshooting. Actual output depends on kernel interfaces, device drivers, enabled collectors, and filters. The node’s /metrics output is authoritative for the complete set. See Node Configuration for enablement.

Names: tables omit the huatuo_bamai_ prefix. For example, cpu_util_total is exposed as huatuo_bamai_cpu_util_total.
Labels: all metrics include host and region. Container metrics also include container_host, container_hostnamespace, container_level, container_name, and container_type. Tables list only additional labels; — means none.
Counts and units: counts measure events or objects, not latency. Use rate() or increase() for cumulative Counters. Some cumulative kernel statistics are exported as Gauges; interpret their field semantics rather than inferring from type or name alone.
Examples: values illustrate output format, not diagnostic thresholds. A missing metric does not mean zero.

# TYPE huatuo_bamai_runqlat_latency counter
huatuo_bamai_runqlat_latency{host="hostname",region="dev",zone="0"} 35100
# TYPE huatuo_bamai_runqlat_container_latency counter
huatuo_bamai_runqlat_container_latency{container_host="coredns-855c4dd65d-8v5kg",container_hostnamespace="kube-system",container_level="burstable",container_name="coredns",container_type="normal",host="hostname",region="dev",zone="0"} 226

This example shows host and container scheduling-latency bucket counts. The zone ranges are listed below.

AutoTracing Fields

Sun, 11 Jan 2026 00:00:00 +0000

This reference covers AutoTracing record fields, units, and examples. See AutoTracing for use cases and Node Configuration for settings. Thresholds below are built-in defaults that configuration files can override.

Results

Automatic tracing events are written to Elasticsearch or local files according to enabled storage settings. Filter by tracer_type=autotracing, tracer_name, host or container identifiers, and time range. The JSON examples illustrate field structure, not measured results.

CPU examples use perf. With profiler, events retain trigger metrics; profiles are stored separately in Elasticsearch with the same tracer_id. See Node Configuration for storage requirements and viewing instructions.

Abnormal Event Fields

Sun, 11 Jan 2026 00:00:00 +0000

This reference covers abnormal event structures, fields, and limitations. See Abnormal Event Diagnosis for capabilities and Node Configuration for collection settings.

Event Records

Results are written to Elasticsearch or local files according to enabled storage settings. Filter by tracer_type=event, event name, host or container, and time range. The JSON examples illustrate structure and omit some common fields. Threshold descriptions use built-in defaults.

Container fields are provided only when a container is associated. Event-specific data is in tracer_data.

Hardware Event Fields

Fri, 06 Mar 2026 00:00:00 +0000

This reference covers RAS metrics, event structures, severity, and source fields. See Hardware Fault Diagnosis for coverage and enablement.

EventTracing.Ras.MceThrBackoff controls the save interval for threshold-interrupt events. Events are still counted during cooldown, so counter increases need not equal the number of saved records.

Results

Error Counts

huatuo_bamai_ras_hw_err_total counts received RAS events by type: mce, edac, acpi, aer, thr, and arm_ghes. The presence of a label does not prove that the current platform supports its probe.

Profiling API

Wed, 21 Oct 2026 00:00:00 +0000

This reference covers profiling task requests, responses, and states. For a first collection, see the Profiling Guide.

huatuo-apiserver creates, supervises, and stops profiling tasks through /v1/profiling. Each HTTP creation request is an independent execution. Profiling tasks are persisted as Jobs in huatuo-apiserver; the node keeps only in-memory operations and profiler processes.

Task control does not depend on Elasticsearch. Raw-result queries and dashboard links require it. huatuo-bamai and huatuo-apiserver must use the same index.

profiler CLI Parameters

Wed, 21 Oct 2026 00:00:00 +0000

This reference lists profiler parameters, defaults, and applicability. See CLI Collection for prerequisites and examples, and Output Formats to choose a format.

Both --output-path and --output-format must be explicit. The declared default for --output-format is collapsed, but validation does not allow the flag to be omitted.

General

Parameter	Default	Applies To	Description
`--type`, `-t`	None	All	Observation type: `cpu` or `memory`; required
`--language`, `-l`	None	All	Target language: `c`, `c++`, `go`, `java`, or `python`; required
`--pid`, `-p`	None	All	Target PID; Java and Python accept comma-separated PIDs, native collection accepts at most one
`--container-id`	None	All	Target container ID; native CPU combines it with `--pid` as an AND filter using the host PID; otherwise mutually exclusive with `--pid`
`--duration`, `-d`	`10`	All	Total collection duration in seconds, minimum 1
`--aggr-interval`	`10`	All	Aggregation interval in seconds; cannot exceed duration
`--freq`, `-F`	`99`	CPU	Samples per second; Java maximum: 1000
`--output-path`	Required	All	Local file, directory, or `stdout`; Unix socket for `remote-store`. End new directory paths with `/`
`--output-format`	`collapsed`	All	`collapsed`, `flamegraph`, `svg`, `json`, or `remote-store`; must be explicit together with `--output-path`
`--max-concurrent-procs`	`0`	Java, Python	Maximum concurrent collection child processes; `0` is unlimited
`--tool-path`	None	Java, Python	Shared root directory for external tools; required
`--binary-match-path`	None	Java, Python	Match target container processes by executable path
`--huatuo-api-address`	`127.0.0.1:19704`	Container targets	HUATUO API address for resolving container metadata
`--task-id`	Empty; generated internally for local output	All; required for `remote-store`	Stable collection task ID shared by toolstream and remote storage
`--enable-pprof`	`false`	The tool itself	Serve this process’s Go runtime profiles on `127.0.0.1:6000`
`--version-format`	`text`	Version queries	Format for `--version`: `text`, `json`, or `short`
`--help`, `-h`	-	All	Show command help
`--version`, `-v`	-	All	Show version and build information

These pprof endpoints describe the profiler process’s own Go runtime, not the --pid target or Java/Python collector subprocesses. The listener is now restricted to loopback rather than all interfaces; an occupied port fails startup.