Development on HUATUO Documentation

Extending Observability

Sat, 18 Jul 2026 00:00:00 +0000

HUATUO supports three extension types: Metrics, Event, and AutoTracing. They use the same registration framework but differ in activation, runtime cost, and data output.

Type	Activation	Data output	Use case
Metrics	Periodic collection	Prometheus	Continuous performance monitoring and long-term trends
Event	Kernel event or threshold	ES, local files, optional Prometheus	Continuous operation with anomaly context capture
AutoTracing	System anomaly	ES, local files, optional Prometheus	On-demand, higher-cost context capture

Collection Modes

Metrics

Metrics periodically collect system state through procfs, sysfs, or eBPF and expose it in Prometheus format. This mode supports real-time monitoring and long-term trend analysis. Built-in collectors cover:

Development Debugging

Sat, 18 Jul 2026 00:00:00 +0000

Full-Stack Integration

The development Compose configuration builds an image from the current workspace and starts the collector, API Server, Elasticsearch, Prometheus, and Grafana. The collector needs access to the host kernel and cgroups, so run the command with root privileges from the repository root on a Linux host:

sudo make compose-dev-up

Compose aggregates all component logs in the foreground. After changing the source, press Ctrl+C and run the command again. Docker reuses the toolchain layers and Go build cache.

Integration Test

Wed, 04 Mar 2026 00:00:00 +0000

This integration test validates that huatuo-bamai can start correctly with mocked /proc and /sys filesystems and expose the expected Prometheus metrics.

The test runs the real huatuo-bamai binary and verifies the /metricsendpoint output without relying on the host kernel or hardware.

What the Script Does

The integration test performs the following steps:

Generates a temporary bamai.conf
Starts huatuo-bamai with mocked procfs and sysfs
Waits for the Prometheus /metrics endpoint to become available
Fetches all metrics from /metrics
Verifies that all expected metrics exist
Stops the service and cleans up resources

If any expected metric is missing, the test fails.

BPF ABI Guide

Wed, 04 Mar 2026 00:00:00 +0000

The C structures on the BPF side are the source of truth for the perf event ABI. Go types are generated automatically from BTF in the BPF objects. Do not maintain equivalent structures manually.

Conventions

Each ABI domain maps to one C header and one generated Go file:

Item	Convention
Domain name	`<domain>`
C header	`bpf/include/abi/<domain>_types.h`
C structure prefix	`<domain>_`
Generated Go file	`internal/bpf/abi/<domain>_types_generated.go`

<domain> must start with a lowercase letter and contain only lowercase letters, digits, and underscores. Avoid domains with overlapping prefixes, such as net and net_rx.

Time Format Contract

Wed, 22 Jul 2026 00:00:00 +0000

Time Format Contract

HUATUO serializes timestamps in UTC with fixed nanosecond precision:

2006-01-02T15:04:05.000000000Z

For example, an event captured at half past midnight on 22 July 2026 is written as:

2026-07-22T00:30:00.123456789Z

Why fixed UTC precision

Every host emits the same timezone (Z), so distributed queries do not need timezone-specific handling.
The nine fractional digits keep the string width fixed. Lexical ordering therefore matches chronological ordering, which is useful for log files and keyword-indexed storage.
Nanoseconds preserve the precision supplied by Go’s time.Time and by event-tracing pipelines.

Integration guidance

When producing data for HUATUO, format times with the canonical helper: