Development

• 1: Extending Observability
• 2: Development Debugging
• 3: Integration Test
• 4: BPF ABI Guide
• 5: Time Format Contract

1 - Extending Observability

HUATUO supports three extension types: Metrics, Event, and AutoTracing. They use the same registration framework but differ in activation, runtime cost, and data output.

Collection Modes

Metrics

Metrics periodically collect system state through procfs, sysfs, or eBPF and expose it in Prometheus format. This mode supports real-time monitoring and long-term trend analysis. Built-in collectors cover:

• CPU: sys, usr, util, load, nr_running, and related metrics.
• Memory: vmstat, memory_stat, directreclaim, and asyncreclaim.
• I/O: d2c, q2c, freeze, and flush.
• Networking: ARP, socket memory, qdisc, netstat, netdev, and sockstat.

Event

Events continuously observe kernel events or threshold conditions and preserve kernel context when an anomaly occurs. This mode is intended for low-overhead, always-on observation. Data is written to Elasticsearch and local files and can also produce Prometheus metrics. Built-in events include:

• Soft interrupt anomalies (softirq_tracing).
• Abnormal memory allocation (oom).
• Soft lockups (softlockup).
• D-state processes (hungtask).
• Memory reclaim (memory_reclaim_events).
• Packet drops (dropwatch).
• Network receive latency (net_rx_latency).

AutoTracing

AutoTracing invokes diagnostic tools after detecting a system anomaly. It is intended for flame graphs, context snapshots, and other diagnostic operations that are too expensive to run continuously. Results are written to Elasticsearch and local files and can also be converted into Prometheus metrics. Built-in capabilities include:

• CPU idle and system-time anomaly tracing (cpuidle, cpusys).
• D-state load tracing (dload).
• Burst memory allocation tracing (memburst).
• Disk I/O anomaly tracing (iotracing).

Event and AutoTracing are both Tracing modes and share the ITracingEvent interface. They can preserve anomaly context for root-cause analysis and expose statistics to Prometheus by also implementing Collector.

Adding Metrics

Custom Metrics collectors expose Prometheus metrics through /metrics.

Implement Collector

Create a type under core/metrics that implements Collector:

type Collector interface {
    Update() ([]*Data, error)
}

type exampleMetric struct{}

func (c *exampleMetric) Update() ([]*metric.Data, error) {
    return []*metric.Data{
        metric.NewGaugeData("example", value, "example value", nil),
    }, nil
}

Register the collector

Use FlagMetric when registering the implementation:

func init() {
    tracing.RegisterEventTracing("example", newExampleMetric)
}

func newExampleMetric() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleMetric{},
        Flag:        tracing.FlagMetric,
    }, nil
}

Manage BPF object

When one implementation provides both Start and Update, the methods may run concurrently. Do not read and write a bpf.BPF interface directly in a collector field. Use Reference and Lease from internal/bpf/bpf_ref.go to manage the object lifetime:

type example struct {
    object bpf.Reference
}

func (c *example) Start(ctx context.Context) (retErr error) {
    object, err := bpf.LoadBpf(bpf.ThisBpfOBJ(), nil)
    if err != nil {
        return err
    }

    if err := object.Attach(); err != nil {
        return errors.Join(err, object.Close())
    }
    if err := c.object.Publish(object); err != nil {
        return errors.Join(err, object.Close())
    }
    defer func() {
        retErr = errors.Join(retErr, c.object.UnPublish())
    }()

    <-ctx.Done()
    return nil
}

func (c *example) Update() ([]*metric.Data, error) {
    lease, ok := c.object.Acquire()
    if !ok {
        return nil, nil
    }
    defer lease.Release()

    items, err := lease.DumpMapByName("example_map")
    if err != nil {
        return nil, fmt.Errorf("dump example_map: %w", err)
    }

    return buildMetrics(items), nil
}

The API has these constraints:

• Publish transfers ownership of the object to Reference. Do not call object.Close() directly after a successful publish.
• The Lease returned by Acquire pins the BPF object until the current Update completes. Always pair it with Release, and do not copy a Lease.
• UnPublish first prevents new acquisitions, then waits for every Lease to be released, closes the BPF object, and returns the close error.
• Calls to Publish and UnPublish must be serialized. The current framework does not run Start concurrently for the same instance.

An Update that has already started can therefore finish with its original BPF object. During shutdown or restart, Start closes that object only after those updates complete.

Adding an Event

An Event implements ITracingEvent:

type ITracingEvent interface {
    Start(ctx context.Context) error
}

type exampleEvent struct{}

func (e *exampleEvent) Start(ctx context.Context) error {
    // Detect the event and capture its context.
    // storage.Save writes the data to ES and local storage.
    storage.Save("example", containerID, time.Now(), eventData)
    return nil
}

Register it with FlagTracing:

func init() {
    tracing.RegisterEventTracing("example", newExampleEvent)
}

func newExampleEvent() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleEvent{},
        Interval:    10,
        Flag:        tracing.FlagTracing,
    }, nil
}

To expose Prometheus metrics for the event, also implement Collector and add tracing.FlagMetric to Flag.

Adding AutoTracing

AutoTracing and Event use the same ITracingEvent interface and registration framework:

type exampleAutoTracing struct{}

func (t *exampleAutoTracing) Start(ctx context.Context) error {
    // Capture context after the anomaly trigger fires.
    storage.Save("example", containerID, time.Now(), tracingData)
    return nil
}

func init() {
    tracing.RegisterEventTracing("example", newExampleAutoTracing)
}

func newExampleAutoTracing() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleAutoTracing{},
        Interval:    10,
        Flag:        tracing.FlagTracing,
    }, nil
}

See core/metrics, core/events, and core/autotracing for complete examples covering BPF map interaction, container metadata, storage, and Prometheus output.

2 - Development Debugging

Full-Stack Integration

The development Compose configuration builds an image from the current workspace and starts the collector, API Server, Elasticsearch, Prometheus, and Grafana. The collector needs access to the host kernel and cgroups, so run the command with root privileges from the repository root on a Linux host:

sudo make compose-dev-up

Compose aggregates all component logs in the foreground. After changing the source, press Ctrl+C and run the command again. Docker reuses the toolchain layers and Go build cache.

Remove the containers, data volumes, and development image after debugging:

sudo make compose-dev-down

This command removes the Elasticsearch data volume. Do not run it when the integration data must be retained.

BPF Debugging

BPF code can use the bpf_dbg() and bpf_dbg_msg() macros to emit debug information from kernel space. The macros are defined in bpf/include/bpf_dbg.h. Debugging has separate build-time and runtime switches and is completely disabled by default.

Add debug trace points

Each BPF source file that uses the macros must declare its own debug map:

#include "bpf_dbg.h"

BPF_DBG_MAP(native_cpu);

SEC("perf_event")
int prog(void *ctx)
{
        bpf_dbg_msg(ctx, native_cpu, "enter prog");
        bpf_dbg(ctx, native_cpu, "pid and addr", pid, addr, 0);
        return 0;
}

bpf_dbg_msg() emits a message only. bpf_dbg() also accepts up to three u64 arguments.

Build debug objects

Set BPF_DEBUG=1 to pass -DDEBUG_BPF to Clang:

make BPF_DEBUG=1

To rebuild only the BPF objects:

make BPF_DEBUG=1 bpf-build

BPF_DEBUG=0 is the default. In that mode the macros expand to no-ops, and the debug perf event array, event structure, bpf_ktime_get_ns, and bpf_perf_event_output are not emitted into the BPF object.

Enable runtime output

After building the debug objects, pass --log-bpf-debug when starting the profiler. The option currently applies only to the native profiler:

./profiler --type cpu --language native --log-bpf-debug ...

When loading the BPF object, bpf.NewDbg(true) rewrites the bpf_dbg_enabled constant to 1 before LoadBpf. When it is disabled, the verifier eliminates the branch as dead code. Each BPF object maintains an independent switch.

Read debug output

User space emits each debug event at Debug level with these fields:

• file: BPF source file.
• line: source line number.
• ts: event timestamp converted to UTC wall-clock time.
• msg: debug message.
• args: up to three u64 arguments, omitted when all values are zero.

bpf_dbg: file=native_oncpu_profiler.c line=120 ts=2026-01-11T08:30:00.123456Z msg=enter prog args=[0x1f4 0xffff8881 0x0]

Debug output requires both a build with BPF_DEBUG=1 and the runtime --log-bpf-debug option.

3 - Integration Test

This integration test validates that huatuo-bamai can start correctly with mocked /proc and /sys filesystems and expose the expected Prometheus metrics.

The test runs the real huatuo-bamai binary and verifies the /metricsendpoint output without relying on the host kernel or hardware.

What the Script Does

The integration test performs the following steps:

1. Generates a temporary bamai.conf
2. Starts huatuo-bamai with mocked procfs and sysfs
3. Waits for the Prometheus /metrics endpoint to become available
4. Fetches all metrics from /metrics
5. Verifies that all expected metrics exist
6. Stops the service and cleans up resources

If any expected metric is missing, the test fails.

How to Run

Run the integration test from the project root:

bash integration/run.sh

Pass a file name to run one integration test. The optional second argument is the repeat count and defaults to 1:

bash integration/run.sh test_metrics_exclude_filter.sh 10

or

make integration

On Failure

• The huatuo-bamai service metrics and logs are printed to stdout
• The temporary working directory is kept for debugging

On Success

• Output the list of successfully validated metrics

How to Add New Metrics Tests

1: Add or Update Fixture Data

If the metric depends on /proc or /sys, add or update mock data under:

integration/fixtures/

The directory structure should match the real kernel filesystem layout.

2: Add Expected Metrics

Create a new file under:

integration/fixtures/expected_metrics/
├── cpu.txt
├── memory.txt
└── ...

Each non-empty, non-comment line represents one expected Prometheus metric line and must match the /metrics output exactly.

New *.txt files are automatically picked up by the test.

3: Run the Test

bash integration/run.sh

The test fails if any expected metric is missing or mismatched.

4 - BPF ABI Guide

The C structures on the BPF side are the source of truth for the perf event ABI. Go types are generated automatically from BTF in the BPF objects. Do not maintain equivalent structures manually.

Conventions

Each ABI domain maps to one C header and one generated Go file:

<domain> must start with a lowercase letter and contain only lowercase letters, digits, and underscores. Avoid domains with overlapping prefixes, such as net and net_rx.

Implementation

The following example creates the sample domain.

1. Define the ABI Header

Create bpf/include/abi/sample_types.h:

// Copyright 2026 The HuaTuo Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#ifndef __BPF_ABI_SAMPLE_H__
#define __BPF_ABI_SAMPLE_H__

#include "bpf_abi.h"

#define SAMPLE_DATA_LEN 4

struct sample_detail {
	u32 code;
	u8 data[SAMPLE_DATA_LEN];
};

struct sample_event {
	u64 timestamp;
	struct sample_detail detail;
	u32 pid;
	u8 kind;
	u8 pad[3];
};

BPF_ABI_EXPORT(sample_detail);
BPF_ABI_EXPORT(sample_event);

#endif /* __BPF_ABI_SAMPLE_H__ */

Requirements:

• Structure names must use the <domain>_ prefix.
• Every structure that requires a generated Go type must call BPF_ABI_EXPORT(<type>), including nested structures.
• The header must contain only ABI types and related constants. Do not add BPF programs, maps, or business logic.
• Array lengths must be compile-time constants and remain consistent across all BPF compilation units.

2. Use the Header in a BPF Program

The ABI header depends on base types such as u8 and u16. Include it after vmlinux.h and the project base headers:

#include "vmlinux.h"

#include "bpf_common.h"
#include "abi/sample_types.h"

Use the ABI structure directly when emitting a perf event:

struct sample_event event = {};

event.timestamp = bpf_ktime_get_ns();
event.pid = bpf_get_current_pid_tgid() >> 32;
event.kind = kind;

bpf_perf_event_output(ctx, &events, COMPAT_BPF_F_CURRENT_CPU,
		      &event, sizeof(event));

The event must be zero-initialized to prevent uninitialized padding from being written to the perf buffer. Any structure passed to bpf_perf_event_output must be defined in an ABI header.

3. Generate and Use the Go Type

Run the following command from the repository root:

make gen-build

The generated file is located at:

internal/bpf/abi/sample_types_generated.go

Reference the generated type directly from Go code:

import "huatuo-bamai/internal/bpf/abi"

var event abi.SampleEvent
if err := reader.ReadInto(&event); err != nil {
	return fmt.Errorf("read sample event: %w", err)
}

The generated file contains the structure, the SampleEventSize size constant, and layout assertions based on unsafe.Sizeof and unsafe.Offsetof. Generated files are read-only. Do not edit them manually.

Types and Layout

Layout requirements:

• Use fixed-width integers such as u8, s16, u32, and s64.
• Order fields according to their alignment requirements. Use u8 pad[N] to make padding explicit when necessary.
• Do not use platform-dependent types such as long or unsigned long.
• Do not use __attribute__((packed)) to bypass layout validation.
• If a structure with the same name appears in multiple BPF objects, its fields, offsets, and size must match exactly.

C names are converted to exported Go names by splitting on underscores. For example, sample_event becomes SampleEvent, and pid_tgid becomes PIDTGID. Avoid C names that map to the same Go name, such as sample_id and sample_i_d.

Verification

Run the following commands after adding or modifying an ABI:

make gen-build
go test ./build/bpfabi-tool
make check

Add a decoding test for each new perf event. At a minimum, cover:

• Integer boundary values and native byte order.
• Nested structures and the first and last array elements.
• Fields before and after padding.
• The sample size and the generated <GoType>Size constant.

Common Errors

5 - Time Format Contract

Time Format Contract

HUATUO serializes timestamps in UTC with fixed nanosecond precision:

2006-01-02T15:04:05.000000000Z

For example, an event captured at half past midnight on 22 July 2026 is written as:

2026-07-22T00:30:00.123456789Z

Why fixed UTC precision

• Every host emits the same timezone (Z), so distributed queries do not need timezone-specific handling.
• The nine fractional digits keep the string width fixed. Lexical ordering therefore matches chronological ordering, which is useful for log files and keyword-indexed storage.
• Nanoseconds preserve the precision supplied by Go’s time.Time and by event-tracing pipelines.

Integration guidance

When producing data for HUATUO, format times with the canonical helper:

timestamp := timeutil.FormatUTC(time.Now())

When consuming timestamps, use timeutil.Parse. It accepts the canonical format and RFC 3339 timestamps with zero to nine fractional digits so that legacy records remain readable. The returned value is always normalized to UTC.

Do not compare timestamps as strings unless both values have been generated by FormatUTC; third-party RFC 3339 values can have variable fractional precision or non-UTC offsets.