Development

• 1: Extending Observability
• 2: Development Debugging
• 3: Integration Test
• 4: Debug a VM in GitHub Actions
• 5: BPF ABI Guide
• 6: Challenges of TCP Retransmission and dropwatch Correlation
• 7: Time Format Contract
• 8: Event Field Naming

1 - Extending Observability

HUATUO supports three extension types: Metrics, Event, and AutoTracing. They use the same registration framework but differ in activation, runtime cost, and data output.

Collection Modes

Metrics

Metrics periodically collect system state through procfs, sysfs, or eBPF and expose it in Prometheus format. This mode supports real-time monitoring and long-term trend analysis. Built-in collectors cover:

• CPU: sys, usr, util, load, nr_running, and related metrics.
• Memory: vmstat, memory_stat, directreclaim, and asyncreclaim.
• I/O: d2c, q2c, freeze, and flush.
• Networking: ARP, socket memory, qdisc, netstat, netdev, and sockstat.

Event

Events continuously observe kernel events or threshold conditions and preserve kernel context when an anomaly occurs. This mode is intended for low-overhead, always-on observation. Data is written to Elasticsearch and local files and can also produce Prometheus metrics. Built-in events include:

• Scheduler tick interval anomalies (sched_tick).
• Abnormal memory allocation (memory_oom_kill).
• Soft lockups (softlockup).
• D-state processes (hungtask).
• Memory reclaim (memory_reclaim_events).
• Packet drops (dropwatch).
• Network receive latency (net_rx_latency).

AutoTracing

AutoTracing invokes diagnostic tools after detecting a system anomaly. It is intended for flame graphs, context snapshots, and other diagnostic operations that are too expensive to run continuously. Results are written to Elasticsearch and local files and can also be converted into Prometheus metrics. Built-in capabilities include:

• CPU idle and system-time anomaly tracing (cpuidle, cpusys).
• D-state load tracing (dload).
• Burst memory allocation tracing (memburst).
• Disk I/O anomaly tracing (iotracing).

Event and AutoTracing are both Tracing modes and share the ITracingEvent interface. They can preserve anomaly context for root-cause analysis and expose statistics to Prometheus by also implementing Collector.

Adding Metrics

Custom Metrics collectors expose Prometheus metrics through /metrics.

Implement Collector

Create a type under core/metrics that implements Collector:

type Collector interface {
    Update() ([]*Data, error)
}

type exampleMetric struct{}

func (c *exampleMetric) Update() ([]*metric.Data, error) {
    return []*metric.Data{
        metric.NewGaugeData("example", value, "example value", nil),
    }, nil
}

Register the collector

Use FlagMetric when registering the implementation:

func init() {
    tracing.RegisterEventTracing("example", newExampleMetric)
}

func newExampleMetric() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleMetric{},
        Flag:        tracing.FlagMetric,
    }, nil
}

Manage BPF object

When one implementation provides both Start and Update, the methods may run concurrently. Do not read and write a bpf.BPF interface directly in a collector field. Use Reference and Lease from internal/bpf/bpf_ref.go to manage the object lifetime:

type example struct {
    object bpf.Reference
}

func (c *example) Start(ctx context.Context) (retErr error) {
    object, err := bpf.LoadBpf(bpf.ThisBpfOBJ(), nil)
    if err != nil {
        return err
    }

    if err := object.Attach(); err != nil {
        return errors.Join(err, object.Close())
    }
    if err := c.object.Publish(object); err != nil {
        return errors.Join(err, object.Close())
    }
    defer func() {
        retErr = errors.Join(retErr, c.object.UnPublish())
    }()

    <-ctx.Done()
    return nil
}

func (c *example) Update() ([]*metric.Data, error) {
    lease, ok := c.object.Acquire()
    if !ok {
        return nil, nil
    }
    defer lease.Release()

    items, err := lease.DumpMapByName("example_map")
    if err != nil {
        return nil, fmt.Errorf("dump example_map: %w", err)
    }

    return buildMetrics(items), nil
}

The API has these constraints:

• Publish transfers ownership of the object to Reference. Do not call object.Close() directly after a successful publish.
• The Lease returned by Acquire pins the BPF object until the current Update completes. Always pair it with Release, and do not copy a Lease.
• UnPublish first prevents new acquisitions, then waits for every Lease to be released, closes the BPF object, and returns the close error.
• Calls to Publish and UnPublish must be serialized. The current framework does not run Start concurrently for the same instance.

An Update that has already started can therefore finish with its original BPF object. During shutdown or restart, Start closes that object only after those updates complete.

Adding an Event

An Event implements ITracingEvent:

type ITracingEvent interface {
    Start(ctx context.Context) error
}

type exampleEvent struct{}

func (e *exampleEvent) Start(ctx context.Context) error {
    // Detect the event and capture its context.
    // storage.Save writes the data to ES and local storage.
    storage.Save("example", containerID, time.Now(), eventData)
    return nil
}

Register it with FlagTracing:

func init() {
    tracing.RegisterEventTracing("example", newExampleEvent)
}

func newExampleEvent() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleEvent{},
        Interval:    10,
        Flag:        tracing.FlagTracing,
    }, nil
}

To expose Prometheus metrics for the event, also implement Collector and add tracing.FlagMetric to Flag.

Adding AutoTracing

AutoTracing and Event use the same ITracingEvent interface and registration framework:

type exampleAutoTracing struct{}

func (t *exampleAutoTracing) Start(ctx context.Context) error {
    // Capture context after the anomaly trigger fires.
    storage.Save("example", containerID, time.Now(), tracingData)
    return nil
}

func init() {
    tracing.RegisterEventTracing("example", newExampleAutoTracing)
}

func newExampleAutoTracing() (*tracing.EventTracingAttr, error) {
    return &tracing.EventTracingAttr{
        TracingData: &exampleAutoTracing{},
        Interval:    10,
        Flag:        tracing.FlagTracing,
    }, nil
}

See core/metrics, core/events, and core/autotracing for complete examples covering BPF map interaction, container metadata, storage, and Prometheus output.

2 - Development Debugging

Full-Stack Integration

The development Compose configuration builds an image from the current workspace and starts the collector, API Server, Elasticsearch, Prometheus, and Grafana. The collector needs access to the host kernel and cgroups, so run the command with root privileges from the repository root on a Linux host:

sudo make compose-dev-up

Compose aggregates all component logs in the foreground. After changing the source, press Ctrl+C and run the command again. Docker reuses the toolchain layers and Go build cache.

Remove the containers, data volumes, and development image after debugging:

sudo make compose-dev-down

This command removes the Elasticsearch data volume. Do not run it when the integration data must be retained.

BPF Debugging

BPF code can use the bpf_dbg() and bpf_dbg_msg() macros to emit debug information from kernel space. The macros are defined in bpf/include/bpf_dbg.h. Debugging has separate build-time and runtime switches and is completely disabled by default.

Add debug trace points

Each BPF source file that uses the macros must declare its own debug map:

#include "bpf_dbg.h"

BPF_DBG_MAP(native_cpu);

SEC("perf_event")
int prog(void *ctx)
{
        bpf_dbg_msg(ctx, native_cpu, "enter prog");
        bpf_dbg(ctx, native_cpu, "pid and addr", pid, addr, 0);
        return 0;
}

bpf_dbg_msg() emits a message only. bpf_dbg() also accepts up to three u64 arguments.

Build debug objects

Set BPF_DEBUG=1 to pass -DDEBUG_BPF to Clang:

make BPF_DEBUG=1

To regenerate build artifacts without rebuilding binaries:

make BPF_DEBUG=1 gen-build

BPF_DEBUG=0 is the default. In that mode the macros expand to no-ops, and the debug perf event array, event structure, bpf_ktime_get_ns, and bpf_perf_event_output are not emitted into the BPF object.

Enable runtime output

After building the debug objects, pass --log-bpf-debug when starting the profiler. The option currently applies only to the native profiler:

./profiler --type cpu --language native --log-bpf-debug ...

When loading the BPF object, bpf.NewDbg(true) rewrites the bpf_dbg_enabled constant to 1 before LoadBpf. When it is disabled, the verifier eliminates the branch as dead code. Each BPF object maintains an independent switch.

Read debug output

User space emits each debug event at Debug level with these fields:

• file: BPF source file.
• line: source line number.
• ts: event timestamp converted to UTC wall-clock time.
• msg: debug message.
• args: up to three u64 arguments, omitted when all values are zero.

bpf_dbg: file=native_oncpu_profiler.c line=120 ts=2026-01-11T08:30:00.123456Z msg=enter prog args=[0x1f4 0xffff8881 0x0]

Debug output requires both a build with BPF_DEBUG=1 and the runtime --log-bpf-debug option.

3 - Integration Test

This integration test validates that huatuo-bamai can start correctly with mocked /proc and /sys filesystems and expose the expected Prometheus metrics.

The test runs the real huatuo-bamai binary and verifies the /metricsendpoint output without relying on the host kernel or hardware.

What the Script Does

The integration test performs the following steps:

1. Generates a temporary bamai.conf
2. Starts huatuo-bamai with mocked procfs and sysfs
3. Waits for the Prometheus /metrics endpoint to become available
4. Fetches all metrics from /metrics
5. Verifies that all expected metrics exist
6. Stops the service and cleans up resources

If any expected metric is missing, the test fails.

How to Run

Run the integration test from the project root:

bash integration/run.sh

Pass a file name to run one integration test. The optional second argument is the repeat count and defaults to 1:

bash integration/run.sh test_metrics_exclude_filter.sh 10

or

make integration

Prerequisites and Results

Both suites use integration/run.sh, which defaults to integration. e2e/run.sh forwards to --suite e2e. Existing make integration, make e2e, and single-case entry points remain available:

bash integration/run.sh --suite e2e
bash integration/run.sh --suite e2e test_metrics.sh 2
bash e2e/run.sh test_metrics.sh 2

Both suites run in separate UTS and mount namespaces with hostname huatuo-dev and private mount propagation. Integration cases start their own services; e2e starts the baseline bamai before each case, then stops it and checks its log. Integration removes workspaces after passed or skipped cases and preserves them on failure. E2E retains its workspaces.

The runner checks only that _output exists, without checking individual project binaries or BPF objects. Run make build before invoking a runner directly. An existing directory does not prove that the build is complete; missing artifacts fail when used.

Use the same helper for commands on PATH and executable file paths. Missing commands skip the case:

require_commands jq curl ss
require_commands "${PROFILER_TOOL_DIR}/bin/asprof"

Use require_readable for input files such as certificates. Unreadable inputs skip the case and report the affected path:

require_readable "${KUBELET_CERT}" "${KUBELET_KEY}"

skip prints its reason and exits with 77; EXIT cleanup still runs. The runner classifies 0 as PASS, 77 as SKIP, and other statuses as FAIL. Cleanup failures also count as FAIL. A suite containing only passed and skipped cases returns 0. A failure stops subsequent cases and prints the results collected so far.

Each execution counts separately, including repetitions. For example:

summary: total=5 passed=3 skipped=2 failed=0

When root privileges or namespace commands are unavailable, both suites count all selected executions as SKIP.

On Failure

• The huatuo-bamai service metrics and logs are printed to stdout
• The temporary working directory is kept for debugging

On Success

• Output the list of successfully validated metrics

Background Process Cleanup

stop_and_wait_by_pid PID [TIMEOUT] stops and waits for a child process, returning its exit status. An omitted or empty PID returns success immediately, so callers can pass an initialized PID variable without checking it first. Clear the variable after waiting; use || true in best-effort cleanup and assert the exit status when it is part of the test contract.

Network Namespace Helpers

Network tests can source integration/lib_namespace.sh after lib.sh:

• namespace_setup NAME creates one network namespace and exposes its name as NETNS. The caller brings up lo and configures any addresses it needs.
• namespace_setup_with_pair PREFIX SERVER_ADDR CLIENT_ADDR [PREFIX_LENGTH] creates two namespaces connected by a veth pair, configures their addresses, and brings up the interfaces. The prefix length defaults to 24. Names and addresses are exposed through NETNS_SERVER, NETNS_CLIENT, NETNS_VETH_SERVER, NETNS_VETH_CLIENT, NETNS_SERVER_ADDR, and NETNS_CLIENT_ADDR.
• namespace_cleanup deletes the namespaces created by either helper and clears their state. Call it from the test’s EXIT cleanup after stopping processes in those namespaces. Repeated cleanup calls are supported.

How to Add New Metrics Tests

1: Add or Update Fixture Data

If the metric depends on /proc or /sys, add or update mock data under:

integration/fixtures/

The directory structure should match the real kernel filesystem layout.

2: Add Expected Metrics

Create a new file under:

integration/fixtures/expected_metrics/
├── cpu.txt
├── memory.txt
└── ...

Each non-empty, non-comment line represents one expected Prometheus metric line and must match the /metrics output exactly.

New *.txt files are automatically picked up by the test.

3: Run the Test

bash integration/run.sh

The test fails if any expected metric is missing or mismatched.

4 - Debug a VM in GitHub Actions

Use the OS Distro QEMU Test workflow to test a branch in a supported Linux VM. No local VM setup is required.

Prepare Your Fork

1. Fork the repository and enable workflows on the fork’s Actions page.
2. Add an SSH public key to your GitHub account. Only the user who starts the workflow can connect.
3. Push the branch that you want to debug to your fork.

Start a VM

1. Open Actions > OS Distro QEMU Test > Run workflow in your fork.
2. Select your branch.
3. Keep the default ubuntu24.04, enter comma-separated distributions, or use all to select every supported distribution.
4. Select when to retain the VM and the retention time under Debug VM.
5. Run the workflow and open its Test in VM job.

If Run workflow is missing, sync the fork’s default branch first. GitHub only lists manual workflows that exist on the default branch.

Use Before test to inspect or change the VM before tests run. Use After test on failure to investigate a failed test, or After test always to keep the VM regardless of the result.

Connect

Open the job’s Debug VM: Retain ... step and follow its commands:

1. Run the displayed SSH command in your local terminal. This connects to the GitHub runner.
2. On the runner, run the displayed ssh -i ... root@<VM_IP> command to enter the VM.

The commands already contain the correct key path and VM address.

Finish

Exit the VM, then release the workflow from the runner:

cd "$GITHUB_WORKSPACE"
touch continue

The workflow continues to the tests or cleanup. Disconnecting SSH alone does not release it; otherwise it waits until the selected retention time expires.

5 - BPF ABI Guide

The C structures on the BPF side are the source of truth for the perf event ABI. Go types are generated automatically from BTF in the BPF objects. Do not maintain equivalent structures manually.

Conventions

Each ABI domain maps to one C header and one generated Go file:

<domain> must start with a lowercase letter and contain only lowercase letters, digits, and underscores. Avoid domains with overlapping prefixes, such as net and net_rx.

Implementation

The following example creates the sample domain.

1. Define the ABI Header

Create bpf/include/abi/sample_types.h:

// Copyright 2026 The HuaTuo Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#ifndef __BPF_ABI_SAMPLE_H__
#define __BPF_ABI_SAMPLE_H__

#include "bpf_abi.h"

#define SAMPLE_DATA_LEN 4

struct sample_detail {
	u32 code;
	u8 data[SAMPLE_DATA_LEN];
};

struct sample_event {
	u64 kernel_observed_ns;
	struct sample_detail detail;
	u32 tgid;
	u8 kind;
	u8 pad[3];
};

BPF_ABI_EXPORT(sample_detail);
BPF_ABI_EXPORT(sample_event);

#endif /* __BPF_ABI_SAMPLE_H__ */

Requirements:

• Structure names must use the <domain>_ prefix.
• Every structure that requires a generated Go type must call BPF_ABI_EXPORT(<type>), including nested structures.
• The header must contain only ABI types and related constants. Do not add BPF programs, maps, or business logic.
• Array lengths must be compile-time constants and remain consistent across all BPF compilation units.

2. Use the Header in a BPF Program

The ABI header depends on base types such as u8 and u16. Include it after vmlinux.h and the project base headers:

#include "vmlinux.h"

#include "bpf_common.h"
#include "abi/sample_types.h"

Use the ABI structure directly when emitting a perf event:

struct sample_event event = {};

event.kernel_observed_ns = bpf_ktime_get_ns();
event.tgid = bpf_get_current_pid_tgid() >> 32;
event.kind = kind;

bpf_perf_event_output(ctx, &events, COMPAT_BPF_F_CURRENT_CPU,
		      &event, sizeof(event));

The event must be zero-initialized to prevent uninitialized padding from being written to the perf buffer. Any structure passed to bpf_perf_event_output must be defined in an ABI header.

3. Generate and Use the Go Type

Run the following command from the repository root:

make gen-build

The generated file is located at:

internal/bpf/abi/sample_types_generated.go

Reference the generated type directly from Go code:

import "github.com/ccfos/huatuo/internal/bpf/abi"

var event abi.SampleEvent
if err := reader.ReadInto(&event); err != nil {
	return fmt.Errorf("read sample event: %w", err)
}

The generated file contains the structure, the SampleEventSize size constant, and layout assertions based on unsafe.Sizeof and unsafe.Offsetof. Generated files are read-only. Do not edit them manually.

Types and Layout

Layout requirements:

• Use fixed-width integers such as u8, s16, u32, and s64.
• Order fields according to their alignment requirements. Use u8 pad[N] to make padding explicit when necessary.
• Do not use platform-dependent types such as long or unsigned long.
• Do not use __attribute__((packed)) to bypass layout validation.
• If a structure with the same name appears in multiple BPF objects, its fields, offsets, and size must match exactly.

C names are converted to exported Go names by splitting on underscores. For example, sample_event becomes SampleEvent, and kernel_observed_ns becomes KernelObservedNS. Field semantics and cross-layer names follow Event Field Naming. Avoid C names that map to the same Go name, such as sample_id and sample_i_d.

Verification

Run the following commands after adding or modifying an ABI:

make gen-build
go test ./build/bpfabi-tool
make check

Add a decoding test for each new perf event. At a minimum, cover:

• Integer boundary values and native byte order.
• Nested structures and the first and last array elements.
• Fields before and after padding.
• The sample size and the generated <GoType>Size constant.

Common Errors

6 - Challenges of TCP Retransmission and dropwatch Correlation

This document describes the evidence boundaries and the implementation of local correlation. The two event streams share no common event ID, so correlation can only make conservative inferences from the namespace, the 4-tuple, TCP sequence/ACK evidence, and monotonic time.

1. Result Semantics

A no-match does not prove that the problem is in the network or in hardware. The drop may have happened before capture started, in another network namespace, or the record may have been delivered without the TCP matching fields.

Shutdown processes the waiting entries with a single time snapshot: expired entries receive warmup or wait_timeout, and entries that have not expired receive interrupted. All of these unmatched results report drop_location=unknown plus independent diagnostics.

2. Three Operating Scenarios

1. tcpshark --bpf-path <tcp_retransmit.o> only captures and directly outputs retransmissions.
2. tcpshark --with-dropwatch --bpf-path-dir <dir> loads tcp_retransmit.o and net_dropwatch.o inside one process, which owns both perf inputs, the timer, the output, and shutdown.
3. The huatuo-bamai standalone dropwatch still runs independently through cmd/dropwatch, outputs only raw DropWatchTracing, and shares no state with the embedded source.

All three retransmit hooks use the synthetic L3 filter in both modes. Correlation mode uses only EventTracing.TCPRetransmit.Filter; an empty value is normalized to tcp, and the same expression is passed to both BPF objects. Expressions that depend on Ethernet addresses and cannot run equivalently on synthetic L3 input are rejected before startup. EventTracing.Dropwatch.Filter controls only standalone dropwatch.

Both commands use HardwareAuto to auto-detect devlink trap support. When the tracepoint is unavailable, they log a warning and continue software capture; when it is available, they accept only driver-reported DROP traps. --device / --device-excluded share their names with standalone dropwatch and are mutually exclusive; they filter only the embedded source’s software and hardware records and require --with-dropwatch. The retransmit input still uses the shared L3 filter.

Drop source and reason are resolved uniformly by internal/dropwatch.ResolveMetadata. Software reasons are looked up in a BTF table loaded once per session; unknown values are decimal numbers, and kernels without support yield NOT_SUPPORTED. Hardware reasons and groups come from the trap name and the trap group respectively and do not consult the software reason table. The reader keeps independent source and reason strings and does not borrow the reusable ABI buffer. After a strict match, the output carries drop_source, drop_reason, and drop_reason_group; a no-match omits these fields and outputs the single correlation terminal reason plus independent diagnostics. drop_location echoes drop_source on a match and is unknown otherwise; it expresses the correlation classification and differs from the standalone dropwatch kernel address. An unknown source is never inferred from a reason or a stack. Strict matching cannot be established when a hardware record lacks the namespace or TCP matching fields.

3. The Two Time Constraints

dropwatch and retransmit use different perf readers, and events can also come from different CPUs, so the userspace arrival order does not equal the kernel occurrence order:

CPU 2: drop       monotonic_ns=180, held back in the dropwatch ring
CPU 0: retransmit monotonic_ns=200, reaches userspace first
userspace: retransmit(200) -> drop(180)

The implementation uses two windows on different dimensions:

retransmitRetentionDuration = 100 * time.Millisecond
maxDropToRetransmitAge     = time.Second

• 100 ms is the userspace out-of-order delivery budget. When a retransmission arrives first, it enters the wait queue; if it is still unmatched when the wait expires, it is reported as unknown. Records whose deadline has already passed are finalized before any new event is processed; the timer is only responsible for waking the loop and does not define the deadline semantics.
• 1s is the causal candidate age, judged with the BPF kernel_observed_ns. A candidate must satisfy:

drop.kernel_observed_ns <= retransmit.kernel_observed_ns
retransmit.kernel_observed_ns - drop.kernel_observed_ns <= 1s

observed_timestamp is userspace wall time, affected by scheduling and system time adjustments, and does not participate in matching. The fixed 1s is the currently supported contract and is not a theoretical bound for all Linux RTOs; results outside the window stay unknown.

4. Bounded State

Waiting retransmissions and not-yet-matched drops share the in-package generic store store[T], each holding an independent instance. The container is responsible only for capacity, TTL, the flow index, and deletion; strict matching and result finalization belong to the correlator.

type storeEntry[T any] struct {
    value    T
    flow     *flowKey
    deadline time.Time
    sequence uint64
    node     *list.Element
}

type store[T any] struct {
    byDeadline   list.List
    byFlow       map[flowKey][]*storeEntry[T]
    capacity     int
    ttl          time.Duration
    nextSequence uint64
}

waitingRetransmit contains only the raw event, the matching fields, and the cross-namespace candidate flag; it no longer holds a list node, a deadline, or an independent ID. Drops likewise need no extra cache wrapper type. The list’s Element.Value and the flow bucket reference the same *storeEntry[T], and the entry’s node points back to its list element. The entry references the flow from the business data, which is never modified, avoiding a duplicate address. Deletion unreferences both indexes at once and clears the tail of the flow slice; an empty bucket is removed after its last entry is deleted.

The flow index is keyed by the canonical order of the two AddrPort values, so both directions share one bucket; the original direction is kept in the business data for strict matching. The namespace is not part of the index key, so similar cross-namespace candidates can still set the diagnostic flag. Match scans and slice deletions are O(k), where k is the number of candidates for that flow; locating the earliest deadline and unlinking from the list are O(1).

Each instance has a fixed TTL, and the correlation loop passes a monotonically non-decreasing processing time, so list insertion order is also deadline order. The list serves expiry cleanup and capacity eviction and exposes no FIFO consumption interface to business code. When the wait capacity is full, the record with the earliest deadline is finalized as queue_full; when the drop capacity is full, the candidate with the earliest deadline is evicted directly. Causal decisions still use kernel_observed_ns.

The two readers deliver events through unbuffered channels, and the container is accessed only by a single correlation loop, with no locks and no separate cleanup goroutine. One timer always takes the earliest deadline of the two instances, so cleanup happens on schedule even when only the drop cache holds data and no retransmission is waiting. On every input and timer wakeup, expired entries are cleaned first, then matching and capacity checks run; now >= deadline means expired. The 100 ms is the matching deadline; blocking in synchronous output can still delay actual cleanup and result delivery.

5. Strict Matching

A forward segment candidate requires:

• the same network namespace, address family, and direction 4-tuple;
• a drop no later than the retransmission, with a time difference of at most 1s;
• an overlap between the TCP SYN/data/FIN sequence range and the retransmitted range;
• RST and unsupported retransmission types produce no forward match.

A reverse ACK candidate uses the reverse 4-tuple and requires the ACK to cover the retransmitted sequence end. SYN and SYN-ACK use their own stricter ACK/SYN conditions.

Among multiple drop candidates, the largest drop.kernel_observed_ns is selected first; on a time tie, the larger insertion sequence number wins. When a drop arrives late, it selects the strictly matched retransmission with the smallest insertion sequence number while scanning the remaining candidates to record each retransmission’s namespace-match status. After a strict match, the entry is removed from both the deadline and flow indexes immediately and can be consumed only once. Forward segment and reverse ACK share the flow index; the namespace is checked first, then the packet and time conditions.

When a drop with the same namespace appears on the same flow, netNamespace is set to true; the flag is retained even when time or sequence fails the strict conditions, and later non-matching candidates do not clear it. A strict match always yields true. With no candidates, only other namespaces, or an incomparable namespace, the value is false, meaning no match was observed; it does not establish that the namespaces differ. The output preserves this status through NetNamespace; the JSON and text field is matched_net_namespace, omitted when false.

6. Mutually Exclusive Correlation Outcomes

types.CorrelationReason is a string type; each correlation outputs exactly one correlation_reason:

Each terminal branch assigns correlationResult.reason directly. The output side uses this enum to classify results and validates that matched must carry a drop while the other five outcomes must not; empty values, unknown values, or contradictory combinations return an error. Correlation-disabled output omits the field; once correlation is enabled, every finalized event contains one valid terminal outcome. The scalar field replaces the previous reason array; tools and receivers migrate in lockstep.

On wait expiry, the terminal outcome is chosen by the retransmission’s kernel time: earlier than source readiness yields warmup, at or after readiness yields wait_timeout, including waits under 1s after readiness. The 1s drop-to-retransmit limit applies only to candidate matching and plays no part in the warmup decision. Userspace processing delay does not change the classification. This decision runs only in the expiry branch; strict matches, unsupported events, queue evictions, and early interruptions keep their own reasons. Warmup neither blocks matching nor ends the wait early.

matched_net_namespace remains an independent diagnostic flag: a drop with the same namespace was observed on the same flow, independent of the packet and time conditions; it cannot substitute for correlation_reason=matched.

The flag can coexist with rate limiting and loss counters without changing the single terminal outcome. Drops that fail normalization and drop-cache capacity evictions do not directly end a retransmission wait and do not produce a retransmission-level reason.

7. Perf Status

The status consists of three independent channels; each reports on its own and none is summed with another:

• perf_lost: the cumulative per-CPU bpf_perf_out_dropwatch map (the common perf output accounting facility bpf_perf_output.h); BPF increments it when bpf_perf_event_output returns a negative value (for example, when the current CPU has no attached reader).
• lost_samples: the userspace cumulative count of ring buffer overflows reported by PERF_RECORD_LOST, visible only while the reader runs, and queryable through ReadStatus once the lost records are processed.
• rate_limited: the total_missed value of the rate-limit status map bpf_rlimit_dropwatch.

On success, ReadStatus sets HasMapCounters=true, serialized as map_counters_available=true. On error the flag is false and PerfLost and RateLimited are zero, meaning unavailable rather than an absence of loss or rate limiting; LostSamples remains valid and the output keeps that snapshot. These values are per-instance cumulative counters, sampled separately by the map and the reader, and cannot explain why any individual retransmission failed to match.

Userspace sums perf_lost across all CPUs and returns the current snapshot each time; it does not check for counter regressions or uint64 addition overflow. This status describes evidence integrity only and never promotes a no-match into a deterministic network classification. The old active-epoch, dual-slot, inflight, frontier, and drain watermark mechanisms have been removed.

The correlation side computes the TCP sequence span from the IPv4 total length or the IPv6 payload length. Under GSO/offload the IP header length may not cover the whole skb; matching is not widened on that basis today.

The dropwatch hot path is:

software: hardware marker lookup/delete -> device/pcap filter
          -> status map lookup -> bpf_ktime_get_ns() -> rate limit -> perf output
hardware: device/pcap filter -> status map lookup -> bpf_ktime_get_ns()
          -> rate limit -> perf output

Consequently, events rejected by the filter pay no status lookup, event-time helper, or counter update cost; a software drop first consumes any existing hardware dedup marker so rejected kfree paths leave no marker behind.

8. Shutdown

tcpshark uses one errgroup.WithContext to manage the retransmit rate-limit reader, the retransmit reader, the embedded dropwatch reader, and the correlation loop. All command workers share the same groupCtx; when any worker returns an error the rest are canceled, and Wait returns the first worker error. Resource close errors are merged by each owner with errors.Join; shutdown no longer maintains a private error-aggregating group.

internal/dropwatch.Tracer owns the dropwatch object, the readers, and the rate-limit alert worker. The instance context derives from groupCtx; a rate-limit alert failure cancels instance reads, and the concrete error is returned by Close. ReadInto checks close and cancellation causes before reading; once the underlying read is entered, the read result is returned as-is and is no longer overwritten by a subsequent close or cancellation. ReadStatus can still read the final counters after the context is canceled. Close first cancels the instance, detaches, and closes the event and alert readers, then waits for the alert worker to exit, and finally releases the object; closing the alert reader wakes an idle read, so no polling deadline needs to elapse. The status type is types.DropwatchStatus, the output field is drop_perf_status, and it includes the map-counter availability flag.

On normal completion or worker failure:

1. Cancel the shared groupCtx;
2. the two ReadInto readers and the correlation loop exit;
3. records still in the dropwatch perf ring that were never handed to the correlator are no longer read;
4. drainRetransmits(now) is called to finalize expired entries with the same time snapshot, then clear the unexpired ones, writing output in deadline order;
5. after all workers exit, the dropwatch and retransmit Tracers are closed, collecting internal alert-worker and resource-release errors;
6. finally the socket output is finished; the caller-provided io.Writer is not closed by the session.

At shutdown, expired entries report warmup or wait_timeout and unexpired entries report interrupted, all retaining drop_location=unknown, the diagnostic flags, and the perf status. Already matched or already evicted events are never finalized twice. Tail drops can still be lost, so a retransmission that could have matched may also be ended early.

9. File Responsibilities

cmd/tcpshark/
├── main.go                         # signals, version, CLI entry point
├── cli.go                          # argument validation, path and filter normalization
├── run.go                          # global BPF lifecycle, timeout, feature dispatch
└── retransmit/                     # retransmit feature package
    ├── config.go                   # Config, RunConfig
    ├── run.go                      # Run: resource assembly, execution, and cleanup
    ├── tracer.go                   # Open, ReadInto, Close, and the rate-limit worker
    ├── bpf_load.go                 # loadBPF, attachBPF, and probe selection
    ├── event_reader.go             # dual-stream reading, conversion, and channel delivery
    ├── correlation_loop.go         # select, timer, and exit finalization
    ├── correlation.go              # correlation state machine, expiry finalization
    ├── correlation_match.go        # candidate selection, consumption, and namespace-match status
    ├── match.go                    # matching types: namespace/flow/sequence/ktime
    ├── store.go                    # entry storage, flow index, capacity and expiry management
    ├── event.go                    # conversion of the two ABI record types
    ├── classify.go                 # retransmission classification
    ├── correlation_output.go       # status completion, symbolization, correlation output fields
    └── output.go                   # text, JSON, and socket output

The top level runs the retransmit feature through retransmit.Run(ctx, *RunConfig) and does not touch ABI records, channels, the correlation timer, or status counters. The CLI validates scalars and argument combinations first, then compile-checks the filter. When a non-empty --output-storage is set, the --output value is not validated; if the latter is given explicitly, a note on stderr says it is ignored. Without storage, only json or text is accepted. RunConfig.Tracing is the capture configuration; when Dropwatch is nil, it reads and outputs directly; when non-nil, it uses two read workers and a single correlation loop. The correlation capability is a composition of the retransmit feature, not a general capture framework.

Run builds the output before attaching probes and, before returning, completes waiting for all session workers, final finalization, Tracer closing, and output completion. Configuration failures, cancellation, and run failures all release the resources already acquired. Draining kernel buffers at close is not guaranteed. Plain mode creates no correlation channels or timers.

retransmit.Tracer still owns only the retransmit BPF object, the readers, and the rate-limit alert worker; event conversion, classification, correlation, and output belong to the feature session in the same package and stay out of the Tracer. An Open failure rolls back every acquired resource; Close is still required after the session context is canceled. A rate-limit worker failure cancels the capture, and the original error is also preserved by Close. Once reading has started, ReadInto keeps the underlying result as-is and no longer checks close or cancellation state. Close may interrupt a read, but callers must serialize Close calls. Process-level bpf.Init/Shutdown and the run timeout are managed centrally at the command layer.

The correlator uniformly cleans up expired candidates while handling each input and timer wakeup; the caches do not repeat expiry checks. The correlator returns match evidence, a single terminal outcome, and diagnostic flags, and does not modify output fields; correlation_output.go builds the final output in one place. Each non-empty output batch reads the dropwatch status once at its entry, and empty batches read nothing; symbols are resolved only for matched drops. Status is read even when the whole batch is matches; on a read failure the batch is still written out and the status error is returned afterwards; if the write fails, batch output stops and the write error is merged with the status error. Only no-match events carry the status snapshot and diagnostic flags; a failed read still keeps valid reader counters, and matched events do not carry these fields. Internal retransmission events keep only the ABI record and the observedAt from a successful read; the wait queue does not store full presentation objects. Matching constructs netip addresses directly from ABI addresses and uses the ABI event enums, with no string conversions. IPv4-mapped IPv6 is still normalized to IPv4. Single-event reads handle lost-record retries and read times in one place; plain mode reuses the event and outputs synchronously, while correlation mode reads into a dedicated event before sending it, avoiding overwriting data that is still waiting. run.go creates the channels and starts the read workers in one place, and the sender closes the channels; read functions start no goroutines and use neither consumer callbacks nor borrowed-event contracts. TCPRetransmitTracing is constructed only at final output; ObservedTimestamp uses the original read time, not the correlation end time. The ABI, SYNACK flag completion, and matching rules stay unchanged.

cmd/tcpshark/retransmit does not enforce a Go internal import restriction; the convention is that only tcpshark uses it, and it does not depend back on the command layer. The capture implementation shared by the two commands lives in:

internal/dropwatch/
├── config.go    # Config, HardwareMode, and configuration validation
├── tracer.go    # Tracer API, newTracer resource takeover, rollback, and worker lifecycle
├── load.go      # loadBPF, attachBPF: loading, map/reader preparation, and probe attachment
├── netdev.go    # device filtering and map configuration
├── status.go    # ReadStatus: output failures, ring loss, and rate-limit counters
├── reason.go    # LoadReasonNames: kernel BTF reason table and numeric fallback
├── metadata.go  # ResolveMetadata: source, software reason, hardware trap/group
└── decode.go    # DecodePacket: ABI header adaptation and packet.Parse

Open(ctx, *Config) returns a concrete *Tracer. Each instance allows only one ReadInto caller; status reads and closing may run concurrently, but Close on the same instance must be serialized by the resource owner. The first close returns cleanup errors; later calls return nil, never retry cleanup, and never return the first error again. The caller owns global BPF initialization and shutdown, and must close the Tracer after the final finalization. DecodePacket results do not borrow the original record memory. Symbolization and the DropWatchTracing presentation conversion stay in the standalone command; TCP correlation keeps KernelObservedNS, the namespace, stack PCs, and the resolved drop source and reason directly from the ABI. No general BPF consumption-loop interface is introduced.

7 - Time Format Contract

Time Format Contract

HUATUO serializes timestamps in UTC with fixed nanosecond precision:

2006-01-02T15:04:05.000000000Z

For example, an event captured at half past midnight on 22 July 2026 is written as:

2026-07-22T00:30:00.123456789Z

Why fixed UTC precision

• Every host emits the same timezone (Z), so distributed queries do not need timezone-specific handling.
• The nine fractional digits keep the string width fixed. Lexical ordering therefore matches chronological ordering, which is useful for log files and keyword-indexed storage.
• Nanoseconds preserve the precision supplied by Go’s time.Time and by event-tracing pipelines.

Integration guidance

When producing data for HUATUO, format times with the canonical helper:

timestamp := timeutil.FormatUTC(time.Now())

When consuming timestamps, use timeutil.Parse. It accepts the canonical format and RFC 3339 timestamps with zero to nine fractional digits so that legacy records remain readable. The returned value is always normalized to UTC.

Do not compare timestamps as strings unless both values have been generated by FormatUTC; third-party RFC 3339 values can have variable fractional precision or non-UTC offsets.

8 - Event Field Naming

Event fields must use consistent semantics before following the naming conventions of each layer. The same name must not represent different concepts.

Layer Rules

CPUs and Tasks

Time and Numeric Units

Containers, Cgroups, and Network Namespaces

Kernel terms inum and ifindex remain single words in Go. Do not split them into INum or IfIndex. User-facing Go and JSON fields expand netns to NetNamespace and net_namespace.

Observation times and compatibility

kernel_observed_timestamp, observed_timestamp, and uploaded_timestamp represent kernel observation, userspace observation, and storage write time. Kernel observation means the hook execution time, which may differ from the onset of a physical hardware error or network fault.

Document stores UTC timestamps at the top level and excludes kernel_observed_ns. Producers without kernel timestamps and older documents omit kernel_observed_timestamp; userspace or upload timestamps must not fill it. Historical RAS documents used observed_timestamp for kernel observation, so their userspace observation time cannot be reconstructed.

tcpshark and dropwatch use kernel_observed_timestamp for kernel observation time in JSON and text output. Upgrade tools and Agent together. Existing documents are not rewritten or backfilled. Conversion requires the same host, boot, and host time namespace. Historical events spanning a realtime clock step or system suspend cannot be mapped exactly to UTC from monotonic time alone.

UTC conversion caches the monotonic-to-realtime offset. The first call one hour after the last successful sample refreshes it; ordinary calls do not extend its lifetime. Sampling failures are returned and retried on the next call. Realtime clock steps and system suspend do not trigger an early refresh; the offset is updated on the first call after the cache expires.