Best Practice

• 1: Storage Service
• 2: Data Source
• 3: Events Watch
• 4: Profiling
• 5: Network Drop Monitoring
• 6: TCP Retransmission Tracing

1 - Storage Service

📖 Overview

HUATUO supports persisting Linux kernel events collected by the Tracer and AutoTracing data to external storage backends. Both Elasticsearch and OpenSearch are supported.

After serialization to JSON, collected events are written concurrently to the local node directory (huatuo-local/) and the configured remote storage backend. The local directory retains a local copy of events; the remote backend provides durable storage and structured query capabilities.

This document covers configuration and verification for both Elasticsearch and OpenSearch. Examples use Docker deployments. In production, replace the addresses with your actual service endpoints — the configuration format is the same.

🎯 Use Cases

Kubernetes Cloud-Native Fault Tracing

In containerized environments, kernel events such as Pod OOM and node Hung Task are transient — logs are often purged shortly after the event occurs. By writing events to Elasticsearch or OpenSearch, operations teams can query the historical timeline of anomalies by time range and precisely identify the root cause of intermittent failures during post-incident reviews.

AI Compute Cluster Stability Auditing

During long-running GPU training workloads, the historical distribution of events such as ras hardware errors and iotracing I/O latency is critical for capacity planning and hardware health assessment. Persisting collected data enables aggregate queries to establish node stability baselines and supports proactive maintenance decisions.

Compliance and Event Retention

Security compliance standards require that system anomaly events be traceable. Writing HUATUO-captured kernel events to OpenSearch and configuring an index lifecycle policy satisfies compliance requirements for event retention periods and query capabilities.

Observability Platform Integration

Both Elasticsearch and OpenSearch provide native data source integrations with Grafana. Once HUATUO events are written to storage, you can build kernel event trend dashboards in Grafana, overlaid with application-layer metrics for historical analysis and alert review.

💎 Value

🚀 Usage

OpenSearch V2

1. Deploy OpenSearch

docker pull opensearchproject/opensearch:2.6.0
docker run -d --name opensearch --network host \
  -e "discovery.type=single-node" \
  opensearchproject/opensearch:2.6.0

2. Verify Service Status

curl -k -u admin:admin https://localhost:9200

Example response:

{
  "name" : "22ca72df78c0",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "yxb3foceQVKzXXO6bHpPHQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.6.0",
    "build_type" : "tar",
    "build_hash" : "7203a5af21a8a009aece1474446b437a3c674db6",
    "build_date" : "2023-02-24T18:57:04.388618985Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

If verification fails, check the container logs:

docker logs opensearch

3. Configure huatuo-bamai

Add the following configuration to huatuo-bamai.conf. The default username and password for the OpenSearch container image are both admin. For a full description of storage configuration options, see the Configuration Guide.

[Storage.Elasticsearch]
    Address = "https://127.0.0.1:9200"
    Index = "huatuo_bamai"
    Username = "admin"
    Password = "admin"

4. Start huatuo-bamai

Use --config-dir to specify the directory containing the configuration file:

./_output/bin/huatuo-bamai --region dev --config-dir .

When files (e.g., net_rx_latency) appear in the local storage directory huatuo-local/, kernel events have been successfully captured. Query data from OpenSearch with:

curl -k -u admin:admin \
  -X GET "https://localhost:9200/huatuo_bamai/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"match_all": {}}}'

Example response:

{
    "_index" : "huatuo_bamai",
    "_id" : "yjPG_50Bu_OF-hukxKR7",
    "_score" : 1.0,
    "_source" : {
      "hostname" : "hostname",
      "region" : "dev",
      "uploaded_time" : "2026-05-07T00:11:49.753166222Z",
      "time" : "2026-05-07 00:11:49.753 +0000",
      "tracer_name" : "net_rx_latency",
      "tracer_time" : "2026-05-07 00:11:49.753 +0000",
      "tracer_type" : "auto",
      "tracer_data" : {
        "comm" : "<nil>",
        "pid" : 0,
        "where" : "RX_STAGE_NETIF",
        "latency_ms" : 1776078133565,
        "saddr" : "127.0.0.1",
        "daddr" : "127.0.0.1",
        "sport" : 37736,
        "dport" : 9200,
        "seq" : 1080592402,
        "ack_seq" : 2465063876,
        "pkt_len" : 781
      }
    }
}

To get the total document count without listing individual records:

curl -k -u admin:admin -X GET "https://localhost:9200/huatuo_bamai/_count?pretty"

Example response: the count value equals the total number of written records.

{
  "count" : 2680,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

Elasticsearch V8

1. Deploy Elasticsearch

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.15.5
docker run -d --name elasticsearch --network host \
  -e "discovery.type=single-node" \
  -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
  -e "ELASTIC_PASSWORD=123456" \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.5

2. Verify Service Status

curl -k -u elastic:123456 https://localhost:9200

Example response:

{
  "name" : "ab0b562f8dbd",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "aVfOVgJTQXuhZ3HGotK3ww",
  "version" : {
    "number" : "8.15.5",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "b10896bcfe167cce44a84ba2771d101fb596d40d",
    "build_date" : "2024-11-21T22:06:13.985834967Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

3. Configure huatuo-bamai

Add the following configuration to huatuo-bamai.conf. The default username for the Elasticsearch container image is elastic; the password is set via the ELASTIC_PASSWORD environment variable. For a full description of storage configuration options, see the Configuration Guide.

[Storage.Elasticsearch]
    Address = "https://127.0.0.1:9200"
    Index = "huatuo_bamai"
    Username = "elastic"
    Password = "123456"

4. Start huatuo-bamai

Use --config-dir to specify the directory containing the configuration file:

./_output/bin/huatuo-bamai --region dev --config-dir .

When files (e.g., net_rx_latency) appear in the local storage directory huatuo-local/, kernel events have been successfully captured. Query data from Elasticsearch with:

curl -k -u elastic:123456 \
  -X GET "https://localhost:9200/huatuo_bamai/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"match_all": {}}}'

Example response:

{
    "_index" : "huatuo_bamai",
    "_id" : "WtNZAJ4BQ8x-thPHEY1i",
    "_score" : 1.0,
    "_source" : {
      "hostname" : "hostname",
      "region" : "dev",
      "uploaded_time" : "2026-05-07T02:51:37.696263325Z",
      "time" : "2026-05-07 02:51:37.696 +0000",
      "tracer_name" : "net_rx_latency",
      "tracer_time" : "2026-05-07 02:51:37.696 +0000",
      "tracer_type" : "auto",
      "tracer_data" : {
        "comm" : "<nil>",
        "pid" : 0,
        "where" : "RX_STAGE_NETIF",
        "latency_ms" : 1776078133565,
        "saddr" : "127.0.0.1",
        "daddr" : "127.0.0.1",
        "sport" : 2379,
        "dport" : 36706,
        "seq" : 950542706,
        "ack_seq" : 1960972383,
        "pkt_len" : 91
      }
    }
}

To get the total document count without listing individual records:

curl -k -u elastic:123456 -X GET "https://localhost:9200/huatuo_bamai/_count?pretty"

Example response: the count value equals the total number of written records.

{
  "count" : 2680,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

Elasticsearch V7

Elasticsearch V7 uses HTTP by default. Replace https with http in all commands.

1. Deploy Elasticsearch

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.10.1
docker run -d --name elasticsearch --network host \
  -e "discovery.type=single-node" \
  -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
  -e "ELASTIC_PASSWORD=123456" \
  docker.elastic.co/elasticsearch/elasticsearch:7.10.1

2. Verify Service Status

curl -k -u elastic:123456 http://localhost:9200

Example response:

{
  "name" : "d88c9e8df48b",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "_ZZefWx4SniAc255t_lIVg",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

3. Configure huatuo-bamai

[Storage.Elasticsearch]
    Address = "http://127.0.0.1:9200"
    Index = "huatuo_bamai"
    Username = "elastic"
    Password = "123456"

4. Start huatuo-bamai

Use --config-dir to specify the directory containing the configuration file:

./_output/bin/huatuo-bamai --region dev --config-dir .

When files (e.g., net_rx_latency) appear in the local storage directory huatuo-local/, kernel events have been successfully captured. Query data from Elasticsearch with:

curl -k -u elastic:123456 \
  -X GET "http://localhost:9200/huatuo_bamai/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"match_all": {}}}'

To get the total document count:

curl -k -u elastic:123456 -X GET "http://localhost:9200/huatuo_bamai/_count?pretty"

⚙️ How It Works

System Architecture

The HUATUO Storage module runs on each node. It writes kernel events captured by the Tracer to the local directory and to Elasticsearch or OpenSearch. Both backends share the same [Storage.Elasticsearch] configuration interface and are differentiated by address.

The remote write path uses the ES/OpenSearch Bulk API (_bulk): events are queued in an in-memory buffer and submitted in batches by background workers based on size and time thresholds, with transport-layer retries on transient failures.

graph TB
    subgraph kernel["Linux Kernel"]
        K1[Kernel Events]
        K2[AutoTracing]
    end

    subgraph huatuo["HUATUO Agent (node-level)"]
        T["Tracer Layer"]
        L["Local Directory\nhuatuo-local/"]
        S["Storage Module\nBulkIndexer Buffer"]
    end

    subgraph backends["Storage Backends"]
        ES[Elasticsearch]
        OS[OpenSearch]
    end

    kernel --> T
    T --> L
    T --> S
    S -->|Bulk API + auto retry| ES
    S -->|Bulk API + auto retry| OS

Write Flow

Save returns immediately after the event is buffered. Background workers flush the buffer to the remote backend when any of the following triggers fire: byte threshold, time threshold, or process shutdown. The local directory write is synchronous and independent of the remote Bulk path.

sequenceDiagram
    participant T as Tracer Layer
    participant L as Local Directory (huatuo-local/)
    participant S as Storage Module (BulkIndexer)
    participant B as ES / OpenSearch

    T->>S: Kernel event captured, serialized to JSON
    par Local path (sync)
        S->>L: Write to local file
    and Remote path (async batch)
        S->>S: Enqueue into bulk buffer, return immediately
        Note over S: Flush on 5 MB / 1 s / shutdown
        S->>B: POST /_bulk (multiple records)
        B-->>S: 200 OK + per-item results
        Note over S: Failed items reported via OnFailure callback
    end

Bulk Write Mechanism

Buffering and Flush Triggers

Two-Tier Retry Policy

Bulk failures are split into two layers with different retry semantics:

Why this design: 429/5xx and transport errors signal transient remote unavailability where retries are effective; 4xx (except 429) and per-item errors are client-side semantic issues (data shape, permissions) where retries would only amplify the failure — they should be surfaced via logs for human investigation.

Data-Loss Scenarios

In all three scenarios below, Save returns nil but the event never reaches the index:

1. Abnormal process exit: SIGKILL or host power loss drops whatever is still buffered in the BulkIndexer (the local directory still keeps a copy).
   • Mitigation: SIGTERM/SIGINT trigger graceful shutdown; Close force-flushes the buffer with a 10 s deadline.
2. Whole-batch permanent rejection: 4xx (non-429) errors discard every record in the batch. Common causes: disabled index, expired credentials, document exceeding the cluster’s http.max_content_length.
   • Diagnosis: OnError log includes ES’s type and reason.
3. Permanent per-item failure: mapping conflict, version conflict, malformed document.
   • Diagnosis: OnFailure log identifies the record by index/id.

The local directory is always a fallback: even if remote writes are lost, events remain available in huatuo-local/ as the eventual-consistency safety net.

Problems This Solves

Replacing per-event Index API calls with a buffered BulkIndexer + auto-retry addresses four classes of problems:

🌟 Stay Connected

🌟 Star us on GitHub: https://github.com/ccfos/huatuo

👀 Follow our official WeChat public account

2 - Data Source

HUATUO integrates with Prometheus for metrics collection and Elasticsearch for log storage. This document covers data source configuration and dashboard provisioning in Grafana.

Two deployment paths are supported:

• Docker Compose — recommended for development and testing; all components are pre-configured.
• Kubernetes — for production clusters; requires manual data source configuration.

Quick Start (Docker Compose)

The build/docker/ directory contains a complete stack. All default credentials and ports listed below match this setup.

cd build/docker
docker compose up -d

This starts four services on the host network:

Default credentials:

Data sources and dashboards are auto-provisioned. Access Grafana at http://<host>:3000.

Verify the Stack

# Elasticsearch
curl -s -u elastic:huatuo-bamai http://localhost:9200/_cluster/health?pretty

# Prometheus — should show huatuo target as "up"
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# Grafana
curl -s http://localhost:3000/api/health | jq .version

# HuaTuo metrics
curl -s http://localhost:19704/metrics | head -5

Provisioned Data Sources

The following data sources are created automatically via build/docker/grafana/datasources/:

Provisioned Dashboards

Six dashboards are loaded from build/docker/grafana/dashboards/:

• Metric Dashboard — Host View
• Metric Dashboard — Container View
• HuaTuo Root Cause Analysis AutoTracing
• Continuous Profiling (Host)
• Continuous Profiling (Container)
• AutoTracing Flame Redirect

No manual import is needed when using Docker Compose.

Metrics Collection (Kubernetes)

1. Verify Metrics Endpoint

After deploying huatuo-bamai to Kubernetes, expose the metrics endpoint:

kubectl port-forward -n default --address=0.0.0.0 pod/huatuo-XXXX 19704:19704

Verify:

curl http://localhost:19704/metrics

Metrics output confirms the agent is running correctly.

2. Configure Prometheus Scraping

Option A: Pod Annotations

Add annotations to the Pod template metadata. This requires a Prometheus setup with Kubernetes pod service discovery enabled (e.g., kubernetes_sd_configs with role: pod).

template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "19704"
        prometheus.io/path: "/metrics"

Option B: ServiceMonitor

Requires Prometheus Operator. Create two resources:

huatuo-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: huatuo
  labels:
    app: huatuo
spec:
  clusterIP: None
  ports:
    - name: metrics
      port: 19704
      targetPort: 19704
      protocol: TCP
  selector:
    app: huatuo

huatuo-servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: huatuo
  namespace: default
  labels:
    release: prometheus
spec:
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app: huatuo
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s

3. Query Metrics in Prometheus

huatuo_*

If results are returned, metrics collection is working properly.

Log Collection (Kubernetes)

Query logs from Elasticsearch:

curl -u elastic:<password> "http://<es-host>:9200/huatuo_bamai/_search?pretty"

Replace <password> and <es-host> with your Elasticsearch credentials and address.

Manual Grafana Data Source Configuration

When not using Docker Compose (e.g., external Grafana), configure data sources manually.

Prometheus Data Source

Refer to build/docker/grafana/datasources/prometheus.yaml for the provisioning file, or configure via Grafana UI:

• URL: http://<prometheus-host>:9090
• Access: Server (proxy)

Elasticsearch Data Source

Configure via Grafana UI or provisioning:

• URL: http://<es-host>:9200
• Authentication: Basic Authentication
• Username: elastic
• Password: <your-elasticsearch-password>
• Index name: huatuo_bamai
• Time field name: uploaded_time

Dashboard Import

When using Docker Compose, dashboards are provisioned automatically. To import additional dashboards from the HUAUO console:

1. Access http://console.huatuo.tech/dashboards (Username: huatuo, Password: huatuo1024)
2. Select the desired dashboard
3. Click Export -> Export as JSON
4. Check “Export the dashboard to use in another instance”
5. Click Copy to clipboard

Then in your Grafana instance:

1. Navigate to Dashboards -> Import
2. Paste the JSON content
3. Click Load
4. Select the correct data sources and click Import

Troubleshooting

“datasource not found” when importing dashboard

This occurs when the dashboard JSON references a datasource UID that does not exist in your Grafana instance.

Solution:

1. Find your Elasticsearch datasource UID from the Grafana UI URL: http://<grafana-host>:3000/connections/datasources/edit/<uid>
2. In the dashboard JSON, replace all occurrences of "uid": "${DS_HUATUO-BAMAI-ES}" with your actual UID
3. Re-import the dashboard

Prometheus target shows “down”

• Verify huatuo-bamai is running: curl http://<host>:19704/metrics
• Check Prometheus configuration matches the agent’s address and port
• For Kubernetes: ensure pod annotations are correct or ServiceMonitor selector matches

Elasticsearch index is empty

• Verify Elasticsearch is reachable: curl -u elastic:<password> http://<host>:9200/_cat/indices
• Check huatuo-bamai config [Storage.ES] section has correct Address, Username, Password
• Default index name is huatuo_bamai

“socket path already exists” on startup

This occurs when a previous huatuo-bamai process was not cleanly stopped.

Solution:

rm -f /var/run/huatuo-toolstream.sock

3 - Events Watch

📖 Overview

/v1/events/watch is HUATUO’s real-time kernel event subscription endpoint. A single HTTP POST long-lived connection streams kernel anomaly events from the node continuously. Events are wrapped in the CloudEvents 1.0 specification and delivered via the Server-Sent Events (SSE) protocol.

🎯 Use Cases

Kernel event subscription surfaces OS-level anomaly signals directly to higher-level systems, eliminating the latency and overhead of traditional polling. The following are typical integration scenarios.

Fault Self-Healing

Kernel events are the primary signal source for self-healing decisions. After subscribing to events/watch, a healing controller can trigger remediation the moment an event occurs, without waiting for an alert to propagate through a monitoring pipeline:

• OOM self-healing: On receiving an oom event, immediately scale, restart, or drain traffic from the triggering container. Reduces service interruption from minutes to seconds.
• Hung task self-healing: On receiving a hungtask event, automatically cordon the node and evict Pods to prevent cascading blockage from spreading across the cluster.
• Network fault self-healing: On receiving a netdev_txqueue_timeout or netdev_bonding_lacp event, trigger a NIC reset or traffic failover to restore the network link within minutes.
• I/O storm self-healing: On receiving an iotracing event, dynamically throttle the affected container’s disk I/O quota via cgroup blkio to protect co-located services on the same node.

Observability Platforms

Integrating HUATUO kernel events into an observability platform adds a kernel-level perspective beyond application metrics and logs:

• Event timeline correlation: Overlay softlockup, oom, and other kernel events onto Grafana timelines, aligning them precisely with application error rates and latency curves for root-cause analysis.
• Anomaly-driven alerting: Replace fixed-threshold alerts with kernel events to reduce false positives. For example, a ras hardware error event triggers a high-priority alert directly, without relying on a CPU error rate crossing a threshold.
• Capacity and stability analysis: Subscribe to memburst, dload, and other AutoTracing events over time to establish a node stability baseline and provide kernel-level data for capacity planning.
• Multi-dimensional drill-down: Events carry container ID, namespace, region, and other context fields. Alert links can drill down directly to the corresponding Pod, Node, or Region view.

Security Auditing and Compliance

• Anomalous behavior detection: A cluster of oom, hungtask, or softlockup events outside business peak hours may indicate resource abuse or a malicious workload, triggering a security review workflow.
• Event retention and traceability: Write the CloudEvents stream to a message queue (Kafka, Pulsar) or object storage to satisfy the event retention requirements of security compliance frameworks.

Chaos Engineering and Load Testing

• Fault injection verification: After injecting network latency or memory pressure via a chaos engineering platform, subscribe to net_rx_latency and memburst events in real time to verify the fault is active, replacing manual observation.
• Load test baseline: Subscribe to all events during a load test. The timestamp of the first kernel anomaly event precisely marks the system’s stress threshold.

AIOps

• Event-driven root-cause analysis: Feed kernel events as features into AI/ML models alongside application metrics for multi-dimensional root-cause inference, reducing manual investigation time.
• Predictive maintenance: Model ras hardware errors and netdev_bonding_lacp hardware-layer events to detect anomalies before a device fails completely, triggering proactive migration.
• Intelligent suppression and aggregation: Automatically aggregate similar events within the same time window to avoid alert storms. Deliver a concise root-cause summary to on-call engineers.

💎 Value

🚀 Usage

1. CloudEvents Specification

1.1 CloudEvents 1.0 Envelope Fields

Each pushed event is a JSON object conforming to the CloudEvents 1.0 specification:

1.2 HUATUO Event Payload (WatchEventData)

The data field contains the standard HUATUO event record:

{
  "specversion": "1.0",
  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "source": "/huatuo/node-1/oom",
  "type": "tech.huatuo.kernel.event",
  "datacontenttype": "application/json",
  "time": "2026-05-18T10:23:45.123456789Z",
  "data": {
    "hostname": "node-1",
    "region": "cn-beijing",
    "observed_timestamp": "2026-05-18T10:23:45Z",
    "tracer_name": "oom",
    "tracer_id": "abc123",
    "tracer_run_type": "auto",
    "container_id": "d3f1a2b4c5e6",
    "container_hostname": "app-pod",
    "container_host_namespace": "prod",
    "container_type": "docker",
    "container_qos": "Guaranteed"
  }
}

WatchEventData field reference:

2. Supported Kernel Events

3. POST Request Reference

3.1 Endpoint

POST /v1/events/watch

3.2 Request Headers

Content-Type: application/json

3.3 Request Body

{
  "filters": {
    "tracer_name": "<regex>",
    "hostname": "<regex>",
    "container_hostname": "<regex>",
    "container_host_namespace": "<regex>",
    "region": "<regex>"
  }
}

filters field reference:

• All filter fields are optional. Omitting or leaving a field empty matches all values.
• When multiple fields are specified, all conditions must be satisfied simultaneously (AND semantics).
• Filters are evaluated server-side; only matching events are pushed to the client.

3.4 Response Format (SSE Stream)

After the connection is established, the server continuously pushes events in SSE format:

data: {"specversion":"1.0","id":"...","source":"/huatuo/node-1/oom",...}\n\n

The server also sends periodic heartbeat comment lines to keep the connection alive:

: ping\n

4. HTTP Server Event Stream Configuration

Configure the event stream controls under [HTTPServer]:

[HTTPServer]
    # Maximum number of concurrent client connections. New connections receive HTTP 429 when the limit is reached.
    # Default: 100
    MaxEventStreamClients = 100

    # SSE heartbeat interval in seconds. Prevents proxies and load balancers from closing idle connections.
    # The connection is closed after three consecutive heartbeat write failures.
    # Default: 30
    EventStreamKeepAliveIntervalSeconds = 30

5. curl Examples

5.1 Subscribe to All Kernel Events

curl -s -N -X POST http://<node-ip>:19704/v1/events/watch \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -H "Cache-Control: no-cache" \
  -H "Connection: keep-alive" \
  -d '{}'

5.2 Subscribe to OOM Events Only

curl -s -N -X POST http://<node-ip>:19704/v1/events/watch \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -H "Cache-Control: no-cache" \
  -H "Connection: keep-alive" \
  -d '{"filters": {"tracer_name": "^oom$"}}'

5.3 Subscribe to Network Events on a Specific Node

curl -s -N -X POST http://<node-ip>:19704/v1/events/watch \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -H "Cache-Control: no-cache" \
  -H "Connection: keep-alive" \
  -d '{
    "filters": {
      "hostname": "^node-1$",
      "tracer_name": "netdev|dropwatch|net_rx_latency"
    }
  }'

5.4 Subscribe to Container Events in the prod Namespace

curl -s -N -X POST http://<node-ip>:19704/v1/events/watch \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -H "Cache-Control: no-cache" \
  -H "Connection: keep-alive" \
  -d '{
    "filters": {
      "container_host_namespace": "^prod$"
    }
  }'

Note: The -N flag disables curl buffering, causing SSE events to be printed to the terminal immediately.

6. Go Client Example

The following example shows how to subscribe to the events/watch endpoint in a Go program and consume CloudEvents in real time.

package main

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"strings"
	"time"
)

// WatchRequest is the request body sent to /v1/events/watch.
type WatchRequest struct {
	Filters WatchFilters `json:"filters"`
}

type WatchFilters struct {
	TracerName             string `json:"tracer_name,omitempty"`
	Hostname               string `json:"hostname,omitempty"`
	ContainerHostname      string `json:"container_hostname,omitempty"`
	ContainerHostNamespace string `json:"container_host_namespace,omitempty"`
	Region                 string `json:"region,omitempty"`
}

// WatchEvent is the CloudEvents 1.0 envelope pushed by HUATUO.
type WatchEvent struct {
	SpecVersion     string          `json:"specversion"`
	ID              string          `json:"id"`
	Source          string          `json:"source"`
	Type            string          `json:"type"`
	DataContentType string          `json:"datacontenttype"`
	Time            string          `json:"time"`
	Data            json.RawMessage `json:"data"`
}

func watchEvents(ctx context.Context, endpoint string, filters WatchFilters) error {
	reqBody, err := json.Marshal(WatchRequest{Filters: filters})
	if err != nil {
		return fmt.Errorf("marshal request: %w", err)
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(reqBody))
	if err != nil {
		return fmt.Errorf("create request: %w", err)
	}
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Accept", "text/event-stream")

	client := &http.Client{Timeout: 0} // no timeout for SSE long-lived connections
	resp, err := client.Do(req)
	if err != nil {
		return fmt.Errorf("connect: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return fmt.Errorf("unexpected status: %d", resp.StatusCode)
	}

	scanner := bufio.NewScanner(resp.Body)
	for scanner.Scan() {
		line := scanner.Text()

		// skip heartbeat comment lines and blank lines
		if line == "" || strings.HasPrefix(line, ":") {
			continue
		}

		// SSE data line format: `data: <json>`
		data, ok := strings.CutPrefix(line, "data: ")
		if !ok {
			continue
		}

		var event WatchEvent
		if err := json.Unmarshal([]byte(data), &event); err != nil {
			slog.Warn("parse event", "err", err)
			continue
		}

		fmt.Printf("[%s] source=%s id=%s\n", event.Time, event.Source, event.ID)
		fmt.Printf("  data: %s\n", event.Data)
	}

	return scanner.Err()
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	defer cancel()

	err := watchEvents(ctx, "http://192.168.1.10:19704/v1/events/watch", WatchFilters{
		TracerName: "oom|hungtask|softlockup",
	})
	if err != nil {
		slog.Error("watch events", "err", err)
		os.Exit(1)
	}
}

6.1 Using the Official pkg/types Package (Recommended)

If your project shares the same Go module as HUATUO, use the official types directly:

import pkgtypes "huatuo-bamai/pkg/types"

var event pkgtypes.WatchEvent
if err := json.Unmarshal([]byte(data), &event); err != nil { ... }

// WatchEvent.Data is json.RawMessage (deferred parsing); a second unmarshal is required to access typed fields
dataBytes, err := json.Marshal(event.Data)
if err != nil {
    slog.Warn("marshal event data", "err", err)
    return
}
var payload pkgtypes.WatchEventData
if err := json.Unmarshal(dataBytes, &payload); err != nil {
    slog.Warn("unmarshal event data", "err", err)
    return
}
fmt.Println("tracer:", payload.TracerName)
fmt.Println("observed_timestamp:", payload.ObservedTimestamp)

6.2 Reconnection

In production, network interruptions or service restarts will drop the connection. Use exponential backoff to reconnect:

func watchWithRetry(ctx context.Context, endpoint string, filters WatchFilters) {
	backoff := time.Second
	for {
		if err := watchEvents(ctx, endpoint, filters); err != nil {
			if ctx.Err() != nil {
				return
			}
			slog.Warn("disconnected, retrying", "err", err, "backoff", backoff)
			// time.NewTimer + Stop releases the timer immediately when the context is cancelled
			timer := time.NewTimer(backoff)
			select {
			case <-ctx.Done():
				timer.Stop()
				return
			case <-timer.C:
			}
			if backoff < 30*time.Second {
				backoff *= 2
			}
		}
	}
}

⚙️ How It Works

Architecture

HUATUO Agent runs on each node. It hooks into critical kernel paths via eBPF, Kprobe, and Tracepoint, collects kernel anomaly events, applies filters, wraps them as CloudEvents, and pushes them to multiple concurrent SSE subscribers.

graph TB
    subgraph kernel["Linux Kernel"]
        K1[OOM Killer]
        K2[Hung Task Detection]
        K3[Soft Lockup Detection]
        K4[RAS Hardware Errors]
        K5[Network Subsystem]
        K6[AutoTracing]
    end

    subgraph huatuo["HUATUO Agent (per node)"]
        T["Tracer Collection Layer\neBPF / Kprobe / Tracepoint"]
        F["Filter\nhostname / tracer / namespace / region"]
        CE["CloudEvents 1.0 Wrapper\nid / source / time / data"]
        EW["EventsWatch Dispatcher\nSSE connection management"]
    end

    subgraph clients["Subscribers"]
        C1[Fault Self-Healing System]
        C2[Observability Platform]
        C3[AIOps System]
        C4[Security Audit System]
    end

    kernel --> T
    T --> F
    F --> CE
    CE --> EW
    EW -->|SSE push| C1
    EW -->|SSE push| C2
    EW -->|SSE push| C3
    EW -->|SSE push| C4

Event Collection and Push

After the client issues a POST request, the connection stays open. Each time the kernel triggers an anomaly event, HUATUO Agent filters and wraps it, then writes it immediately to all matching SSE streams. No client polling is required.

sequenceDiagram
    participant C as Client
    participant EW as EventsWatch
    participant T as Tracer Layer
    participant K as Linux Kernel

    C->>EW: POST /v1/events/watch {"filters": {...}}
    EW-->>C: 200 OK (Content-Type: text/event-stream)

    loop SSE long-lived connection
        K->>T: Kernel event triggered (oom / hungtask / softlockup ...)
        T->>EW: Report raw event
        EW->>EW: Apply filter
        alt Filter matched
            EW-->>C: data: {CloudEvents JSON}\n\n
        else No match
            note over EW: Discard, do not push
        end
        EW-->>C: : ping (keepalive, configured interval)
    end

Event Processing Pipeline

From kernel event generation to client delivery, three stages are involved: collection, filtering, and wrapping. End-to-end latency is under 1 second.

flowchart LR
    A([Kernel anomaly triggered]) --> B["Tracer collection\neBPF / Kprobe"]
    B --> C{Filter matched?}
    C -- No --> D([Discard])
    C -- Yes --> E["Wrap as CloudEvents 1.0\nid / source / time / data"]
    E --> F[Write to SSE stream]
    F --> G([Push to subscribers])

🌟 Stay Connected

🌟 Star us on GitHub: https://github.com/ccfos/huatuo

👀 Follow our official WeChat public account

4 - Profiling

Flame Graph Formats

In profiling, collapsed and flamegraph are the two most common formats, corresponding to the “raw data” and “visual view” layers respectively.

Collapsed Format

Standard Syntax and Format

The collapsed format (also called folded stacks) was defined by Brendan Gregg and serves as the raw text input format for flame graphs. Each line represents a unique call stack and its sample count.

Basic rule:

frame1;frame2;frame3;...;frameN COUNT

Format details:

• One unique call stack per line; samples with the same stack path have their counts merged
• Frame order: left to right is root → leaf (call chain direction)
• Blank lines and lines starting with # are treated as comments and ignored during parsing
• The semantics of COUNT depend on the analysis mode: for CPU sampling it is the number of samples, for memory allocation it is the number of bytes allocated, for lock analysis it is the contention time in milliseconds

Extended specification:

Some profiling tools (e.g. async-profiler) add frame type annotations on top of the standard format to identify the runtime category of a frame:

frameName_{type} COUNT

Additionally, some tools support a weighted collapsed format for differential flame graphs:

frame1;frame2;frameN WEIGHT

Where WEIGHT is a floating-point number representing the weight of the stack rather than a simple count.

Sample Examples

CPU profiling example (data from the async-profiler official documentation):

FileConverter.main;FileConverter.convertFile;FileConverter.saveResult 21
FileConverter.main;FileConverter.convertFile;FileConverter.saveResult;java/io/DataOutputStream.writeInt 1
FileConverter.main;FileConverter.convertFile;FileConverter.saveResult;java/io/DataOutputStream.writeInt;java/io/ByteArrayOutputStream.write 5
FileConverter.main;FileConverter.convertFile;FileConverter.saveResult;java/io/DataOutputStream.writeUTF;java/io/DataOutputStream.writeUTF 12
FileConverter.main;FileConverter.convertFile;FileConverter.saveResult;java/io/DataOutputStream.writeUTF;java/io/DataOutputStream.writeUTF;java/lang/String.length 3
FileConverter.main;FileConverter.convertFile;FileConverter.saveResult;java/io/DataOutputStream.writeUTF;java/io/DataOutputStream.writeUTF;java/io/DataOutputStream.write 6
start_thread;thread_native_entry;Thread::call_run;VMThread::run;VMThread::inner_execute;VMThread::evaluate_operation;VM_Operation::evaluate;VM_GenCollectForAllocation::doit;GenCollectedHeap::satisfy_failed_allocation;GenCollectedHeap::do_collection;GenCollectedHeap::collect_generation;DefNewGeneration::collect;DefNewGeneration::FastEvacuateFollowersClosure::do_void 12

Example with frame type annotations (async-profiler extension):

Main.run_[j];Service.process_[j];DAO.query_[j];mysql_real_query_[n] 45
Main.run_[j];Service.process_[j];DAO.query_[j];recv_[k] 18

Core Use Cases

Generation command example:

# Using async-profiler as an example
asprof -d 30 -f profile.collapsed -o collapsed <PID>

Flamegraph Format

Standard Syntax and Format

The flamegraph format is a self-contained HTML file with embedded SVG visualization and JavaScript interaction logic, which can be opened directly in a browser.

Structural composition:

flamegraph.html
├── HTML skeleton + CSS styles
├── SVG flame graph body
│   ├── <g> rectangle block for each frame
│   │   ├── <title> frame name + sample count/percentage
│   │   └── <rect> position, width, height, color
│   └── ...
├── JavaScript interaction logic
│   ├── Click to zoom (zoom into subtree)
│   ├── Search & highlight
│   ├── Tooltip on hover
│   └── Reset zoom
└── Metadata (title, total samples, etc.)

Visual encoding rules:

Frame color specification (based on async-profiler):

Note: Flame graph color schemes are not a cross-tool standard. The original flamegraph.pl by Brendan Gregg uses random warm tones with no semantic meaning; perf/bpftrace typically colors by DSO or uses random colors; async-profiler colors by frame type semantics. The following is the async-profiler color specification:

Extended features (based on async-profiler):

• Icicle Graph: Displays the call chain top-down (root at the top), which better suits top-down reading habits. Toggle via the --reverse option or the Reverse button in the browser
• Multi-thread view: Call stacks from different threads are displayed side by side at the root level
• Search highlighting: Matching frames are highlighted in purple; non-matching frames are dimmed
• Sample info tooltip: Hover to display frame name, sample count, and percentage of total samples
• Cutoff frames: Frames marked as [...] indicate stack truncation (e.g. due to stack depth limits)

Sample Examples

Generation command example:

# Using async-profiler as an example
asprof -d 30 -f flamegraph.html <PID>

Interactive operations:

• Click a frame: Zoom to make the frame full-width, showing only its subtree
• Search box: Enter a keyword; matching frames are highlighted
• Hover: Display frame name, sample count, and percentage
• Reset Zoom: Restore the global view

Core Use Cases

Format Comparison

Typical workflow:

Collect ──► collapsed ──► flamegraph.html (manual analysis)
                   │
                   ├──► Differential flame graph (regression detection)
                   ├──► Custom aggregation scripts
                   └──► Archive storage

5 - Network Drop Monitoring

Overview

dropwatch observes software drops through tracepoint/skb/kfree_skb and hardware drops reported by capable drivers through raw_tracepoint/devlink_trap_report. It outputs protocol fields, the IP tuple, network device, drop reason, and kernel stack.

dropwatch supports kernel-side filtering based on tcpdump-style filter expressions. The filter logic is compiled into eBPF bytecode at load time by the built-in pure-Go pcap compiler internal/pcapfilter. Filtering is performed entirely in kernel mode — only matching packets are reported to user space, reducing performance impact on the host.

In addition, dropwatch supports device whitelist/blacklist filtering, global per-second rate limiting, and integration with huatuo-bamai to store drop events in Elasticsearch for long-term analysis.

Scenarios

1. Kubernetes Cloud-Native Network Drop Diagnosis

In scenarios such as container migration, frequent Pod restarts, and Service port conflicts, dropwatch captures kfree_skb events in real time and correlates them with specific containers to quickly identify the root cause of packet drops. Combined with --filter "tcp and port <service-port>" to filter specific business traffic, the mean time to root cause is reduced from hours to minutes.

2. Network Performance Spike Analysis

For intermittent spikes in network latency or drops in throughput, dropwatch collects drop events and, together with the kernel call stack, identifies the specific kernel function where the drop occurred (e.g. tcp_v4_rcv, ip_output). This helps distinguish whether the cause is a firewall drop, routing failure, buffer overflow, or other reasons.

3. Multi-Tenant Network Isolation Troubleshooting

In container environments that share network namespaces or veth devices, use --device to filter by network device and --filter to filter by protocol. This precisely captures drop events for the target container, preventing other tenants’ traffic from interfering with the diagnosis.

4. Observability Platform Integration

Use --output-storage to send drop events to huatuo-bamai, which stores them in Elasticsearch for multi-dimensional correlation with metrics and logs. Overlay drop events on a Grafana timeline, aligned with application error rates and latency curves, to correlate kernel drops with application anomalies precisely.

Usage

1. Filter Expressions

Filter expressions use tcpdump syntax. The built-in pure-Go pcap compiler internal/pcapfilter compiles them into eBPF bytecode at load time. Filtering is performed entirely in kernel mode, reducing host impact — only matching packets are reported to user space.

1.1 Supported Expressions

internal/pcapfilter supports a subset of the standard tcpdump syntax. The following primitives are reliable:

Protocols

ip   ip6   tcp   udp   icmp   icmp6   igmp   pim   esp   ah   vrrp   arp   rarp
ip proto tcp      ip6 proto udp        (protocol names only; numeric protocol numbers not supported)

Host addresses

host 10.0.0.1
src host 10.0.0.1
dst host 10.0.0.1

Ports

port 80
src port 443
dst port 8080

Networks (CIDR)

net 10.0.0.0/8
src net 192.168.1.0/24
dst net 172.16.0.0/12

Multicast and Ethernet addresses

ip multicast    ip6 multicast    multicast    ether multicast
ether host 00:11:22:33:44:55

Boolean operators and grouping

tcp and port 80
tcp or udp
not arp
tcp and (port 80 or port 443)
ip and src net 192.168.1.0/24 and tcp dst port 3306

1.2 Unsupported Expressions

The following expressions are not supported. Using them causes compilation failures or incorrect match results:

1.3 Examples

# Monitor all TCP drops (default — reliable in both L2 and L3 contexts)
--filter "tcp"

# TCP and UDP
--filter "tcp or udp"

# Specific destination host (applies to both TCP and UDP)
--filter "dst host 10.0.0.1"

# Specific port
--filter "tcp and port 443"

# Exclude a noisy host
--filter "tcp and not host 169.254.169.254"

# Specific subnet + specific port
--filter "src net 192.168.1.0/24 and tcp dst port 3306"

# Monitor non-TCP drops (UDP and ICMP only — avoid "not tcp", which captures unknown L3 events)
--filter "udp or icmp"

# Monitor ARP drops only (effective only in L2 context; never matches at L3)
--filter "arp"

--filter "ip" / --filter "ip6" now correctly match the corresponding IP protocol family (L2 by EtherType, L3 by version nibble). If you only care about a specific transport layer or host, prefer the more precise tcp, udp, host, or ip proto <name>.

2. Running dropwatch

dropwatch [flags]

--filter and device filtering are orthogonal; when both are specified, both apply (AND semantics). If neither --device nor --device-excluded is specified, all devices are collected. --device and --device-excluded are mutually exclusive; whitelist mode drops SKBs without a net_device, while blacklist mode passes them.

At startup, dropwatch detects devlink:devlink_trap_report. When supported, it loads both software and hardware drop probes. Otherwise, it logs a warning and loads only the software drop probe. Hardware collection also requires a driver that registers devlink drop traps and a target trap whose action is trap. With action drop, hardware sends no packet copy to the CPU, so dropwatch cannot inspect it.

devlink Hardware Drop Detection

dropwatch requires no additional startup flags. Before using hardware drop detection, verify that the kernel, driver, and target trap meet the requirements:

# 1. Verify that the kernel provides the devlink trap tracepoint
test -e /sys/kernel/tracing/events/devlink/devlink_trap_report/id || \
  test -e /sys/kernel/debug/tracing/events/devlink/devlink_trap_report/id

# 2. List devlink devices and traps registered by the driver
sudo devlink dev show
sudo devlink trap show <bus/device>

# 3. Enable packet reporting for the target DROP trap
sudo devlink trap set <bus/device> trap <trap-name> action trap

# 4. Start dropwatch and display only hardware drops
sudo dropwatch --bpf-path bpf/dropwatch.o --output json 2>/dev/null | \
  jq -c 'select(.drop_source == "hardware")'

<bus/device> is the device identifier returned by devlink dev show, such as pci/0000:03:00.0. After diagnosis, restore the trap to its previous action.

This capability collects only packets that the driver reports through DEVLINK_TRAP_TYPE_DROP. It does not capture all hardware packets and does not replace NIC hardware-drop counters. Drops such as hardware queue overflows are visible only when the driver implements and reports them as devlink drop traps. Traps of type exception or control are not reported as drop events.

--filter, --device, --device-excluded, and --max-events-per-second apply to both software and hardware events. Text output formats a hardware reason as reason=<group>/<trap> drop_source=hardware. JSON output uses the separate drop_reason_group, drop_reason, and drop_source fields.

Examples

# Text output, monitor TCP drops on all devices
sudo dropwatch --bpf-path bpf/dropwatch.o --filter "tcp"

# Monitor drops on eth0 only
sudo dropwatch --bpf-path bpf/dropwatch.o --device eth0 --output json

# Exclude loopback
sudo dropwatch --bpf-path bpf/dropwatch.o --device-excluded lo --output json

# Combine device and protocol filters
sudo dropwatch --bpf-path bpf/dropwatch.o --device eth0 --filter "tcp and port 443" --output json

# Capture for 60 seconds and exit
sudo dropwatch --bpf-path bpf/dropwatch.o --filter "tcp and port 443" --duration 60 --output json

# Forward events to a running huatuo-bamai instance
sudo dropwatch --bpf-path bpf/dropwatch.o --filter "tcp" --output-storage /var/run/huatuo-toolstream.sock

# Use jq to filter and show only RST packets
sudo dropwatch --bpf-path bpf/dropwatch.o --output json 2>/dev/null | jq 'select(.layers.tcp.flags == "RST")'

# Capture 10 seconds of JSON output, excluding events whose stack contains ip_finish_output
sudo dropwatch --output json --duration 10 --bpf-path bpf/dropwatch.o | jq -c 'select(.stack | test("ip_finish_output") | not)'

# Capture 10 seconds of JSON output, printing all fields except stack
sudo dropwatch --output json --duration 10 --bpf-path bpf/dropwatch.o | jq -c 'del(.stack)'

jq -c compresses each matching event into a single-line JSON, convenient for saving as NDJSON or further pipe processing. test("ip_finish_output") checks whether stack matches the regex; not negates the result, so the command above excludes stacks containing ip_finish_output. Remove | not to keep only those containing ip_finish_output. del(.stack) removes the stack field from the jq output, useful for viewing just the timestamp, device, process, packet_* metadata, and layers protocol fields. For userspace call-stack filtering before storage, configure EventTracing.IssuesList in huatuo-bamai (see Section 4).

3. Event Data Structure

Each drop event is represented as an NDJSON object (types.DropWatchTracing).

For hardware events, stack is the kernel call stack at which the driver reports the devlink trap. It does not identify the actual drop location inside the ASIC. Use drop_reason_group, drop_reason, device information, and driver documentation to diagnose hardware drops.

layers uses fixed fields to express the protocol stack, without relying on a separate protocol enumeration:

4. Integration with huatuo-bamai

huatuo-bamai launches dropwatch as a subprocess and uses --output-storage to send events to the built-in processing pipeline, which ultimately stores them in Elasticsearch. Typical parameters:

dropwatch \
  --bpf-path <CoreBpfDir>/dropwatch.o \
  --output-storage /var/run/huatuo-toolstream.sock \
  --filter "tcp"

4.1 Configuration Reference (huatuo-bamai.conf)

[EventTracing]
    # Optional call-stack filters. dropwatch discards events whose stack matches a configured regex.
    # Default: []
    IssuesList = []

[EventTracing.Dropwatch]
    # tcpdump filter expression, forwarded to dropwatch --filter.
    # Default: "tcp"
    Filter = "tcp"

    # Forwarded to dropwatch --max-events-per-second.
    # Default: 100
    MaxEventsPerSecond = 100

4.2 Noise Filtering

No call-stack noise rule is enabled by default. When EventTracing.IssuesList is configured, huatuo-bamai discards matching events. The following patterns are possible operator-configured filters; validate them against the local kernel and workload before enabling them:

Closing

6 - TCP Retransmission Tracing

Overview

tcpshark --mode retransmit observes TCP retransmission-related kernel activity through the tcp/tcp_retransmit_skb and tcp/tcp_retransmit_synack tracepoints. It can also observe the tcp_send_loss_probe kprobe when TLP collection is explicitly enabled. Depending on the event type, an event can include the IP 4-tuple, TCP state, congestion-control state, retransmission counters, sequence information, and socket metadata used for container resolution.

The userspace classifier derives a connection phase and a reason label from the event type, sk_state, ca_state, and reorder counters. These labels are operational heuristics, not packet-loss root-cause proof.

Filter expressions are compiled at load time by internal/pcapfilter and run in the kernel. Filters apply only to events that have an SKB (tcp_retransmit_skb); SYN-ACK and TLP events bypass the pcap filter.

Scenarios

1. TCP Network Quality and Retransmission Diagnosis

Continuously observe RTO, fast retransmission, reorder-prone retransmission, and TLP events to identify abnormal retransmissions during connection establishment, data transfer, and connection teardown. These signals help investigate packet loss, congestion, reordering, and peer reachability problems.

2. Kubernetes Container Network Troubleshooting

Use the container ID, network namespace, and socket cgroup metadata to identify the workload experiencing retransmissions. Apply --filter "tcp and port <service-port>" to focus on a specific service and reduce interference from other host connections.

3. Application Latency and Throughput Anomaly Analysis

Align TCP retransmission events with application latency, error-rate, and throughput timelines. This helps determine whether RTOs or repeated retransmissions coincide with service degradation and distinguish slow application processing from underlying network problems.

4. Locating Packet Loss with dropwatch Correlation

Run dropwatch and tcp_retransmit in the same huatuo-bamai process to correlate packet drops with retransmissions by SKB pointer or connection 4-tuple. The result helps indicate whether the problem is more likely in the host network stack or the external network, but remains heuristic evidence that should be validated with stack traces and network metrics.

Usage

1. Running tcpshark

tcpshark --mode retransmit [flags]

When both --output and --output-storage are explicitly specified, --output is ignored and a warning is printed.

1.1 Examples

# Text output for all retransmission-related events
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o

# NDJSON output
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json

# BPF-side filter for regular retransmitted SKBs to one destination host and port
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --filter "dst host 10.0.0.1 and dst port 443"

# Include Tail Loss Probe events (disabled by default)
sudo tcpshark --mode retransmit --enable-tlp --bpf-path bpf/tcp_retransmit.o

# Emit at most 100 events/second; overflow prints a rate limit hit log
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --max-events-per-second 100

# Filter all formatted event types to destination port 443 in userspace
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json \
  | jq -c 'select(.tcp_dport == 443)'

# Keep only events classified as RTO for 60 seconds
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --duration 60 --output json \
  | jq -c 'select(.tcp_reason == "RTO")'

# Forward events to a running huatuo-bamai instance
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --output-storage /var/run/huatuo-toolstream.sock

jq -c emits compact single-line JSON, which is convenient for NDJSON files and downstream pipelines.

1.2 Integration with huatuo-bamai

tcpshark uses the same --output-storage and toolstream flow as dropwatch. For the common storage workflow, refer to the dropwatch documentation. TCP retransmission tracing adds the following configuration:

[EventTracing.TCPRetransmit]
    # Forwarded to tcpshark --filter; applies only to tcp_retransmit_skb.
    # Default: ""
    Filter = ""

    # Forwarded as tcpshark --enable-tlp. Default: false.
    EnableTLP = false

    # Forwarded as tcpshark --max-events-per-second. Default: 100; 0 disables it.
    MaxEventsPerSecond = 100

The tcp_retransmit tracer is in the global BlackList by default. Remove it from the list and restart huatuo-bamai to enable the tracer. Its drop-correlation cache is enabled only while the tracer is running and is cleared when the tracer stops. After enabling it, use the HTTP API to start or stop tracing:

curl -X PUT http://localhost:19704/tracers/tcp_retransmit/start
curl -X PUT http://localhost:19704/tracers/tcp_retransmit/stop

2. Filter Expressions

tcpshark uses the same tcpdump-style filter expressions as dropwatch. For complete syntax, limitations, and additional examples, refer to the dropwatch documentation.

# Select one destination host and port
--filter "dst host 10.0.0.1 and dst port 443"

# Select traffic in both directions between two networks
--filter "(src net 10.10.0.0/16 and dst net 10.20.0.0/16) or (src net 10.20.0.0/16 and dst net 10.10.0.0/16)"

--filter applies only to tcp_retransmit_skb. The tcp_retransmit_synack and enabled tcp_send_loss_probe events have no SKB and bypass the filter.

3. Event Data Structure

Each event is an NDJSON object (types.TCPRetransmitTracing). Fields tagged with omitempty are absent when their value is empty or zero.

icsk_pending is a timer-state snapshot at the hook, not a stable retransmission-reason enum. TLP classification uses the explicit event_type=tcp_send_loss_probe and does not depend on icsk_pending=5.

3.1 Text Output Format

Text retains its terminal-friendly layout while covering the same event variables as JSON. Variables tagged with omitempty appear only when non-zero or non-empty, and string values are not JSON-quoted or escaped. For compatibility with the original text format, state, skb, seq, end, ack, flags, ca, and retrans correspond to the JSON fields tcp_state, skb_addr, tcp_seq, tcp_end_seq, tcp_ack_seq, tcp_flags, ca_state, and icsk_retransmits, respectively.

<timestamp> [<phase>/<tcp_reason>] <saddr>:<sport> > <daddr>:<dport> state=<STATE> event_type=<TYPE> [SYNACK] [skb=<ADDR>] seq=<N> [end=<N>] ack=<N> [flags=<FLAGS>] pid=<N> comm=<COMM> ca=<N> retrans=<N> icsk_pending=<N> [reord_seen=<N>] [dsack_dups=<N>] [container_id=<ID>] [memory_cgroup_css_addr=<ADDR>] [net_namespace_cookie=<N>] [net_namespace_inum=<N>] [drop_location=<LOCATION>] [source=<SOURCE>]

Example:

2026-07-23T02:14:40.304775546Z [data/RTO] 127.0.0.1:19996 > 127.0.0.1:42128 state=ESTABLISHED event_type=tcp_retransmit_skb skb=0xffff931c14fdf800 seq=3154974646 end=3154991030 ack=948393597 flags=ACK|PSH pid=1420 comm=kube-apiserver ca=4 retrans=4 icsk_pending=0 net_namespace_inum=4026531992

The pid and comm in this example describe the execution context in which the hook ran; use container_id and socket metadata for workload attribution.

3.2 Container ID Resolution

tcpshark cannot access the Pod manager directly. In standalone output, container_id is normally absent, while socket memcg and network-namespace metadata are still emitted when available. In huatuo-bamai mode, an empty container_id is resolved in this order: memory_cgroup_css_addr, net_namespace_cookie, then net_namespace_inum.

If all lookups miss, the event is still stored without container_id. Do not use pid or comm as a fallback for socket ownership because they describe the hook execution context.

4. Kernel Events and Classification

4.1 Kernel Hook Points

The BPF program uses CO-RE field reads (BPF_CORE_READ and related helpers), so supported kernel layouts do not require rebuilding the C source for each kernel version.

4.2 Connection Phase

The regular-SKB phase is derived from sk_state. SYN-ACK events use a fixed phase in userspace.

The TCP three-way handshake below shows the connect phase and its retransmission hook points:

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: Initial states: CLOSED / LISTEN
    C->>S: ① SYN
    Note left of C: SYN_SENT(2)<br/>phase=connect
    opt SYN is not acknowledged
        C-->>S: SYN retransmission<br/>tcp_retransmit_skb
    end
    Note right of S: SYN_RECV(3) or NEW_SYN_RECV(12)<br/>phase=connect
    S->>C: ② SYN + ACK
    opt Final ACK does not arrive
        S-->>C: SYN-ACK retransmission<br/>tcp_retransmit_synack
    end
    C->>S: ③ ACK
    Note over C,S: ESTABLISHED(1)<br/>subsequent regular data-SKB events use phase=data

The three solid arrows are the initial handshake packets and do not produce tcpshark events. Only the retransmission paths inside the optional blocks are observed. Active-open SYN retries are reported by tcp_retransmit_skb, while passive-open SYN-ACK retries are reported by tcp_retransmit_synack; both are classified as connect.

The complete phase mapping is:

4.3 Reason Classification

The classifier observes socket state at the hook and cannot reconstruct the complete ACK/loss history. Treat tcp_reason as a grouping label rather than a verified root cause.

4.4 Reorder Heuristic

The reorder-prone label is selected when either reord_seen or dsack_dups is non-zero. Once a flow has reorder history, subsequent Recovery-state SKB events can be labeled reorder_prone_fast. This is a flow-level heuristic, not proof that the current retransmission was caused by reordering.

4.5 Operational Guidance

No event type is unconditionally safe to discard. Prefer rate, ratio, and service-impact thresholds over filtering solely by event_type or tcp_reason. For the common huatuo-bamai noise-filtering mechanism, refer to the dropwatch documentation.

When building alerts, aggregate by service or connection and compare against traffic volume. A small absolute count on a busy host can be benign, while a burst affecting a low-volume critical service can be significant.

5. Correlation with dropwatch

When dropwatch and tcpshark feed the same huatuo-bamai process, dropwatch events are retained in a userspace cache for two seconds from their arrival time. A tcpshark event immediately queries previously received, unexpired drop events using a direction-independent connection key. The implementation does not wait for later drop events and does not revise an event after storage.

5.1 Correlation Results

network_or_host_hardware can also be produced when dropwatch is disabled, its filter does not cover the flow, an event is suppressed or lost, delivery is reordered, or the relevant drop falls outside the retention window. Likewise, a 4-tuple match can pair unrelated packets from a busy connection. The cache key does not include a network-namespace or container identifier, so identical address/port tuples in different network namespaces can also collide.

5.2 Requirements and Troubleshooting

For reliable negative evidence, dropwatch must be active with a filter that is at least as broad as the tcpshark traffic of interest. The current schema has no separate unknown or dropwatch_not_observed value, so consumers should treat network_or_host_hardware as an investigation hint rather than a fact.

Closing