Best Practice on HUATUO Documentation

Storage Service

Mon, 27 Jul 2026 00:00:00 +0000

🎯 About HUATUO

HUATUO is an open-source OS-level deep observability project initiated by DiDi and incubated by the CCF (China Computer Federation). It provides kernel-level observability for cloud-native computing, AI computing, cloud services, and foundational infrastructure.

📖 Overview

HUATUO supports persisting Linux kernel events collected by the Tracer and AutoTracing data to external storage backends. Both Elasticsearch and OpenSearch are supported.

After serialization to JSON, collected events are written concurrently to the local node directory (huatuo-local/) and the configured remote storage backend. The local directory retains a local copy of events; the remote backend provides durable storage and structured query capabilities.

Data Source

Tue, 05 May 2026 00:00:00 +0000

HUATUO integrates with Prometheus for metrics collection and Elasticsearch for log storage. This document covers data source configuration and dashboard provisioning in Grafana.

Two deployment paths are supported:

Docker Compose — recommended for development and testing; all components are pre-configured.
Kubernetes — for production clusters; requires manual data source configuration.

Quick Start (Docker Compose)

The build/docker/ directory contains a complete stack. All default credentials and ports listed below match this setup.

Events Watch

Mon, 27 Jul 2026 00:00:00 +0000

🎯 About HUATUO

HUATUO is an operating system observability project open-sourced by DiDi and incubated under CCF (China Computer Federation). It provides kernel-level deep observability for cloud-native general computing, AI computing, cloud services, and foundational services.

📖 Overview

/v1/events/watch is HUATUO’s real-time kernel event subscription endpoint. A single HTTP POST long-lived connection streams kernel anomaly events from the node continuously. Events are wrapped in the CloudEvents 1.0 specification and delivered via the Server-Sent Events (SSE) protocol.

Profiling

Thu, 18 Jun 2026 00:00:00 +0000

Flame Graph Formats

In profiling, collapsed and flamegraph are the two most common formats, corresponding to the “raw data” and “visual view” layers respectively.

Collapsed Format

Standard Syntax and Format

The collapsed format (also called folded stacks) was defined by Brendan Gregg and serves as the raw text input format for flame graphs. Each line represents a unique call stack and its sample count.

Network Drop Monitoring

Fri, 05 Jun 2026 00:00:00 +0000

About HUATUO

HUATUO is an OS-level deep observability project open-sourced by DiDi and incubated under CCF (China Computer Federation). It provides kernel-level deep observability for cloud-native general computing, AI computing, cloud services, and infrastructure services.

Overview

dropwatch observes software drops through tracepoint/skb/kfree_skb and hardware drops reported by capable drivers through raw_tracepoint/devlink_trap_report. It outputs protocol fields, the IP tuple, network device, drop reason, and kernel stack.

dropwatch supports kernel-side filtering based on tcpdump-style filter expressions. The filter logic is compiled into eBPF bytecode at load time by the built-in pure-Go pcap compiler internal/pcapfilter. Filtering is performed entirely in kernel mode — only matching packets are reported to user space, reducing performance impact on the host.

TCP Retransmission Tracing

Mon, 20 Jul 2026 00:00:00 +0000

About HUATUO

Overview

tcpshark --mode retransmit observes TCP retransmission-related kernel activity through the tcp/tcp_retransmit_skb and tcp/tcp_retransmit_synack tracepoints. It can also observe the tcp_send_loss_probe kprobe when TLP collection is explicitly enabled. Depending on the event type, an event can include the IP 4-tuple, TCP state, congestion-control state, retransmission counters, sequence information, and socket metadata used for container resolution.