Continuous Profiling

🚀 Quick Start

This guide uses build/docker/docker-compose.yml to start all services, create a host CPU profiling job, and view the flame graph in Grafana.

1. Start Services

First, configure Elasticsearch credentials so that huatuo-bamai and huatuo-apiserver can persist profile data. The config files are volume-mounted into the containers, so edit them directly in the project root.

In huatuo-bamai.conf:

[Storage]
    [Storage.Elasticsearch]
        Address = "http://127.0.0.1:9200"
        Index = "huatuo_bamai"
        Username = "elastic"
        Password = "huatuo-bamai"

In huatuo-apiserver.conf:

[Elasticsearch]
    Address = "http://127.0.0.1:9200"
    Username = "elastic"
    Password = "huatuo-bamai"
    Index = "huatuo_bamai"

[Auth]
    [[Auth.Users]]
        ID = "administrator"
        BearerToken = "REPLACE_WITH_RANDOM_HEX"
        Admin = true

Then start all services from the project root:

docker compose --project-directory ./build/docker up

Run without -d to observe startup logs. Add -d for background mode.

2. Verify Services

In a new terminal, confirm all services are ready:

# Agent health check
$ curl -s http://localhost:19704/version | jq .data.name

"huatuo-bamai"

# API Server health check
$ curl -s http://localhost:12740/version | jq .data.name

"huatuo-apiserver"

# ES index status
$ curl -s -u elastic:huatuo-bamai "http://localhost:9200/_cat/indices/huatuo_bamai?v"

health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   huatuo_bamai 147fzHJhQ820GjCKFLh5ZQ   1   1         42            0    297.8kb        297.8kb      297.8kb

Set environment variables for subsequent API calls:

API_BASE="http://127.0.0.1:12740"
API_TOKEN="REPLACE_WITH_RANDOM_HEX"

3. Create a Host CPU Profiling Job

Use c language (native, covers C/C++/Go) to sample the entire host for 30 seconds:

# Use the actual hostname of the node
HOSTNAME=$(hostname)

JOB_ID=$(curl -s -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{
    \"type\": \"cpu\",
    \"language\": \"c\",
    \"duration_seconds\": 30,
    \"hostname\": \"${HOSTNAME}\"
  }" \
  "${API_BASE}/v1/profiles" | jq -r .data.id)

echo "Job ID: $JOB_ID"

4. Verify Data in Elasticsearch

Each aggregation window is 10 seconds. A 30-second job produces approximately 3 profile documents. Wait for completion, then verify:

$ curl -s -u elastic:huatuo-bamai "http://localhost:9200/huatuo_bamai/_count" \
  -H "Content-Type: application/json" \
  -d '{"query":{"exists":{"field":"tracer_data.flamedata"}}}' | jq .count

3

5. View the Flame Graph in Grafana

Open the Continuous Profiling (host) dashboard:

• URL: http://localhost:3000/d/continuous-profiling-host (replace localhost:3000 with your environment)
• Credentials: admin / admin (skip the default password change prompt)

Steps:

1. Select a time range covering the profiling period
2. Choose your hostname and set type to process_cpu:cpu:nanoseconds:cpu:nanoseconds
3. The flame graph loads aggregated call stacks for the selected time range and updates dynamically
4. Click a frame to zoom in; use the top table for symbol sorting, filtering, and statistics

For more profiling dimensions, see the Profiles API section below.

🌐 Profiles API

huatuo-apiserver exposes /v1/profiles for service-based continuous profiling. Clients can create CPU or memory profiling jobs, query job status and results, and stop or delete jobs. huatuo-apiserver schedules each job on the HUATUO Agent running on the specified node. Profiling results are available through the returned Grafana URL or the raw data endpoint.

1. Request Conventions

By default, huatuo-apiserver listens on :12740. The following examples use environment variables for the server address and bearer token:

API_BASE="http://127.0.0.1:12740"
API_TOKEN="REPLACE_WITH_RANDOM_HEX"

Every request must pass the configured bearer token:

Authorization: Bearer REPLACE_WITH_RANDOM_HEX

A non-administrator user requires both /v1/profiles and /v1/profiles/** permissions. Permissions may include an HTTP method, such as GET /v1/profiles/**. The API uses the following common JSON response envelope:

{
  "code": 0,
  "message": "success",
  "data": {}
}

2. Query Profiling Capabilities

Before creating a job, query the profiling types, languages, CPU modes, memory modes, and runtime settings supported by the server:

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiles/capabilities"

The data object contains these fields:

CPU profiling supports oncpu and offcpu for c, c++, and go; java and python support only oncpu. Memory profiling supports these combinations:

3. Create a Profiling Job

POST /v1/profiles accepts the following JSON fields:

duration_seconds must cover at least two aggregation_interval_seconds periods, and their sum must be less than 3600 seconds. If the same user already has a running profiling job on the same node, the server returns 409 Conflict.

Create a Go CPU profiling job on a host:

curl -sS -i \
  -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "cpu",
    "language": "go",
    "duration_seconds": 60,
    "hostname": "node-01"
  }' \
  "${API_BASE}/v1/profiles"

Create a Java live-object profiling job in a container:

curl -sS -i \
  -X POST \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "memory",
    "language": "java",
    "memory_mode": "object_usage",
    "duration_seconds": 60,
    "container_id": "9f4c2f1a8b7d",
    "hostname": "node-01"
  }' \
  "${API_BASE}/v1/profiles"

A successful request returns 201 Created. The Location response header identifies the new job, and the response body contains the job ID used by subsequent requests:

{
  "code": 0,
  "message": "success",
  "data": {
    "id": "<profile-job-id>"
  }
}

JOB_ID="<profile-job-id>"

4. List Profiling Jobs

GET /v1/profiles supports these query parameters:

List the 20 most recent running CPU profiling jobs on node-01:

curl -sS -G \
  -H "Authorization: Bearer ${API_TOKEN}" \
  --data-urlencode "hostname=node-01" \
  --data-urlencode "status=running" \
  --data-urlencode "type=cpu" \
  --data-urlencode "limit=20" \
  --data-urlencode "offset=0" \
  --data-urlencode "sort=-created_at" \
  "${API_BASE}/v1/profiles"

data.items contains the job array. data.total is the number of matching jobs before pagination, while data.limit and data.offset are the effective pagination parameters. Non-administrator users can list only jobs they created.

5. Get a Profiling Job

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiles/${JOB_ID}"

The data object contains the job details:

Profiling jobs use these statuses:

6. Get Raw Profiling Data

GET /v1/profiles/:id/raw returns the raw profiling windows associated with the job. The response can be large, so it can be written directly to a file:

curl -sS \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -o profile-raw.json \
  "${API_BASE}/v1/profiles/${JOB_ID}/raw?limit=100&offset=0"

The profiling windows are in data.items; data.limit, data.offset, and data.has_more describe the page. Each item contains uploaded_at, captured_at, profile_type, and the pprof-compatible profile payload.

7. Stop a Profiling Job

Only jobs in pending or running status can be stopped. The PATCH request accepts only stopped as the status value:

curl -sS \
  -X PATCH \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"status":"stopped"}' \
  "${API_BASE}/v1/profiles/${JOB_ID}"

A successful stop returns 200 OK. A job that has already ended returns 400 Bad Request.

8. Delete a Profiling Job

Deletion removes only the job record. Jobs in pending or running status cannot be deleted directly and must be stopped first:

curl -sS -i \
  -X DELETE \
  -H "Authorization: Bearer ${API_TOKEN}" \
  "${API_BASE}/v1/profiles/${JOB_ID}"

A successful deletion returns 204 No Content with no response body. If the job is still active, the endpoint returns 409 Conflict.

📖 profiler CLI Overview

profiler is HUATUO’s standalone performance profiling CLI. It samples host processes or processes inside containers without requiring huatuo-apiserver, Elasticsearch, or Grafana. The tool supports C, C++, Go, Java, and Python processes and writes call stacks as folded stacks or SVG flame graphs.

C, C++, and Go use the eBPF-based native collector to observe on-CPU usage, off-CPU blocking and scheduling delay, virtual memory allocation, physical memory allocation, and physical memory residency. Java uses async-profiler to observe CPU usage, object allocation, and live objects. Python uses py-spy to observe CPU usage. The results can be used to locate hot functions, attribute memory growth, analyze processes inside containers, and preserve performance data for later diagnosis.

The remainder of this section covers standalone use of _output/bin/profiler. For service-based continuous profiling, see the Profiles API section above.

🎯 Use Cases

1. Locate CPU Hotspots and Call Paths

Sample the call stacks of C, C++, Go, Java, or Python processes at a fixed frequency and use stack width to identify the primary consumers of CPU time. The native collector can also limit sampling to selected CPUs with --cpuid, which is useful for analyzing CPU-pinned workloads or per-CPU hotspots.

2. Attribute Native Process Memory

Observe virtual address-space allocation, physical page allocation, and current physical page residency for C, C++, and Go processes. These modes distinguish between how much address space was requested, how much physical memory was allocated, and how much physical memory remains resident. They help locate call paths responsible for mmap activity, page-fault allocation, and resident memory growth.

3. Analyze JVM Object Allocation and Live Objects

Use async-profiler to collect Java object allocation or live-object call stacks. Object allocation profiles help locate high allocation rates and sources of GC pressure. Live-object profiles help identify objects that remain referenced during the collection window and the paths where they were allocated.

4. Analyze Containers and Multi-process Workloads

Use a container ID to resolve and profile target processes inside Docker or containerd workloads. Java and Python also accept comma-separated PID lists and can limit the number of concurrently running collector subprocesses, which is useful for service replicas and parent-child process groups.

🚀 Usage

1. Build and Runtime Requirements

Build all artifacts from the repository root:

make all

The resulting executable is _output/bin/profiler. Native profiling depends on Linux eBPF, perf events, and the BPF objects built from this repository. It generally requires root privileges and a kernel.perf_event_paranoid setting that permits sampling. Java profiling requires async-profiler; --tool-path must point to a directory containing bin/asprof and lib/libasyncProfiler.so. Python profiling requires py-spy; --tool-path must point to a directory containing the py-spy executable.

Display the complete help for the current version:

_output/bin/profiler --help

The basic command structure is:

sudo _output/bin/profiler \
  --type <cpu|memory> \
  --language <c|c++|go|java|python> \
  --pid <pid> \
  --duration 30 \
  --aggr-interval 10 \
  --output-format flamegraph \
  --output-path ./profiles

--type and --language are required. Java, Python, and native memory profiling require exactly one target specified with either --pid or --container-id. Native CPU profiling can sample the entire host when neither target is specified.

2. General CLI Options

Native profiling options:

Logging options:

3. Observing C, C++, and Go

C, C++, and Go use the same native eBPF collector; only the --language value changes. CPU mode counts call-stack samples and includes user-space and kernel-space stacks when symbols can be resolved.

sudo _output/bin/profiler \
  --type cpu \
  --language go \
  --pid 12345 \
  --duration 30 \
  --aggr-interval 10 \
  --freq 99 \
  --output-format flamegraph \
  --output-path ./profiles/go-cpu

Add --thread-group to include worker threads in the same process. Add --cpuid 2,4-7 to limit collection to selected CPUs. Native CPU profiling also supports container-level and host-level collection:

Native on-CPU profiling first uses hardware CPU-cycle events and falls back to the software CPU clock when the hardware PMU is unavailable. --freq remains samples per second for either source. Use --require-hardware-pmu when software clock fallback would hide IRQ-disabled CPU time.

# Profile a specific container
sudo _output/bin/profiler \
  --type cpu --language c --container-id <container-id> \
  --duration 30 --aggr-interval 10 \
  --output-format collapsed --output-path ./profiles/container

# Profile the host without specifying a PID or container
sudo _output/bin/profiler \
  --type cpu --language c \
  --duration 30 --aggr-interval 10 \
  --output-format flamegraph --output-path ./profiles/host

To attribute time spent outside the CPU to the call path that descheduled, select off-CPU mode:

sudo _output/bin/profiler \
  --type cpu --language go --pid 12345 --thread-group \
  --cpu-mode offcpu --offcpu-phase all \
  --cpuid 2,4-7 \
  --offcpu-min-duration-us 1000 \
  --duration 30 --aggr-interval 10 \
  --output-format flamegraph --output-path ./profiles/go-offcpu

Off-CPU output is event-driven, so --freq does not apply. With --cpuid, an interval is collected only when the task switches out from a selected CPU; later wakeup or switch-in on another CPU does not change that attribution. Flame graphs use nanoseconds directly and add roots such as off-CPU blocked, scheduling delay (preempted), and scheduling delay (yielded). The all phase includes blocked and runqueue time but keeps them separated by these roots. A single stable BPF stack map is used so a long sleep cannot be resolved against a later rotating stack-map generation.

Native memory profiling supports these dimensions:

sudo _output/bin/profiler \
  --type memory \
  --language c++ \
  --memory-mode physical_usage \
  --pid 12345 \
  --thread-group \
  --physical-memory-probability 100 \
  --duration 30 \
  --aggr-interval 10 \
  --output-format flamegraph \
  --output-path ./profiles/native-memory

--physical-memory-probability applies only to physical_alloc and physical_usage. Lowering it reduces processing for high-frequency memory events, but flame-graph values are then estimates based on sampled events rather than counts of every event.

4. Observing Java

Java CPU profiling depends on async-profiler. It supports a single PID, a container, or multiple PIDs:

_output/bin/profiler \
  --type cpu \
  --language java \
  --pid 12345,12346 \
  --tool-path /opt/async-profiler \
  --max-concurrent-procs 2 \
  --duration 30 \
  --aggr-interval 10 \
  --freq 99 \
  --output-format flamegraph \
  --output-path ./profiles/java-cpu

Java memory profiling supports two dimensions:

_output/bin/profiler \
  --type memory \
  --language java \
  --memory-mode object_usage \
  --pid 12345 \
  --tool-path /opt/async-profiler \
  --duration 30 \
  --aggr-interval 10 \
  --output-format flamegraph \
  --output-path ./profiles/java-memory

To target a container, replace --pid with --container-id <container-id>. If a container has multiple candidate processes, use --binary-match-path to select the target executable path.

5. Observing Python

Python currently supports CPU profiling only. --aggr-interval must equal --duration, so one collection produces one aggregation window. --tool-path must point to the directory containing py-spy.

_output/bin/profiler \
  --type cpu \
  --language python \
  --pid 12345,12346 \
  --tool-path /opt/py-spy \
  --max-concurrent-procs 2 \
  --duration 30 \
  --aggr-interval 30 \
  --freq 99 \
  --output-format flamegraph \
  --output-path ./profiles/python-cpu

Python does not support --type memory. Use a separate memory analysis tool for Python memory profiling; the current profiler command does not invoke memray to generate Python memory profiles.

6. Choosing a Flame Graph and Output Format

A flame graph shows the call direction from bottom to top. Rectangle width represents the cumulative value for that call stack in the selected profiling mode. For CPU profiles, width represents the proportion of CPU time derived from sample counts. For memory profiles, it represents virtual allocation, physical allocation, physical residency, Java object allocation, or live-object volume, depending on the selected mode. Horizontal position does not represent chronological order.

Example folded stacks:

main;handleRequest;parsePayload 428
main;handleRequest;writeResponse 172

Choose collapsed when you need to retain raw data and later render it with different colors or filters. Choose flamegraph when you want to inspect hotspots directly. remote depends on the HUATUO toolstream Unix socket, requires a non-empty --tracer-id, and should not be selected for standalone offline use.

7. Reproducing Integration Test Examples

The repository’s integration tests provide executable end-to-end examples. Each test creates a target process, runs profiler, and verifies the expected call stack in the output:

# Native CPU
sudo ./integration/run.sh test_profiler_native_cpu.sh

# Native off-CPU blocking and scheduling delay
sudo ./integration/run.sh test_profiler_native_cpu_offcpu.sh

# Native virtual and physical memory
sudo ./integration/run.sh test_profiler_native_mem_virtual_alloc.sh
sudo ./integration/run.sh test_profiler_native_mem_physical_usage.sh

# Java CPU and memory
sudo ./integration/run.sh test_profiler_java_cpu_multi_pid.sh
sudo ./integration/run.sh test_profiler_java_memory_usage_alloc.sh

# Python multi-process CPU
sudo ./integration/run.sh test_profiler_python_cpu_multi_pid.sh

Container, thread-group, and CPU-selection examples are available in test_profiler_native_cpu_container.sh, test_profiler_native_cpu_thread_group.sh, and test_profiler_native_cpu_cpuid.sh, respectively. Run make all first and configure the Java or Python profiler path in integration/env.sh as needed.

⚙️ How It Works

profiler first selects a collector based on the language and profile type. The native on-CPU collector attaches eBPF programs to perf events; off-CPU mode attaches scheduler switch, wakeup, exit, and task-free tracepoints. Native memory collectors record allocation and release paths through kernel events. The Java and Python collectors start async-profiler and py-spy subprocesses, respectively. Collected records enter a common aggregation pipeline, which merges counts by call stack and then writes a local file or uploads the result to remote storage.

flowchart LR
    CLI[profiler CLI options] --> Select{Language and profile type}
    Select -->|C/C++/Go| Native[Native eBPF collector]
    Select -->|Java| Java[async-profiler]
    Select -->|Python| Python[py-spy]
    Native --> Queue[Sample record queue]
    Java --> Queue
    Python --> Queue
    Queue --> Aggregate[Aggregate by call stack]
    Aggregate --> Folded[Collapsed stacks]
    Aggregate --> SVG[Interactive SVG flame graph]
    Aggregate --> Remote[Remote upload through Unix socket]

--duration controls the collection lifetime, while --aggr-interval controls the snapshot interval for remote uploads. Local collapsed, flamegraph, and svg modes write the final aggregate when collection ends. remote creates and uploads snapshots at the aggregation interval. The queue decouples collection from symbolization, aggregation, and output so file rendering does not block the sampling path.

🌟 Conclusion

🌟 Star HUATUO on GitHub: https://github.com/ccfos/huatuo

👀 Follow the official WeChat account