AutoTracing Fields

This reference covers AutoTracing record fields, units, and examples. See AutoTracing for use cases and Node Configuration for settings. Thresholds below are built-in defaults that configuration files can override.

Results

Automatic tracing events are written to Elasticsearch or local files according to enabled storage settings. Filter by tracer_type=autotracing, tracer_name, host or container identifiers, and time range. The JSON examples illustrate field structure, not measured results.

CPU examples use perf. With profiler, events retain trigger metrics; profiles are stored separately in Elasticsearch with the same tracer_id. See Node Configuration for storage requirements and viewing instructions.

Common Fields

Records contain these common fields. Container fields are provided only when a container is associated.

Host CPU

Tracer: cpusys. Select perf (default) or profiler with AutoTracing.CPUSys.Tool.

Periodically reads /proc/stat to detect host system, user (user + nice), and executing-total (user + nice + system + irq + softirq) CPU bursts, including container work. Any group triggers profiling when both utilization and its increase strictly exceed its thresholds. Default utilization/increase thresholds are system 45%/20 percentage points, user 75%/45 percentage points, and total 90%/55 percentage points.

All three groups share a capture and a default 30-minute cooldown. Thresholds are configured under CPUSys independently of CPUIdle; see the configuration reference. Records retain system fields and add matching user_percent* and total_percent* fields; zero-valued additional fields are omitted.

Example with some common fields omitted:

{
    "tracer_name": "cpusys",
    "tracer_data": {
        "system_percent": 52,
        "system_percent_threshold": 45,
        "system_percent_delta": 25,
        "system_percent_delta_threshold": 20,
        "flamedata": [
            {"level": 0, "value": 1000, "self": 0, "label": "all"},
            {"level": 1, "value": 350, "self": 350, "label": "do_syscall_64"}
        ]
    }
}

Container CPU

Tracer: cpuidle. Select perf (default) or profiler with AutoTracing.CPUIdle.Tool.

Periodically reads container cgroup CPU statistics. Container-level CPU sampling starts when user, system, or total utilization exceeds both its utilization threshold and its increase threshold between consecutive samples.

The default cooldown is 30 minutes per container. AutoTracing.CPUIdle.Filter controls target scope. Built-in thresholds:

Example with some common fields omitted:

{
    "tracer_name": "cpuidle",
    "tracer_data": {
        "user_percent": 80,
        "user_percent_threshold": 75,
        "user_percent_delta": 48,
        "user_percent_delta_threshold": 45,
        "system_percent": 12,
        "system_percent_threshold": 45,
        "system_percent_delta": 5,
        "system_percent_delta_threshold": 20,
        "total_percent": 92,
        "total_percent_threshold": 90,
        "total_percent_delta": 53,
        "total_percent_delta_threshold": 55,
        "flamedata": [
            {"level": 0, "value": 1000, "self": 0, "label": "all"},
            {"level": 1, "value": 800, "self": 800, "label": "java/com.example.App.main"}
        ]
    }
}

Container D-State Load

Tracer: dload.

Reads container process states through netlink and cgroups to calculate an exponential moving average (EMA) of uninterruptible (D-state) process load. It triggers when the EMA exceeds the threshold (default: 5), with a default 30-minute cooldown per container.

Results include kernel stacks of container and host D-state processes. AutoTracing.IssuesList labels known issues.

Example with some common fields omitted:

{
    "tracer_name": "dload",
    "tracer_data": {
        "threshold": 5,
        "nr_sleeping": 120,
        "nr_running": 4,
        "nr_stopped": 0,
        "nr_uninterruptible": 8,
        "nr_iowait": 3,
        "load_avg": 7.23,
        "dload_avg": 6.81,
        "known_issue": "",
        "stack": "task:java            state:D stack:    0 pid: 12345 tgid: 12345 ...\n  io_schedule+0x18/0x40\n  ext4_file_write_iter+0x..."
    }
}

Disk I/O

Tracer: iotracing.

Reads /proc/diskstats every 5 seconds. A trigger condition must hold for two consecutive intervals. Non-NVMe devices check utilization; NVMe devices require both utilization and either read or write throughput to exceed thresholds. Each device also checks mean read and write wait times independently. Results include process and file I/O rates and I/O scheduling wait stacks.

When running iotracing directly, --enable-pprof (disabled by default) serves profiles at http://127.0.0.1:6000/debug/pprof/; if the address is already in use, the command fails to start. These profiles describe only the iotracing process’s Go runtime, not huatuo-bamai or the application processes performing disk I/O. Automatic tracing does not enable this option.

Example with some common fields omitted:

{
    "tracer_name": "iotracing",
    "tracer_data": {
        "reason_snapshot": {
            "type": "ioutil",
            "device": "sda",
            "major_num": 8,
            "minor_num": 0,
            "iostatus": {
                "read_bps": 125829120,
                "read_iops": 450,
                "read_await": 12,
                "write_bps": 2202009600,
                "write_iops": 890,
                "write_await": 145,
                "io_util": 95,
                "queue_size": 32
            }
        },
        "process_file_io_stats": [
            {
                "pid": 12345,
                "comm": "java",
                "container_hostname": "app-pod-xxx",
                "total_fs_read_bps": 0,
                "total_fs_write_bps": 52428800,
                "total_disk_read_bps": 0,
                "total_disk_write_bps": 49152000,
                "total_files": [
                    {
                        "path": "/data/logs/app.log",
                        "dev_name": "sda",
                        "fs_write_bps": 52428800,
                        "disk_write_bps": 49152000
                    }
                ],
                "total_file_count": 1
            }
        ],
        "io_schedule_timeout_stacks": [
            {
                "pid": 12345,
                "tid": 12346,
                "cpu": 2,
                "comm": "java",
                "container_hostname": "app-pod-xxx",
                "schedule_latency_us": 250000,
                "stack": [
                    "io_schedule+0x18/0x40",
                    "ext4_file_write_iter+0x2a0/0x4c0"
                ]
            }
        ]
    }
}

Host Anonymous Memory

Tracer: memburst.

By default, samples host anonymous memory every 10 seconds and evaluates after filling a 60-sample sliding window. It triggers when current anonymous memory reaches twice the oldest window value and at least 70% of total host memory. The default cooldown is 30 minutes.

Results list the top N processes by descending RSS (default: 10) and attempt runtime snapshots serially for the first min(N, 3) processes. Go, HotSpot, and CPython use their respective providers. Unsupported runtimes report unavailable and retain process memory counters when available. A failed capture does not mean zero memory usage.

Each process has a two-second capture budget. Each snapshot retains at most 10 memory object entries and 512 KiB of JSON. Timeouts are cooperative and cannot interrupt synchronous system calls, so capture may take longer than two seconds.

The complete tracer_data is limited to 2 MiB, with at most 1,000 persisted ranking entries. When limits are exceeded, the lowest-ranked snapshots are removed first, then ranking tails if needed, and snapshot_reason is set. Cancellation and fallback records follow the same limits.

Example (some common fields omitted):

{
    "tracer_name": "memburst",
    "tracer_data": {
        "top_memory_usage": [
            {
                "PID": 3456,
                "ProcessName": "go-service",
                "MemSize": 268435456
            }
        ],
        "process_snapshots": [
            {
                "pid": 3456,
                "process_name": "go-service",
                "language": "go",
                "snapshot": {
                    "runtime_version": "go1.24.0",
                    "status": "complete",
                    "duration_ms": 25,
                    "entries": [
                        {
                            "kind": "inuse_space_objects",
                            "name": "main.allocate",
                            "bytes": 1048576,
                            "objects": 16,
                            "stack": [
                                "main.allocate fixture.go:42"
                            ]
                        }
                    ]
                },
                "process_memory": {
                    "status": "complete",
                    "rss_bytes": 268435456,
                    "rss_anon_bytes": 260046848
                }
            }
        ]
    }
}

snapshot fields:

process_memory contains language-independent process memory counters. Its status and optional status_reason describe availability. All counters below are optional; missing values mean unavailable, not zero: