TCP Retransmission Tracing

Overview

tcpshark --mode retransmit observes TCP retransmission-related kernel activity through the tcp/tcp_retransmit_skb and tcp/tcp_retransmit_synack tracepoints. It can also observe the tcp_send_loss_probe kprobe when TLP collection is explicitly enabled. Depending on the event type, an event can include the IP 4-tuple, TCP state, congestion-control state, retransmission counters, sequence information, and socket metadata used for container resolution.

The userspace classifier derives a connection phase and a reason label from the event type, sk_state, ca_state, and reorder counters. These labels are operational heuristics, not packet-loss root-cause proof.

Filter expressions are compiled at load time by internal/pcapfilter and run in the kernel. Filters apply only to events that have an SKB (tcp_retransmit_skb); SYN-ACK and TLP events bypass the pcap filter.

Scenarios

1. TCP Network Quality and Retransmission Diagnosis

Continuously observe RTO, fast retransmission, reorder-prone retransmission, and TLP events to identify abnormal retransmissions during connection establishment, data transfer, and connection teardown. These signals help investigate packet loss, congestion, reordering, and peer reachability problems.

2. Kubernetes Container Network Troubleshooting

Use the container ID, network namespace, and socket cgroup metadata to identify the workload experiencing retransmissions. Apply --filter "tcp and port <service-port>" to focus on a specific service and reduce interference from other host connections.

3. Application Latency and Throughput Anomaly Analysis

Align TCP retransmission events with application latency, error-rate, and throughput timelines. This helps determine whether RTOs or repeated retransmissions coincide with service degradation and distinguish slow application processing from underlying network problems.

4. Locating Packet Loss with dropwatch Correlation

Run dropwatch and tcp_retransmit in the same huatuo-bamai process to correlate packet drops with retransmissions by SKB pointer or connection 4-tuple. The result helps indicate whether the problem is more likely in the host network stack or the external network, but remains heuristic evidence that should be validated with stack traces and network metrics.

Usage

1. Running tcpshark

tcpshark --mode retransmit [flags]

When both --output and --output-storage are explicitly specified, --output is ignored and a warning is printed.

1.1 Examples

# Text output for all retransmission-related events
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o

# NDJSON output
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json

# BPF-side filter for regular retransmitted SKBs to one destination host and port
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --filter "dst host 10.0.0.1 and dst port 443"

# Include Tail Loss Probe events (disabled by default)
sudo tcpshark --mode retransmit --enable-tlp --bpf-path bpf/tcp_retransmit.o

# Emit at most 100 events/second; overflow prints a rate limit hit log
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --max-events-per-second 100

# Filter all formatted event types to destination port 443 in userspace
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json \
  | jq -c 'select(.tcp_dport == 443)'

# Keep only events classified as RTO for 60 seconds
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --duration 60 --output json \
  | jq -c 'select(.tcp_reason == "RTO")'

# Forward events to a running huatuo-bamai instance
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --output-storage /var/run/huatuo-toolstream.sock

jq -c emits compact single-line JSON, which is convenient for NDJSON files and downstream pipelines.

1.2 Integration with huatuo-bamai

tcpshark uses the same --output-storage and toolstream flow as dropwatch. For the common storage workflow, refer to the dropwatch documentation. TCP retransmission tracing adds the following configuration:

[EventTracing.TCPRetransmit]
    # Forwarded to tcpshark --filter; applies only to tcp_retransmit_skb.
    # Default: ""
    Filter = ""

    # Forwarded as tcpshark --enable-tlp. Default: false.
    EnableTLP = false

    # Forwarded as tcpshark --max-events-per-second. Default: 100; 0 disables it.
    MaxEventsPerSecond = 100

The tcp_retransmit tracer is in the global BlackList by default. Remove it from the list and restart huatuo-bamai to enable the tracer. Its drop-correlation cache is enabled only while the tracer is running and is cleared when the tracer stops. After enabling it, use the HTTP API to start or stop tracing:

curl -X PUT http://localhost:19704/tracers/tcp_retransmit/start
curl -X PUT http://localhost:19704/tracers/tcp_retransmit/stop

2. Filter Expressions

tcpshark uses the same tcpdump-style filter expressions as dropwatch. For complete syntax, limitations, and additional examples, refer to the dropwatch documentation.

# Select one destination host and port
--filter "dst host 10.0.0.1 and dst port 443"

# Select traffic in both directions between two networks
--filter "(src net 10.10.0.0/16 and dst net 10.20.0.0/16) or (src net 10.20.0.0/16 and dst net 10.10.0.0/16)"

--filter applies only to tcp_retransmit_skb. The tcp_retransmit_synack and enabled tcp_send_loss_probe events have no SKB and bypass the filter.

3. Event Data Structure

Each event is an NDJSON object (types.TCPRetransmitTracing). Fields tagged with omitempty are absent when their value is empty or zero.

icsk_pending is a timer-state snapshot at the hook, not a stable retransmission-reason enum. TLP classification uses the explicit event_type=tcp_send_loss_probe and does not depend on icsk_pending=5.

3.1 Text Output Format

Text retains its terminal-friendly layout while covering the same event variables as JSON. Variables tagged with omitempty appear only when non-zero or non-empty, and string values are not JSON-quoted or escaped. For compatibility with the original text format, state, skb, seq, end, ack, flags, ca, and retrans correspond to the JSON fields tcp_state, skb_addr, tcp_seq, tcp_end_seq, tcp_ack_seq, tcp_flags, ca_state, and icsk_retransmits, respectively.

<timestamp> [<phase>/<tcp_reason>] <saddr>:<sport> > <daddr>:<dport> state=<STATE> event_type=<TYPE> [SYNACK] [skb=<ADDR>] seq=<N> [end=<N>] ack=<N> [flags=<FLAGS>] pid=<N> comm=<COMM> ca=<N> retrans=<N> icsk_pending=<N> [reord_seen=<N>] [dsack_dups=<N>] [container_id=<ID>] [memory_cgroup_css_addr=<ADDR>] [net_namespace_cookie=<N>] [net_namespace_inum=<N>] [drop_location=<LOCATION>] [source=<SOURCE>]

Example:

2026-07-23T02:14:40.304775546Z [data/RTO] 127.0.0.1:19996 > 127.0.0.1:42128 state=ESTABLISHED event_type=tcp_retransmit_skb skb=0xffff931c14fdf800 seq=3154974646 end=3154991030 ack=948393597 flags=ACK|PSH pid=1420 comm=kube-apiserver ca=4 retrans=4 icsk_pending=0 net_namespace_inum=4026531992

The pid and comm in this example describe the execution context in which the hook ran; use container_id and socket metadata for workload attribution.

3.2 Container ID Resolution

tcpshark cannot access the Pod manager directly. In standalone output, container_id is normally absent, while socket memcg and network-namespace metadata are still emitted when available. In huatuo-bamai mode, an empty container_id is resolved in this order: memory_cgroup_css_addr, net_namespace_cookie, then net_namespace_inum.

If all lookups miss, the event is still stored without container_id. Do not use pid or comm as a fallback for socket ownership because they describe the hook execution context.

4. Kernel Events and Classification

4.1 Kernel Hook Points

The BPF program uses CO-RE field reads (BPF_CORE_READ and related helpers), so supported kernel layouts do not require rebuilding the C source for each kernel version.

4.2 Connection Phase

The regular-SKB phase is derived from sk_state. SYN-ACK events use a fixed phase in userspace.

The TCP three-way handshake below shows the connect phase and its retransmission hook points:

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: Initial states: CLOSED / LISTEN
    C->>S: ① SYN
    Note left of C: SYN_SENT(2)<br/>phase=connect
    opt SYN is not acknowledged
        C-->>S: SYN retransmission<br/>tcp_retransmit_skb
    end
    Note right of S: SYN_RECV(3) or NEW_SYN_RECV(12)<br/>phase=connect
    S->>C: ② SYN + ACK
    opt Final ACK does not arrive
        S-->>C: SYN-ACK retransmission<br/>tcp_retransmit_synack
    end
    C->>S: ③ ACK
    Note over C,S: ESTABLISHED(1)<br/>subsequent regular data-SKB events use phase=data

The three solid arrows are the initial handshake packets and do not produce tcpshark events. Only the retransmission paths inside the optional blocks are observed. Active-open SYN retries are reported by tcp_retransmit_skb, while passive-open SYN-ACK retries are reported by tcp_retransmit_synack; both are classified as connect.

The complete phase mapping is:

4.3 Reason Classification

The classifier observes socket state at the hook and cannot reconstruct the complete ACK/loss history. Treat tcp_reason as a grouping label rather than a verified root cause.

4.4 Reorder Heuristic

The reorder-prone label is selected when either reord_seen or dsack_dups is non-zero. Once a flow has reorder history, subsequent Recovery-state SKB events can be labeled reorder_prone_fast. This is a flow-level heuristic, not proof that the current retransmission was caused by reordering.

4.5 Operational Guidance

No event type is unconditionally safe to discard. Prefer rate, ratio, and service-impact thresholds over filtering solely by event_type or tcp_reason. For the common huatuo-bamai noise-filtering mechanism, refer to the dropwatch documentation.

When building alerts, aggregate by service or connection and compare against traffic volume. A small absolute count on a busy host can be benign, while a burst affecting a low-volume critical service can be significant.

5. Correlation with dropwatch

When dropwatch and tcpshark feed the same huatuo-bamai process, dropwatch events are retained in a userspace cache for two seconds from their arrival time. A tcpshark event immediately queries previously received, unexpired drop events using a direction-independent connection key. The implementation does not wait for later drop events and does not revise an event after storage.

5.1 Correlation Results

network_or_host_hardware can also be produced when dropwatch is disabled, its filter does not cover the flow, an event is suppressed or lost, delivery is reordered, or the relevant drop falls outside the retention window. Likewise, a 4-tuple match can pair unrelated packets from a busy connection. The cache key does not include a network-namespace or container identifier, so identical address/port tuples in different network namespaces can also collide.

5.2 Requirements and Troubleshooting

For reliable negative evidence, dropwatch must be active with a filter that is at least as broad as the tcpshark traffic of interest. The current schema has no separate unknown or dropwatch_not_observed value, so consumers should treat network_or_host_hardware as an investigation hint rather than a fact.

Closing