TCP Retransmission Tracing

Overview

tcpshark --mode retransmit observes TCP retransmission-related kernel activity through the tcp/tcp_retransmit_skb and tcp/tcp_retransmit_synack tracepoints. It can also observe the tcp_send_loss_probe kprobe when TLP collection is explicitly enabled. Depending on the event type, an event can include the IP 4-tuple, TCP state, congestion-control state, retransmission counters, sequence information, and socket metadata used for container resolution.

The userspace classifier derives a connection phase and a reason label from the event type, sk_state, ca_state, and reorder counters. These labels are operational heuristics, not packet-loss root-cause proof.

Filter expressions are compiled at load time by internal/pcapfilter and run in the kernel. The SKB, SYN-ACK, and TLP hooks always evaluate the expression against the same synthetic L3 TCP packet, with or without local correlation. Protocol, address, network, and port conditions are supported. Ethernet addresses, payload, real packet lengths, IP/TCP options, and raw byte-offset expressions are not available. Safe ethertype checks such as ether proto ip are rewritten for L3. A pair of IPv4-mapped IPv6 socket addresses is filtered as IPv4; the raw perf record remains AF_INET6 and userspace normalizes the addresses before matching. In local correlation mode, tcpshark also applies the exact same expression to embedded dropwatch. Use a direction-symmetric expression when reverse ACK or SYN-ACK evidence must remain in scope.

Scenarios

1. TCP Network Quality and Retransmission Diagnosis

Continuously observe RTO, fast retransmission, reorder-prone retransmission, and TLP events to identify abnormal retransmissions during connection establishment, data transfer, and connection teardown. These signals help investigate packet loss, congestion, reordering, and peer reachability problems.

2. Kubernetes Container Network Troubleshooting

Use the container ID, network namespace, and socket cgroup metadata to identify the workload experiencing retransmissions. Apply --filter "tcp and port <service-port>" to focus on a specific service and reduce interference from other host connections.

3. Application Latency and Throughput Anomaly Analysis

Align TCP retransmission events with application latency, error-rate, and throughput timelines. This helps determine whether RTOs or repeated retransmissions coincide with service degradation and distinguish slow application processing from underlying network problems.

4. Locating Packet Loss with dropwatch Correlation

Run tcpshark in local correlation mode to correlate retransmissions with packet drops in the same process. Matching checks the network namespace, tuple direction, TCP sequence or ACK evidence, and kernel monotonic ordering. A strict match identifies an observed drop, with its software or hardware source taken from the record. A no-match remains unknown because source readiness does not establish that the retransmission’s earlier causal history was observed.

Usage

1. Running tcpshark

tcpshark --mode retransmit [flags]

When both --output and --output-storage are explicitly specified, --output is ignored and a warning is printed.

1.1 Examples

# Text output for all retransmission-related events
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o

# NDJSON output
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json

# BPF-side filter for regular retransmitted SKBs to one destination host and port
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --filter "dst host 10.0.0.1 and dst port 443"

# Correlate locally; both BPF inputs use the same direction-symmetric filter
sudo tcpshark --mode retransmit --with-dropwatch --bpf-path-dir bpf \
  --filter "tcp and port 443"

# Include Tail Loss Probe events (disabled by default)
sudo tcpshark --mode retransmit --enable-tlp --bpf-path bpf/tcp_retransmit.o

# Emit at most 100 events/second; overflow prints a rate limit hit log
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --max-events-per-second 100

# Filter all formatted event types to destination port 443 in userspace
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --output json \
  | jq -c 'select(.tcp_dport == 443)'

# Keep only events classified as RTO for 60 seconds
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o --duration 60 --output json \
  | jq -c 'select(.tcp_reason == "RTO")'

# Forward events to a running huatuo-bamai instance
sudo tcpshark --mode retransmit --bpf-path bpf/tcp_retransmit.o \
  --output-storage /var/run/huatuo-toolstream.sock

jq -c emits compact single-line JSON, which is convenient for NDJSON files and downstream pipelines.

1.2 Integration with huatuo-bamai

tcpshark uses the same --output-storage and toolstream flow as dropwatch. For the common storage workflow, refer to the dropwatch documentation. TCP retransmission tracing adds the following configuration:

[EventTracing.TCPRetransmit]
    # Used by tcpshark in both modes. Default: empty.
    Filter = ""

    # Forwarded as tcpshark --enable-tlp. Default: false.
    EnableTLP = false

    # Run tcpshark with an embedded dropwatch source. Default: false.
    EnableDropwatch = false

    # Forwarded as tcpshark --max-events-per-second. Default: 100; 0 disables it.
    MaxEventsPerSecond = 100

EventTracing.TCPRetransmit.Filter controls retransmission collection in both modes. With local correlation disabled, an empty value passes no --filter flag. With local correlation enabled, the normalized expression is passed to both tcpshark inputs and an empty value becomes tcp. EventTracing.Dropwatch.Filter remains independent and controls only standalone dropwatch.

The tcp_retransmit tracer is in the global BlackList by default. Remove it from the list and restart huatuo-bamai to enable the tracer. Standalone dropwatch may remain blacklisted because local correlation owns a private dropwatch source.

2. Filter Expressions

tcpshark uses the same tcpdump-style filter expressions as dropwatch. For complete syntax, limitations, and additional examples, refer to the dropwatch documentation.

# Select one destination host and port
--filter "dst host 10.0.0.1 and dst port 443"

# Select traffic in both directions between two networks
--filter "(src net 10.10.0.0/16 and dst net 10.20.0.0/16) or (src net 10.20.0.0/16 and dst net 10.10.0.0/16)"

In local mode the same expression must cover both traffic directions. A directional selector can exclude reverse ACK or SYN-ACK drop evidence and make the result less useful.

Local mode rejects Ethernet-address primitives such as ether host 02:00:00:00:00:01. Ethertype primitives such as ether proto ip and ether proto ip6 are supported because they can be rewritten as raw-IP version checks.

3. Event Data Structure

Each event is an NDJSON object (types.TCPRetransmitTracing). Fields tagged with omitempty are absent when their value is empty or zero.

icsk_pending is a timer-state snapshot at the hook, not a stable retransmission-reason enum. TLP classification uses the explicit event_type=tcp_send_loss_probe and does not depend on icsk_pending=5.

3.1 Text Output Format

Text retains its terminal-friendly layout while covering the same event variables as JSON. Optional variables appear only when non-zero or non-empty, and string values are not JSON-quoted or escaped. For compatibility with the original text format, state, skb, seq, end, ack, flags, ca, retrans, and reason correspond to the JSON fields tcp_state, skb_addr, tcp_seq, tcp_end_seq, tcp_ack_seq, tcp_flags, ca_state, icsk_retransmits, and correlation_reason, respectively.

<timestamp> [<phase>/<tcp_reason>] <saddr>:<sport> > <daddr>:<dport> state=<STATE> event_type=<TYPE> [kernel_observed_timestamp=<UTC>] [SYNACK] [skb=<ADDR>] seq=<N> [end=<N>] ack=<N> [flags=<FLAGS>] pid=<N> comm=<COMM> ca=<N> retrans=<N> icsk_pending=<N> [reord_seen=<N>] [dsack_dups=<N>] [container_id=<ID>] [memory_cgroup_css_addr=<ADDR>] [net_namespace_cookie=<N>] [net_namespace_inum=<N>] [drop_location=<LOCATION>] [drop_source=<SOURCE>] [drop_reason=<REASON>] [drop_reason_group=<GROUP>] [reason=<REASON>] [matched_net_namespace=true] [dropwatch_map_counters_available=<true|false> dropwatch_perf_lost=<N> dropwatch_lost_samples=<N> dropwatch_rate_limited=<N>] [source=<SOURCE>]

Example:

2026-07-23T02:14:40.304775546Z [data/RTO] 127.0.0.1:19996 > 127.0.0.1:42128 state=ESTABLISHED event_type=tcp_retransmit_skb kernel_observed_timestamp=2026-07-23T02:14:40.304Z skb=0xffff931c14fdf800 seq=3154974646 end=3154991030 ack=948393597 flags=ACK|PSH pid=1420 comm=kube-apiserver ca=4 retrans=4 icsk_pending=0 net_namespace_inum=4026531992

The pid and comm in this example describe the execution context in which the hook ran; use container_id and socket metadata for workload attribution.

A non-empty drop_stack is rendered as indented lines after the event line, not as an inline drop_stack= token.

3.2 Container ID Resolution

tcpshark cannot access the Pod manager directly. In standalone output, container_id is normally absent, while socket memcg and network-namespace metadata are still emitted when available. In huatuo-bamai mode, an empty container_id is resolved in this order: memory_cgroup_css_addr, net_namespace_cookie, then net_namespace_inum.

If all lookups miss, the event is still stored without container_id. Do not use pid or comm as a fallback for socket ownership because they describe the hook execution context.

4. Kernel Events and Classification

4.1 Kernel Hook Points

The BPF program uses CO-RE field reads (BPF_CORE_READ and related helpers), so supported kernel layouts do not require rebuilding the C source for each kernel version.

4.2 Connection Phase

The regular-SKB phase is derived from sk_state. SYN-ACK events use a fixed phase in userspace.

The TCP three-way handshake below shows the connect phase and its retransmission hook points:

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: Initial states: CLOSED / LISTEN
    C->>S: ① SYN
    Note left of C: SYN_SENT(2)<br/>phase=connect
    opt SYN is not acknowledged
        C-->>S: SYN retransmission<br/>tcp_retransmit_skb
    end
    Note right of S: SYN_RECV(3) or NEW_SYN_RECV(12)<br/>phase=connect
    S->>C: ② SYN + ACK
    opt Final ACK does not arrive
        S-->>C: SYN-ACK retransmission<br/>tcp_retransmit_synack
    end
    C->>S: ③ ACK
    Note over C,S: ESTABLISHED(1)<br/>subsequent regular data-SKB events use phase=data

The three solid arrows are the initial handshake packets and do not produce tcpshark events. Only the retransmission paths inside the optional blocks are observed. Active-open SYN retries are reported by tcp_retransmit_skb, while passive-open SYN-ACK retries are reported by tcp_retransmit_synack; both are classified as connect.

The complete phase mapping is:

4.3 Reason Classification

The classifier observes socket state at the hook and cannot reconstruct the complete ACK/loss history. Treat tcp_reason as a grouping label rather than a verified root cause.

4.4 Reorder Heuristic

4.5 Operational Guidance

No event type is unconditionally safe to discard. Prefer rate, ratio, and service-impact thresholds over filtering solely by event_type or tcp_reason. For the common huatuo-bamai noise-filtering mechanism, refer to the dropwatch documentation.

When building alerts, aggregate by service or connection and compare against traffic volume. A small absolute count on a busy host can be benign, while a burst affecting a low-volume critical service can be significant.

5. Correlation with dropwatch

With --with-dropwatch, one tcpshark process owns both perf inputs. A retransmission waits up to 100 ms for a delayed dropwatch delivery. A candidate drop must precede the retransmission by no more than one second in kernel monotonic time. The embedded source never emits raw drop documents; separately enabled standalone dropwatch remains an independent raw-event stream.

For the limitations around out-of-order reads across the two perf streams, the 100 ms arrival window, the one-second causal window, and negative evidence, see Challenges of TCP Retransmission and dropwatch Correlation.

Like standalone dropwatch, the embedded source automatically detects and enables devlink DROP trap capture (HardwareAuto). No additional hardware flag is needed. If the tracepoint is unavailable, it logs a warning and continues software capture. Visibility depends on driver reporting; see the dropwatch hardware requirements.

--device, --device-excluded, --filter, and --max-events-per-second use the same names as dropwatch. The interface allowlist or denylist limits only embedded dropwatch, covering both software and hardware events. The allowlist rejects records without device information; the denylist permits them. Retransmissions still use the shared L3 filter, so device filtering can reduce available drop evidence.

Matched output retains drop_source, drop_reason, and the hardware drop_reason_group. drop_reason describes the observed drop, tcp_reason classifies the retransmission trigger, and correlation_reason identifies how this correlation attempt ended. Independent diagnostic fields describe observation limits. The BTF reason table loads once per session. A load failure logs a warning and falls back to numeric software reasons; hardware trap decoding remains available.

# Use the same interface and traffic options as dropwatch to inspect matched hardware drops
sudo tcpshark --mode retransmit --with-dropwatch --bpf-path-dir bpf \
  --device eth0 --filter "tcp and port 443" --output json \
  | jq -c 'select(.drop_source == "hardware")'

Hardware drops must meet the same namespace, TCP, and time constraints. Missing namespace or TCP evidence leaves a retransmission unmatched; no-match does not establish hardware loss. A hardware drop_stack is the driver’s trap-reporting kernel stack, not a location inside the ASIC.

An unrecognized ABI source produces unknown for both drop_source and drop_location, even when packet evidence matches. The source is never inferred from a reason or stack. Such a match still reports correlation_reason=matched.

5.1 Correlation Results

There is no tuple-only, SKB-pointer-only, cross-namespace, or ambiguous positive match. The matched_net_namespace diagnostic records a same-namespace drop on the same flow in either direction before packet and time checks. It can be true without a strict match; correlation_reason=matched remains the indication of full correlation. A false or absent flag means no namespace match was observed, not that a different namespace was confirmed. A matched drop is consumed once, while later drops on the same connection remain available. Stack symbolization runs only after a match.

5.2 Correlation Outcome

Each finalized event has exactly one correlation_reason:

Expired entries are finalized before matching or checking capacity. Shutdown uses one time snapshot: expired waits receive warmup or wait_timeout; remaining waits receive interrupted. Each event is finalized once and removed from the queue. Drop records rejected during decoding or evicted from the drop cache do not directly finalize a retransmission.

The scalar field replaces the previous reason array; update tcpshark and its receiver together. Text output uses reason=<value>. Correlation-disabled output omits the field.

warmup is selected only for an expired wait when the retransmission’s kernel timestamp predates embedded-source readiness. Retransmissions at or after readiness receive wait_timeout on expiry, including those within the first second. The one-second drop-to-retransmit limit constrains candidate matching; it does not define warmup. Processing delays do not change this classification. Strict matches, unsupported events, queue evictions, and early interruptions retain their own reasons. The outcome describes observation timing; it does not prove why a drop failed to match.

matched_net_namespace remains an independent boolean diagnostic for both matched and unmatched results. It can coexist with loss counters without changing the terminal reason.

At shutdown, tcpshark stops reading dropwatch records still in the perf ring. An unread tail drop could otherwise have matched an interrupted retransmission.

5.3 Dropwatch Perf Status

Each non-empty output batch reads status once, including batches containing only matches. Unmatched events retain the snapshot:

On a status read error, map_counters_available is false. The zero values of perf_lost and rate_limited mean unavailable; the snapshot still retains lost_samples. tcpshark attempts to output the batch before returning the status error. A write failure stops the batch and preserves both errors. Matched events retain matched and their drop metadata without an unmatched status snapshot.

Counters belong to this dropwatch instance and reset on reload. They are sampled separately, not atomically, and cannot prove why an individual retransmission failed to match. Loss and rate limiting never replace or add a correlation reason.

5.4 Requirements and Troubleshooting

huatuo-bamai passes one normalized EventTracing.TCPRetransmit.Filter value to both local-correlation inputs. Keeping those scopes identical prevents the two sources from observing different traffic, but it does not make a no-match conclusive without a reliable causal-start boundary.

Closing