Diagnose conntrack exhaustion and validate capacity changes

When a service experiences intermittent connection timeouts while existing sessions continue transferring data and application processes and listeners appear healthy, check the Linux connection-tracking table. Stateful firewalls, address translation, and some container networking rely on conntrack. If new entries cannot be created, new traffic may be dropped before it reaches the application. Using Debian 12 as an example, this guide covers evidence collection, cause analysis, capacity adjustment, and rollback without changing firewall rules or clearing existing connections.

1. Identify the network namespace where the failure occurs

CloudDDR dedicated servers offer AMD EPYC 7543P processors and 128GB or 256GB of ECC memory, but memory capacity does not directly determine the conntrack limit. The kernel, network namespace, and service traffic are what need to be examined. After payment, retrieve the credentials from the console and connect through SSH. If multiple addresses are listed, use the first one.

Run the commands in the network namespace that actually handles the traffic. Results from the host do not represent the guest system. If container traffic passes through address translation on the host, check the host as well. Start by recording the kernel version, current network namespace, entry count, and limits:

uname -r
readlink /proc/self/ns/net
sudo sysctl net.netfilter.nf_conntrack_count
sudo sysctl net.netfilter.nf_conntrack_max
sudo sysctl net.netfilter.nf_conntrack_buckets

If a key does not exist, the corresponding parameter is not exposed in the current namespace. That alone does not establish exhaustion. First check whether the relevant modules are loaded and whether the service uses connection tracking. Do not enable new functionality during an incident merely to make every command return output.

Do not run conntrack -F. Flushing the table deletes tracking state, potentially disrupting existing sessions and address-translation mappings. It also destroys evidence needed to diagnose the problem.

2. Build an evidence chain from counter deltas

Installing the userspace tools does not change existing rules, but installation should still follow your established change process. The following commands capture the current state and take two samples at an interval to determine whether counters are still increasing:

sudo apt-get install conntrack
umask 077
sudo conntrack -S > /tmp/conntrack-before.txt
sudo journalctl -k --since "-10 min" --no-pager \
  | grep -Ei 'nf_conntrack|table full'
sleep 30
sudo conntrack -S > /tmp/conntrack-after.txt
diff -u /tmp/conntrack-before.txt /tmp/conntrack-after.txt

conntrack -S typically reports cumulative counters per CPU. Compare matching CPU rows and focus on increases in insert_failed, drop, and early_drop, rather than treating totals since boot as evidence of the current incident. Counter meanings and output fields can vary by kernel, and not every insertion failure is caused by insufficient capacity.

Also record service timeout counts, table occupancy, and kernel logs. The evidence is more complete when occupancy approaches the limit, failure counters increase, and service errors occur at the same time. A lack of matching log messages does not rule out the problem, since logging may be rate-limited. Brief spikes between samples may go undetected. Shorten the observation interval if necessary, but do not repeatedly dump the entire table in a loop.

3. Distinguish connection count from connection creation rate

Use a sample to understand why entries accumulate

The number of entries is not the number of current requests. A single TCP connection can carry multiple requests, and UDP also consumes tracking entries. After communication ends, entries may remain until their respective timeouts expire. Distinguish long-lived, stable connections from bursts of short-lived connections and anomalous sources.

Take a single bounded sample rather than repeatedly dumping the full connection table under high load:

sudo timeout 5s conntrack -L -o extended 2>/tmp/conntrack-sample.err \
  | head -n 200 > /tmp/conntrack-sample.txt

This command captures only a limited amount of output. It may terminate early and return a nonzero exit status, and the sample is not a complete statistical view. The files contain address information, so restrict read access and redact ticket attachments as needed. The output includes tuples for both the original and reply directions. Do not combine the two sets of addresses when analyzing sources.

Inspect the protocol, TCP state, remaining timeout, and source in the original direction. For large numbers of short-lived connections, also examine application connection pools and retry logs. If a small number of sources keep creating new flows, use access records to assess their behavior. If UDP entries accumulate over time, verify the actual communication intervals. A state name alone is not enough to identify an attack, and anomalous traffic should not simply be reframed as a capacity issue.

If the connection creation rate does not fall, increasing table capacity only delays the next time the limit is reached. Fixing unbounded retries, restoring connection reuse, or addressing anomalous sources often requires validation separate from the capacity change.

4. Assess memory, then change only one variable

The limit is neither preallocated memory nor a throughput guarantee

Increasing nf_conntrack_max does not immediately create all possible entries, but sustained table growth consumes kernel memory. Use free -h, /proc/meminfo, and sudo slabtop -o to observe memory and slab usage. The reported cache object size does not include every cost, such as extension fields and allocation overhead, so a fixed “bytes per connection” formula is not appropriate.

nf_conntrack_buckets is the number of hash buckets, not the connection capacity. Raising only the entry limit may lengthen bucket chains and increase lookup costs, so monitor CPU usage and service latency as well. Whether the bucket count can be changed online depends on the kernel and namespace. It is not changed alongside the limit in this procedure.

The following example demonstrates only a temporary change to a target limit of 262144. Before running it, confirm that this value exceeds the current limit, fits the memory budget, and that another administrative session is available. It is not a recommended value for every machine. Run the commands in a shell with administrative privileges:

sudo -i
OLD=$(sysctl -n net.netfilter.nf_conntrack_max)
test "$OLD" -lt 262144 || exit 1
umask 077
printf '%s\n' "$OLD" > /run/conntrack-max.before
sysctl -w net.netfilter.nf_conntrack_max=262144
sysctl -n net.netfilter.nf_conntrack_max

After saving the original value, repeat the same service probes and counter sampling. Do not shorten the TCP established timeout at the same time. Idle, long-lived connections may still be in use by the service, and globally reducing the timeout can introduce a new cause of disconnections.

5. Separate validation, rollback, and persistence

Validation must cover at least the traffic involved in the original failure and long-lived connections that resume communication after being idle. Check the success rate of new connections, failure-counter deltas, occupancy growth, available memory, CPU usage, and service latency. A higher limit alone does not mean the problem is resolved.

If memory pressure, CPU usage, or latency worsens, roll back using the saved value:

sudo sysctl -w \
  net.netfilter.nf_conntrack_max="$(sudo cat /run/conntrack-max.before)"

Lowering the limit does not immediately remove existing entries. If occupancy remains above the restored limit, new entries may still fail to be created. Wait for entries to expire naturally and address the traffic sources. Do not flush the table to speed up recovery.

Once the temporary adjustment has passed validation, check /etc/sysctl.conf and /etc/sysctl.d/ for existing settings with the same name, so that multiple files do not override one another. Write the validated value to a dedicated, version-controlled configuration file, for example:

net.netfilter.nf_conntrack_max = 262144

Load only that file and recheck the effective value. Do not reload unrelated parameters in bulk during an incident. During a subsequent controlled reboot, also verify the final value after the module has loaded: writing a configuration file does not guarantee that the setting will take effect at boot.

The final record should include traffic sources, counters before and after the change, service probe results, memory observations, the original value, and recovery steps. Connection-tracking capacity management is complete when the incident evidence can be explained and the adjustment results can be reproduced—not when the entry limit merely shows a larger number.

Frequently asked questions