Skip to content

Understanding conntrack and Connection Tracking

Connection tracking (conntrack) is the kernel subsystem that makes stateful firewalling possible — it's what lets a rule like ct state established,related accept (from the firewall article) know which packets belong to a connection that was already allowed out. It's also a frequently-overlooked source of real production incidents when its table fills up.

How it works

What conntrack actually tracks

Every TCP connection, and every UDP "flow" (UDP has no real connections, but conntrack tracks request/response pairs heuristically by timeout), gets an entry in an in-kernel table: source/dest IP, source/dest port, protocol, and current state.

cat /proc/sys/net/netfilter/nf_conntrack_max
# 262144

This is the maximum number of tracked connections the kernel will hold at once. Each entry consumes a small, fixed amount of kernel memory (roughly 300-350 bytes depending on kernel version) — 262144 entries is on the order of 80-90MB of RAM, not a large amount on a modern server, which is why this default is often left far too low for high-connection-count workloads (a busy reverse proxy or NAT gateway can exhaust it in minutes under real load).

Inspecting the table

sudo conntrack -L | head -5

Representative output:

tcp 6 431999 ESTABLISHED src=10.0.0.5 dst=203.0.113.10 sport=51234 dport=443 \
    src=203.0.113.10 dst=10.0.0.5 sport=443 dport=51234 [ASSURED] mark=0 use=1

Reading this: protocol (tcp), protocol number (6), remaining timeout in seconds (431999), state (ESTABLISHED), then both directions of the flow — the original direction and the expected reply direction, which is exactly the information ct state established,related rules rely on. [ASSURED] means at least one packet has been seen in both directions, marking this as a genuine, bidirectional connection (vs. a one-sided attempt that might just be a scan or a connection that was reset before completing).

sudo conntrack -C                        # current table count
sudo conntrack -L -p tcp --dport 443     # filter: show only HTTPS entries

The failure mode: table exhaustion

dmesg | grep -i "conntrack: table full"

When the table fills, the kernel drops new connection attempts silently — not with an error a client can meaningfully interpret, just as if the packet never arrived. This produces a very specific, confusing symptom: a service that works fine at low load and mysteriously starts refusing new connections (while existing ones keep working) once concurrent connection count crosses some threshold — classic conntrack exhaustion, easy to misdiagnose as an application bug.

Raising the limit

# Runtime (lost on reboot):
sudo sysctl -w net.netfilter.nf_conntrack_max=1048576

# Persistent:
echo "net.netfilter.nf_conntrack_max = 1048576" | sudo tee -a /etc/sysctl.d/99-conntrack.conf
sudo sysctl --system

A reasonable sizing approach: estimate realistic peak concurrent connections for the workload, multiply by a safety margin (2-3x), and set the max accordingly — rather than picking an arbitrarily large number, which simply reserves more kernel memory than necessary.

Deliberately bypassing conntrack (when it's not needed)

For very high-throughput, stateless-by-design traffic (some DNS resolvers, certain load-balancer configurations), it's sometimes correct to explicitly exempt specific traffic from connection tracking entirely with the NOTRACK target (iptables) or notrack (nftables), reducing per-packet overhead — but only once you've confirmed nothing downstream (a stateful firewall rule, NAT) actually depends on that traffic being tracked.

Ubuntu/Debian vs. RHEL-family differences

The conntrack subsystem itself is identical kernel code across distributions. The userspace tools may not be installed by default: the package is conntrack on Debian/Ubuntu (sudo apt install conntrack) and conntrack-tools on RHEL-family (sudo dnf install conntrack-tools).

Practical exercise

  1. Check your current nf_conntrack_max and current usage: cat /proc/sys/net/netfilter/nf_conntrack_max and sudo conntrack -C.
  2. Open several outbound connections at once (e.g., a handful of parallel curl requests to different hosts) and confirm the count in conntrack -C rises correspondingly.
  3. Filter the table to show only entries related to one of those connections, using conntrack -L -p tcp --dport <port>.

How to verify you did it correctly

  • Step 2: conntrack -C's count should increase by roughly the number of new connections opened, and settle back down as they close and their table entries time out.
  • Step 3: the filtered output should show exactly the entries matching your test connections, not an empty result (confirming the filter syntax itself is correct) and not the entire table (confirming the filter is actually restricting, not matching everything).

Common mistakes

  • Diagnosing intermittent "connection refused under load" as an application-level bug without ever checking dmesg for conntrack table-full messages — this is a kernel-level symptom that application logs won't show at all.
  • Setting nf_conntrack_max to an extremely large number "just in case" without considering the (modest but non-zero) memory cost, on memory-constrained systems.
  • Forgetting the persistent sysctl config, fixing the symptom only until the next reboot silently reverts it to the old default.

⚠ Dangerous commands

  • conntrack -F (flush) drops every tracked connection's state at once. NAT translations are lost, so established NAT'd connections break immediately. Plain TCP flows are usually re-adopted as ESTABLISHED on their next packet (nf_conntrack_tcp_loose=1, the default), but UDP flows and strict rulesets can see sudden drops. On a busy gateway/firewall host this can disconnect legitimate traffic en masse.

Review questions

  1. Why does a conntrack table filling up produce a "silent drop" rather than an explicit error response to the client?
  2. What does [ASSURED] mean in a conntrack -L entry, and why does it matter?
  3. Why would a service work fine at low connection counts and suddenly start failing new connections only once load crosses a certain threshold?
  4. What's the practical risk of running conntrack -F on a production firewall/gateway host with an established-state-dependent ruleset?