Fixing Spontaneous 'Connection Reset by Peer' Errors in Kubernetes kube-proxy
What's the problem?
Resolve intermittent TCP connection resets in Kubernetes by configuring iptables to drop invalid conntrack packets. Improve stability for long-lived connections.
Why does this happen?
The issue occurs because the Linux kernel's conntrack module allows packets marked as 'INVALID' to pass through the KUBE-FORWARD chain. These invalid segments trigger the kernel's TCP state machine to issue a RST packet, forcefully terminating established connections.
Code Example
iptables -A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROPHow to fix it
To resolve this, update your kube-proxy configuration to ensure explicit dropping of invalid conntrack packets. You can achieve this by ensuring your CNI or kube-proxy implementation includes an iptables rule that identifies and silently drops these packets before they reach the TCP stack. In managed environments, ensure your cluster version is updated to a release containing the 'INVALID' conntrack drop fix, or manually inject the rule into the KUBE-FORWARD chain.