JupiteX Get the app
Science & Technology21 Nov 2026 · about 7 min

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

The brief

The timeouts came from three bugs across different networking layers, not one simple configuration problem. Pods trying to reach the Kubernetes API server sometimes encountered connections that could not complete. The issue mattered because it blocked developers from deploying to production and was difficult to diagnose from the timeout alone. Packet captures showed the control plane replying to new TCP connection requests with stale ACK packets from previously closed connections. Google traced the problem to packet reordering in Private Service Connect, Envoy’s handling of half-closed connections, and a Linux kernel bug that discarded certain reordered FIN packets. Some sockets were left indefinitely in FIN WAIT 2. Google addressed the causes in stages: it rolled back a Private Service Connect change to restore connectivity, corrected proxy protocol handling, and added regression tests. A Linux kernel patch then fixed the underlying TCP issue. The fix was backported to Linux 6.6 and 6.1 LTS branches and reached GKE customers in version 1.33.5 and later. The case shows why captures from multiple layers can be essential.

01

What caused some pods to time out when connecting to the Kubernetes API server?

The timeouts came from three bugs across different networking layers, not one simple configuration problem. Pods trying to reach the Kubernetes API server sometimes encountered connections that could not complete. The issue mattered because it blocked developers from deploying to production and was difficult to diagnose from the timeout alone.

Packet captures showed the control plane replying to new TCP connection requests with stale ACK packets from previously closed connections. Google traced the problem to packet reordering in Private Service Connect, Envoy’s handling of half-closed connections, and a Linux kernel bug that discarded certain reordered FIN packets. Some sockets were left indefinitely in FIN_WAIT_2.

Google addressed the causes in stages: it rolled back a Private Service Connect change to restore connectivity, corrected proxy protocol handling, and added regression tests. A Linux kernel patch then fixed the underlying TCP issue. The fix was backported to Linux 6.6 and 6.1 LTS branches and reached GKE customers in version 1.33.5 and later. The case shows why captures from multiple layers can be essential.

02

About how many developers were blocked from deploying to production?

The reported scale was around 800 developers blocked from deploying to production. That figure gives a sense of the operational impact: an intermittent network problem can disrupt work even when it does not affect every connection continuously. The article describes one customer’s experience, rather than a broader estimate of how often this kind of failure occurs.

The customer ran multiple GKE clusters and used Private Service Connect for private access to the control plane. Pods experienced intermittent timeouts when they tried to reach the Kubernetes API server. Packet captures showed stale acknowledgments appearing on new TCP connection attempts, and investigators found three interacting causes across PSC, Envoy, and the Linux kernel.

Google first rolled back a PSC change to restore connectivity, then corrected proxy protocol handling and added regression tests. A Linux kernel patch addressed the underlying TCP issue. The article does not provide a population-wide failure rate or say that all 800 developers were affected in the same way. The kernel fix reached GKE customers in version 1.33.5 and later.

03

What is Private Service Connect, and how does it let GKE clusters reach a private control plane?

Private Service Connect, or PSC, is a Google Cloud networking service for private connectivity between a consumer network and a service. In general, it lets a consumer reach a service through a private endpoint in its network instead of relying on a public connection. That basic definition is general background; the article does not describe the customer’s PSC configuration in detail.

The customer ran multiple GKE clusters and used PSC for private control-plane access. Pods in those clusters tried to reach the Kubernetes API server, but some connections timed out. The investigation found that PSC packet reordering was one of three interacting causes, alongside Envoy’s half-close behavior and a Linux kernel bug involving reordered FIN packets.

Google first rolled back a PSC change, restoring connectivity, then corrected proxy protocol handling and added regression tests. A Linux kernel patch fixed the underlying TCP issue. The case shows that private connectivity depends on more than the cloud networking layer: proxy behavior and operating-system TCP handling also mattered. The fix reached GKE customers in version 1.33.5 and later.

04

What happened to connectivity after Google rolled back the Private Service Connect change?

The rollback restored connectivity for the customer, providing an immediate response to the intermittent API access failures. But it addressed the problem at the Private Service Connect layer; it did not by itself correct every cause investigators later identified. That distinction matters because recovering service and fixing the underlying state-machine problem were separate steps.

The investigation found three interacting causes: packet reordering in PSC, Envoy’s half-close behavior preventing TCP connections from closing properly, and a Linux kernel bug that discarded certain reordered FIN packets. The combined behavior could leave sockets stuck in FIN_WAIT_2, while new connection attempts encountered stale ACK packets from older connections.

Google then corrected proxy protocol handling and added regression tests. Finally, a Linux kernel patch fixed the underlying TCP issue. The patch was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches; it reached GKE customers in version 1.33.5 and later. The staged response shows why service recovery did not replace fixing the deeper kernel behavior.

05

How did packet reordering, Envoy’s half-close behavior, and a Linux kernel bug combine to cause the failures?

The failures arose because individually plausible behavior across three layers combined badly. PSC packet reordering changed when packets arrived. Envoy’s half-close behavior prevented some TCP connections from being properly closed. A Linux kernel bug then discarded certain FIN packets when they arrived reordered. No single layer’s symptom fully explained the intermittent API timeouts.

The kernel problem could leave sockets stuck indefinitely in FIN_WAIT_2. In packet captures, engineers saw the control plane answer new TCP connection requests with stale ACK packets belonging to connections that had already closed. That made a new connection appear to encounter old connection state. The article describes this as a cross-layer problem involving PSC, virtual network interfaces, proxies, and the TCP state machine.

Google rolled back the PSC change to restore connectivity, corrected proxy protocol handling, and added regression tests. The final fix was a Linux kernel patch authored by Eric Dumazet. It was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches, reaching GKE customers in version 1.33.5 and later.

06

What does it mean for a TCP socket to be stuck in the FIN_WAIT_2 state?

FIN_WAIT_2 is a TCP connection state reached during an orderly close. In general TCP behavior, one endpoint has sent a FIN to say it has finished sending data, and the other endpoint has acknowledged that FIN. The first endpoint is then waiting for the peer to send its own FIN. This explanation is standard TCP background; the article does not define the state.

The article reports that a Linux kernel bug discarded certain reordered FIN packets. In the GKE case, this could leave sockets stuck indefinitely in FIN_WAIT_2. Packet captures also revealed stale ACK packets from previously closed connections appearing as responses to new connection requests. That made the timeouts hard to distinguish from problems at other networking layers.

Google’s final fix was a Linux kernel patch, after a PSC rollback and corrections to proxy protocol handling. The patch was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches. It reached GKE customers in version 1.33.5 and later. The case illustrates why TCP state and packet captures mattered alongside cloud and proxy investigations.

07

How does TCP use sequence numbers, acknowledgments, and FIN packets to deliver data and close connections reliably?

TCP uses sequence numbers to keep track of data in a connection, and acknowledgments tell the sender what data the receiver has received and what it expects next. This lets endpoints detect missing data and retransmit it. These are general TCP principles; the article focuses on a failure involving acknowledgments and connection closing rather than explaining the protocol basics.

To close a connection, an endpoint sends a FIN to signal that it has no more data to send. The other endpoint acknowledges it, and can still send its own remaining data before sending a FIN in return. A FIN therefore closes one direction of sending, not necessarily both directions at once. Packet ordering matters because endpoints rely on receiving and processing these control packets correctly.

In the reported GKE failure, PSC packet reordering, Envoy’s half-close behavior, and a Linux kernel bug combined; the kernel discarded certain reordered FIN packets. Some sockets remained in FIN_WAIT_2, and captures showed stale ACKs on new connection attempts. A kernel patch addressed the underlying issue and reached GKE customers in version 1.33.5 and later.

This brief was written by AI from the original reporting and checked by other models. Names, figures and quotes come from the source; read it for full context.

Read more in the JupiteX app

Pulse is free. New stories every 4 hours, each one broken into the questions that explain it.

Or read more news on the web