Kubernetes Gateway API vs Ingress vs Service LoadBalancer: what breaks

Another “minor” networking change. What broke this time, and what did the changelog not bother to mention?

I’ve watched teams treat Ingress and Gateway API as a YAML refactor, then spend a weekend chasing 404s, missing client IPs, and TLS weirdness. The docs say “portable.” Your controller says “conformant.” Production says, “cute.”

Concerns first: the stuff that actually pages you

Start here, not with the pretty resource model. Your edge is a shared dependency, so every new controller, CRD, and policy object expands the blast radius unless you build guardrails.

• Controllers lie by omission: A conformance badge does not tell you if your exact features work together. HTTPRoute might pass, but your combo of SNI, gRPC, websocket upgrades, and header rewrites can still fail in one controller build.
• Migrations fail on “boring” integrations: DNS automation, certificate rotation, WAF defaults, and identity-aware proxies do not migrate 1:1. Your old Ingress annotations hid that complexity. Gateway API forces you to name it.
• Two controllers can fight over one edge: If you run Ingress and Gateway side-by-side, scope them. Otherwise you get flapping config, duplicated load balancers, or a webhook rejecting objects “randomly.”
• Rollback often depends on things that are already broken: GitOps rollback works only if the controller still reconciles fast and predictably. DNS rollback still wins because it ignores your cluster’s mood.

If your rollback plan takes longer than your outage budget, do not migrate the edge yet.

The mental model (so you stop swapping these like Lego)

These are not substitutes. They sit at different layers, and they fail differently.

• Service type=LoadBalancer: You ask the cloud integration for an external VIP and forward TCP or UDP to Service endpoints. You get simplicity, plus one bill per Service. You do not get standard L7 behavior without provider features or annotations.
• Ingress: You get an L7 route object (host and path to Service) and then you bolt on everything that matters with annotations or controller CRDs. It’s boring. Boring keeps uptime intact.
• Gateway API: You split ownership. Platform owns the shared data plane (Gateway). Teams own attachable Routes. Policies attach “near” what they control. In theory, this reduces annotation debt. In practice, controller support decides what you can actually ship.

Pick by requirement (and be honest about your org)

People ask “Gateway API vs Ingress?” but the real question is, “Do we need multi-team delegation and safe routing changes, or do we need fewer moving parts?”

• You should default to Service type=LoadBalancer when: You expose raw TCP or UDP, you want a stable IP for allowlists, and one team can own the cost and the failure. This stays clean for databases and custom protocols.
• You should default to Ingress when: One team owns most routes, your annotation set already behaves like an internal standard, and you value operational predictability over clean abstractions. Some folks hate that. I don’t.
• You should reach for Gateway API when: You share the edge across teams and you cannot keep playing “please do not touch my Ingress” in Slack. Delegation and attachment controls matter more than raw simplicity.

So. If you run a single-team cluster and your nginx annotations already work, Gateway API probably adds risk without paying rent.

Concrete patterns (the ones that survive audits and incident reviews)

The thing nobody mentions is RBAC. If you let application namespaces edit the shared Gateway, you just recreated the “anyone can break prod” problem with nicer nouns.

• Platform-owned Gateway + team-owned HTTPRoutes: Platform owns the listener, hostname, TLS termination, and the guardrails. Teams attach routes explicitly via parentRefs, so you can audit who attached what and when.
• Canary via weighted backends: Gateway API puts weights in the route spec, which beats the Ingress annotation scavenger hunt. Controller behavior still varies, so verify whether weights apply per request or per connection in your controller.
• Ingress with enforced annotation hygiene: If you keep Ingress, standardize allowed annotations and lint them in CI. “Just paste this annotation from a blog” is how you buy outages.

Migration paths that do not pretend you can “just switch”

I’d wait a week before you flip anything user-facing, unless you already run the new controller in staging with real traffic patterns. Why rush this?

Path 1: Ingress to Gateway API without changing the external IP

This works only if your chosen controller can run both APIs on the same data plane, or at least keep the same front door while you swap the config model. They claim it’s smooth, but I’ll believe it when I see it under load.

• Start with one low-stakes hostname: Mirror routing for one internal or low-revenue host. Compare status codes, headers, timeouts, and auth redirects. Do not skip the redirects. Nobody tests redirects.
• Freeze annotation creep: Stop accepting new Ingress annotations for “one last feature.” Push new behavior into whatever your Gateway policy story is, even if that story includes vendor CRDs.
• Cut over with a rollback you can do half-asleep: DNS with a low TTL still gives the simplest emergency exit. Listener reassignment can be fast too, but only if your controller makes it boring.

Path 2: One giant shared Ingress to delegated Gateway + Routes

This is the migration most platform teams actually need. You already centralize ingress-nginx because you do not trust every namespace with edge config.

• Create a platform namespace for the edge: Put Gateway and cert material in something like gateway-system. Lock it down.
• Use allowedRoutes like you mean it: Start with a namespace label selector such as exposure=public. Then enforce that only approved namespaces carry that label. Otherwise “delegation” becomes “free-for-all.”
• Split Gateways by risk tier: Keep revenue endpoints on their own Gateway or listener with tighter change control. Do not let a “marketing microsite” route share fate with checkout.

Production gotchas in 2026 (still true, still ignored)

Gateway API looks tidy on a slide. Real traffic makes it messy.

• Feature support is a matrix, not a checkbox: Verify your controller for TLS mode, SNI behavior, HTTP/2 and gRPC, websocket upgrades, header rewrites, request timeouts, retries, and traffic weights. Then test combos, not single features.
• Annotations do not map cleanly to policies: Timeouts might live at route level in one controller and only at listener level in another. Auth policies usually land in vendor CRDs. That’s fine, but call it what it is.
• Status conditions matter, but they are not magic: Treat “Accepted” and “Programmed” as a gating signal, not a guarantee. If your pipeline pushes a Route and the controller never programs it, you want CI to fail before users do.
• Node and CNI problems impersonate routing bugs: I’ve seen conntrack exhaustion look like “Gateway broke TLS.” Check node readiness, conntrack, and dataplane pod health before you rewrite routes in a panic.

How I test routing changes (because hope is not a strategy)

Test one hostname on a shadow domain first. Then you can break things without breaking trust.

• Use a parallel hostname: Serve pay-gw.example.com from the new path while pay.example.com stays on the old one. Run synthetic probes that assert headers, cookies, redirects, and latency, not just “it returns 200.”
• Promote by weight before DNS when you can: Start at 1 to 5 percent for a full business cycle, not five minutes. Alert on 5xx rate, p95 latency, and auth error rate. Then increase.
• Validate the API server will accept your config: Run server-side dry-run in CI, then enforce policy with OPA or Kyverno. Block wildcards and forbidden hostnames at the door.

Other stuff you should probably check too: cert rotation timing, DNS automation behavior, cloud LB quotas, and the usual.

Bottom line (grudging recommendation)

Use Service type=LoadBalancer for simple L4 exposure with clear ownership and a clean failure boundary. Keep Ingress if you value stability and your annotations behave like a standard. Adopt Gateway API when you actually need multi-team delegation and you can afford the extra moving parts.

If you still feel tempted to migrate everything this week, ask yourself one question. What’s not in your controller’s release notes that you will discover at peak traffic?