ManifestBasedAdmissionControlConfig: kube-apiserver startup risks

Manifest-based admission config (KEP-5793): ‘policy as files’ with a boot-time guillotine

KEP-5793 formalizes manifest-loaded admission configuration behind the ManifestBasedAdmissionControlConfig feature gate. Enable it, point --admission-control-config-file at an AdmissionConfiguration containing staticManifestsDir, and the API server starts reading policy objects off disk instead of (only) etcd. Sounds boring. It’s not.

The gotcha is intentional strictness: malformed manifests or duplicate names across files cause the kube-apiserver to fail startup. No degraded mode. No ‘skip and continue’. If you’ve ever had a control-plane node reboot during an incident, you can already see the shape of the postmortem.

So what? This is an ops feature masquerading as a security feature. It’s paying down the architectural debt of ‘we can’t enforce cluster policy if etcd is unhealthy’ and ‘we can’t prevent privileged actors from deleting guardrails’. It also shifts your blast radius: you’re now one bad YAML away from a control-plane outage unless you treat those files like code (review, validation, rollout, canary, rollback).

Admission behavior change: ExcludeAdmissionWebhookVirtualResources default-on

Also in KEP-5793: ExcludeAdmissionWebhookVirtualResources becomes default enabled in 1.37. That changes what traffic hits ValidatingAdmissionWebhook / MutatingAdmissionWebhook. Any team using webhooks as a catch-all policy net needs to re-check what’s actually being evaluated after the upgrade. Expect ‘why didn’t our webhook fire?’ tickets.

Graceful Leader Transition (KEP-5366): fewer lease-timeout stalls during rolling upgrades

KEP-5366 adds a beta mechanism to release leader-election leases on graceful shutdown, behind ControllerManagerReleaseLeaderElectionLockOnExit. The point is simple: stop waiting out the full lease duration during planned restarts. That’s a quality-of-life improvement for HA control planes, and it reduces the window where your controllers are effectively paused.

So what? If your automation or SLOs implicitly depend on the old ‘lease expires eventually’ behavior, you’ll see different timing. That’s usually good. Timing changes are still timing changes—test your upgrade choreography.

Concurrent Watch Object Decode (KEP-6178): watch-path pressure relief, with new observability hooks

KEP-6178 (beta) focuses on the watch pipeline, where large clusters burn CPU doing decode work that scales with ‘number of watchers × object churn’. The KEP calls out watch/list latency SLOs and references standard API server latency metrics, plus a proposed gauge apiserver_watch_decode_inflight to make concurrency visible.

So what? If you’re running high-cardinality watches (controllers, agents, service meshes, custom operators), you should care. The watch path is where clusters get slow first, then fall over in more creative ways.

Tooling foot-gun: JSON tag change for inlined TypeMeta (PR #138260)

PR #138260 switches inlined TypeMeta JSON tags from ,inline to json:"". The maintainers say it doesn’t change Go JSON behavior (because inline wasn’t a recognized option), but it does change assumptions in generators and schema tooling. If you’ve got bespoke parsers or brittle reflection-based code, run it against v1.37 prereleases now, not after you cut a release branch.

Dependency churn signal: etcd image bump to 3.7.0-rc.0 (PR #139427)

PR #139427 bumps the default etcd image to 3.7.0-rc.0 in the v1.37 pre-release line. You don’t need to panic, but you do need to remember that etcd compatibility issues don’t show up in unit tests—they show up as slow elections, compaction weirdness, and tail-latency spikes when you’re already on fire.

Kubernetes has been slowly moving from ‘flexible, sometimes ambiguous operational guidance’ to ‘hard requirements enforced at runtime.’ v1.37 continues that trend. The admission changes are about control-plane determinism: policy should exist even when the cluster is sick, and it should be hard to remove by accident or malice. The watch-path work is about scale: cloud cost isn’t going down, and the easiest way to cut spend is to stop wasting CPU on redundant decode/serialization in hot loops.

Competitors didn’t force this. Reality did. Multi-tenant clusters, compliance requirements, and fleets that reboot themselves under you are the forcing function. Operators asked for guardrails that survive bad days. The maintainers are obliging—with sharper edges than most teams are ready for.

Test the prerelease (alpha/beta/rc) the boring way—in a disposable environment with production-like admission and webhook load.

# Grab the latest v1.37 artifacts (alpha/beta/rc) from upstream releases
# (exact URL varies by cut; start from the release tag and pull server/node tarballs)

# If you are kubeadm-based, validate your control-plane flags and feature gates explicitly.
# Example (API server):
--feature-gates=ManifestBasedAdmissionControlConfig=true
--admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml

# In AdmissionConfiguration, set:
# staticManifestsDir: /etc/kubernetes/admission/static

Red Flags (watch your logs and metrics):

• kube-apiserver fails to start with validation errors: you shipped an invalid manifest or duplicate object name across files (expected behavior, but catastrophic if it happens during reboot).
• apiserver_manifest_admission_config_controller_automatic_reloads_total{status="failure"} increments after a file change: your rollout didn’t apply; you’re running on last-known-good.
• Webhook volume drops unexpectedly after upgrade: likely impacted by ExcludeAdmissionWebhookVirtualResources default-on behavior in 1.37.
• Generator / OpenAPI tooling errors around inline fields: PR #138260 fallout; fix your toolchain assumptions before blaming Kubernetes.
• Node churn / NotReady after upgrade attempts: treat runtime version validation as a hard precheck, not a post-upgrade cleanup task.

🛠️ Try These Free Tools

See all free tools →