Agents as Code: A Kubernetes-Native Approach

An alert fires at 3am. Someone asks an AI agent to triage it. The agent reads logs, forms a theory, and posts a summary. It works.

Now answer the questions your auditor will ask next month: Which model ran? What instructions did it see? What could it have reached if it had gone wrong? Who was allowed to invoke it? Where is the run?

For most agents in production today, every one of those answers lives in a chat window on someone's laptop. That makes the agent a favor someone did for you once, not a workload you operate.

Desired state vs. desired behavior

Infrastructure as Code made infrastructure a versioned, testable, reproducible artifact. Agents as Code applies the same engineering model to an agent: its runtime, instructions, knowledge, reachability, permissions, and eventually its expected behavior, declared as artifacts you can review, deploy, and inspect after the fact.

The difference is in what gets managed:

Infrastructure as Code manages desired state.
Agents as Code manages desired behavior.

The second half is not solved. Behavior is probabilistic, so checking it needs evals, policy checks, and trajectory analysis, and nobody reconciles those as reliably as they reconcile replicas. What is solvable today is everything underneath: the part I'll call agent infrastructure as code. That is the scope of this post.

Beyond agent.yaml

Putting an agent config in Git is useful and incomplete. Most teams stop at agent configuration as code: one file with a prompt, a model name, and a tool list.

The prompt is the least dangerous thing about an agent. A versioned prompt tells you nothing about whether that agent could read every Secret in the cluster, exec into pods, or push to a private repo. If Agents as Code only versions prompts, you can ship a perfectly reviewed prompt attached to an agent that can do anything the cluster can do.

The unit worth versioning is the whole envelope around the prompt: who may invoke, which runtime, which behavior pack, which knowledge, what it may reach, where results go, and what the run left behind.

Why an operator

Running an agent framework inside a Pod schedules the framework once. The runs inside it are threads in one process, sharing one credential set, one blast radius, and one queue, and leaving no separate record of any individual run.

Making each run a cluster object changes what you get without building it:

• Credentials stay out of the agent definition. Keys live in Kubernetes Secrets and are mounted into the run, so no API key sits in a prompt, a repo, or a chat log, and rotating a key does not touch the recipe.
• Isolation is per run rather than per platform. Each run gets its own namespace scope, resource limits, and read only root filesystem, so one agent's reach is not every agent's reach.
• The scheduler places the work. Hundreds of concurrent runs spread across nodes by bin packing you did not write, land on GPU or spot pools when you ask for them, and trigger the cluster autoscaler when a burst needs more capacity.
• You inherit an orchestrator. Retries and backoff, cron schedules, TTL and garbage collection through owner references, eviction and node drain handling all come from Job and controller semantics instead of your code.
• Everything else in the cluster is already in reach, because an agent run is an ordinary workload. Secrets and CSI drivers supply credentials, Prometheus scrapes it, your existing log pipeline collects it, and a webhook can create work through PilotClient.
• Declarative means reviewable. Composition lives in Git, diffs and rollbacks work normally, and admission validation rejects a bad agent config before a Pod starts.
• Bursty work costs less. Agent runs are short and spiky, so Jobs on autoscaled nodes bill for the run instead of for an always-on framework server, with namespace quotas and PilotBudget capping spend per tenant.

Running hundreds of agents becomes an ordinary capacity question instead of a new distributed system to operate.

Put together: desired composition plus a task becomes an actual Job or session and an observed status, continuously.

One way to model all of that is a single resource:

# Illustrative strawman — not a real API.
apiVersion: agents.example/v1
kind: Agent
spec:
  model: reasoning-model
  prompt: "Triage the incoming alert..."
  tools: [kubectl-get, kubectl-logs, send-mail]
  apiKey: sk-...

This is the mega-CRD, and it fails for the same reason a single app.yaml fails: nothing is reusable, the credential is inline, and reachability is implied by a tool name rather than declared and reviewable. Better to compose a run from smaller versioned objects.

Alteus Pilot: composition as cluster objects

At Alteus we build this layer as Alteus Pilot: a multi-CRD Kubernetes operator, written in Go, that runs AI agent work as ordinary cluster workloads and uses Kubernetes mechanisms (Jobs, Secrets, PVCs, ServiceAccounts, namespaces) for isolation, scale, and credentials.

An operator is a Pod that does a human admin's job: it watches objects and acts when reality drifts from what you declared. A CRD (Custom Resource Definition) extends the Kubernetes API so your declaration becomes a first-class object, which is why you can run kubectl get pilottask and not only kubectl get pods. Cloud native, in this context, means the declaration is code and the thing it describes runs in containers on Kubernetes.

The 3am triage agent, composed as Pilot objects. Field shapes are simplified for reading; the API reference is normative.

apiVersion: pilot.io/v1alpha1
kind: PilotProvider          # which runtime: agent image + model + AI credential ref
metadata:
  name: cursor
spec:
  image: ghcr.io/alteus/pilot-provider-cursor:v0.3.0
  model: reasoning-model
  apiKeySecretRef:
    name: cursor-api-key
---
apiVersion: pilot.io/v1alpha1
kind: PilotSkill             # the behavior pack: instructions + files, materialized on disk
metadata:
  name: first-responder
spec:
  git:
    url: https://git.internal/agents/first-responder.git
    ref: v1.4.0
---
apiVersion: pilot.io/v1alpha1
kind: PilotAccess            # external reach, declared and reviewable
metadata:
  name: k8s-readonly
spec:
  kubeconfigSecretRef:
    name: triage-readonly-kubeconfig
---
apiVersion: pilot.io/v1alpha1
kind: PilotDelivery          # where results go when the run finishes
metadata:
  name: mail-support
spec:
  smtp:
    to: support@example.com
    credentialsSecretRef:
      name: smtp-creds
---
apiVersion: pilot.io/v1alpha1
kind: PilotAgent             # the reusable recipe, composed from the above
metadata:
  name: first-responder
  namespace: client
spec:
  provider:
    name: cursor
  skill:
    name: first-responder
  access:
    - name: k8s-readonly
  delivery:
    - name: mail-support
  budget:
    name: first-responder

Nothing has run yet: PilotAgent is only a recipe. One execution is a separate object.

apiVersion: pilot.io/v1alpha1
kind: PilotTask
metadata:
  name: alert-2f91c4
  namespace: client
spec:
  agent:
    name: first-responder
  mode: job                  # or: interactive
  prompt: |
    PrometheusAlert: KubePodCrashLooping in namespace payments.
    Triage and summarize probable cause.

The recipe gets reviewed once and versioned in Git. The tasks that reference it are the audit trail.

The resource set

Pilot's CRDs fall into five groups. You do not need all of them to run an agent: a Provider, a Skill, an Agent, and a Task are enough.

Two of these are less finished than the rest. PilotClient's webhook path is the tested one, and its frontend web interface is still being wired. PilotSwitch is new. If you are evaluating Pilot, evaluate it on Provider, Skill, Agent, Task, Access, and Delivery.

The filesystem contract

Cluster APIs are half the infrastructure. The other half is a fixed layout inside the container, so every provider image and every sidecar sees the same world:

/workspace/                    portable run state (PVC or emptyDir)
  agent/{chats,history}/       session JSONL — what the agent actually did
  skill/                       PilotSkill material
  prompt/prompt.md             the instruction
  assets/<name>/               PilotAsset clones and downloads
  variables/                   mode, model, budget, token usage
  access/<name>/               PilotAccess mounts
  delivery/{out,artifacts}/    results for sinks
  delivery/status.jsonl        delivery attempt trail
/provider/
  bin                          vendor wrapper entrypoint
  api_key                      AI provider key — remounted each run

Three choices in that tree matter most:

• The prompt is a file, not argv, so large instructions never hit a process table or an "argument list too long" wall.
• Skill, assets, access, session trails, and delivery output are all paths under /workspace. The run stays portable and inspectable after the Job exits, and the same Task shape survives a provider swap.
• The AI key lives at /provider/api_key, outside /workspace, so a workspace can be forked or debugged without baking the credential into portable state.

Two modes follow from the same contract: job runs skill plus prompt to completion, and interactive keeps a session alive under a TTL. Only /workspace and /tmp are writable, and the root filesystem is read only, the same hardening you would apply to any production workload.

The contract is also what makes providers swappable. Change the image and the API key in PilotProvider, and tasks, skills, assets, and deliveries stay exactly as composed.

Isolation by default

A task cannot use the Kubernetes API, exec into pods, read logs, clone private repos, or reach external systems unless that was explicitly configured. Four distinct layers do that work, and conflating them is the usual source of confusion:

• Caller identity, through PilotClient: who may create work, and against which agent.
• Object scope: namespaced CRs stay local, and anything cluster-scoped is an explicit choice.
• In-cluster rights: a capability list bound to the task's own ServiceAccount and Role, never the operator's.
• External reach: PilotAccess plus Secrets only, with no inline credentials.

So the triage agent above reaches a cluster because someone reviewed a read-only kubeconfig into a Secret and referenced it, not because a tool named kubectl-get appeared in a list.

Runs as first-class artifacts

Every execution should leave more than a log line. In Pilot, the PilotTask is the run. The operator owns the agent Job, maps Job status onto task phase, and can attach a workspace PVC so the tree survives for debugging, forking, or a re-run.

A run happens roughly like this: compose the objects in Git or via the API; a caller creates a PilotTask; the operator resolves the agent's defaults and creates an owned Job with /workspace and /provider/api_key mounted; init containers and sidecars materialize skill and assets; the provider image reads prompt, skill, and assets from disk. Status lands on the Task, deliveries push results out, and the PVC keeps the tree readable after exit.

After the 3am triage finishes, you can answer the auditor: which agent recipe and skill version ran, what prompt and assets it saw, which access it was granted, how many tokens it burned, where the summary was delivered, and what phase the Task ended in. That lives on the CRs and under /workspace, not in a chat history.

Two forms of auditability, doing different jobs:

Git and kubectl apply tell you what you declared the agent infrastructure to be.
Tasks and their workspaces tell you what actually ran.

Monitoring starts where you would expect, with reconcile health, Job phase, token usage, and delivery success or failure. It has somewhere to grow, because the run is an object rather than a log afterthought.

Why this is a business case, not just an architecture

• Audit and compliance get a trail of what was declared and what actually ran, from two independent sources.
• Least privilege means an agent cannot wander the cluster or the network unless that reach was reviewed as config.
• Provider independence means swapping the agent image does not rewrite callers, skills, or deliveries.
• Reproducible workspaces mean a stuck or disputed run gets forked and inspected instead of reconstructed from memory.

We are building toward schedules and watchers that spawn tasks from cluster events, stronger ServiceAccount and access enforcement, fork-and-continue as a normal engineering loop, and deeper observability of agent trajectories, with evals and drift response after that. All of it treats agent work as engineered cluster workloads.

Closing

Running an agent in a Pod is easy. Operating hundreds of agents safely is a platform problem. You need declared composition, isolated credentials and permissions, distributed execution, standard integrations, and a record of every run.

Kubernetes already provides most of the machinery. Pilot turns agent work into Kubernetes resources that teams can review, schedule, isolate, observe, and audit like their other workloads. That is the practical foundation of Agents as Code.