In 2014, Docker made running containers on a laptop easy, but running them in production was messy. Before Kubernetes standardized container orchestration, teams spent months writing shell scripts, task queues, and databases to schedule workloads, manage credentials, and collect logs.
A decade later, AI agents are in a similar spot. Harnesses like Claude Code, Codex, or the Gemini CLI work well in a terminal. When you try to run them at scale in the cloud, you run into the same old issues with job dispatchers, databases, sandboxes, and secret management. The problem is no longer the agent harness. The problem is cloud orchestration.
What broke when we ran agents in-process#
We ran into this problem while building Lore, our software factory. In the first version, we ran Claude Code in the same process as our backend service. It worked for simple tests, but real software engineering workloads broke it.
Long tasks like compiling code, searching big repositories, or running test suites would stall or use up all the memory. When an agent process hung, it starved the service of CPU and worker threads. Stopping a stuck child process without crashing the parent service was hard.
Cost tracking was another problem. Compute time and token counts were tangled together, so we could not see how much a pull request review cost or which prompt loop ran up a big bill.
We were also locked into Claude. The backend had to know the CLI flags and output formats of Claude Code. When we wanted to test Codex or the Gemini CLI, we had to write new code for each tool. We wanted to swap models and harnesses without touching our application code, the same way you swap a container image in Docker.
Moving execution to a separate subsystem#
The solution was to move agent execution out of the application service and into a dedicated system. We used Kubernetes for this work because it already handles scheduling, retries, and state reconciliation.
We built ai-agent-subsystem, open source under the Apache-2.0 license, around three clean parts. An AgentDefinition defines the work, including the prompt template, the model provider, and allowed tools. A Station defines the runtime environment, including the container image, resource limits, security settings, and how many past runs to keep. An Agent is a single run with specific parameters. The controller watches these resources, creates a Kubernetes Job, streams the logs, and updates the status when the run finishes.
A major benefit of using standard cluster objects is environment parity. You can run the entire system on a developer laptop inside minikube or kind using the exact same manifests that run on production clusters in the cloud. We test and develop against local clusters daily. You install it, create an Agent, and watch the controller schedule the pod and stream results.
To keep container images clean, we use an injected-kernel model. Instead of building and maintaining custom container images packed with agent tools for every project, a lightweight init container copies the agent CLI and a supervisor binary into a shared volume when the pod starts. Any base container image with glibc can run an agent immediately. If your repo needs Python 3.12, you use your standard Python image. If it needs Node or Go, you use those images without baking agent binaries into your application Dockerfiles.
We also removed the need to hardcode models or harness APIs in the caller. The controller includes a ModelCatalog that reads cluster secrets and discovers which models are ready to run across Anthropic, OpenAI, and Gemini. If you add an API key to the cluster, the catalog makes the matching models available to recipes automatically. Callers do not need to write raw YAML strings. We generate client libraries like @re-cinq/agent-contracts for TypeScript and Go straight from our D data structures in continuous integration, so client code and controller schemas never drift apart.
Another important design decision was separating git credentials. Running an agent with permission to run bash commands while holding write tokens for your code repository creates a large security risk. When opening pull requests, the system can withhold push credentials from the agent container entirely. The agent completes its work in an isolated workspace, the supervisor writes the changes to a git bundle, and a separate courier container pushes the branch and creates the pull request. The agent never sees the write token.
Numbers and operational gains#
Running agents as cluster resources changed our costs and operational overhead.
Many tasks do not need an interactive tool loop. For pull request summaries, lint reviews, or triage, an AgentDefinition can specify batch execution. The controller skips creating a pod, sends the prompt straight to the batch API for Anthropic, Gemini, or OpenAI, and checks for the result during its reconcile loop. That cuts token costs by roughly half and uses no pod compute time.
The controller itself is small. It is written in D and statically compiled, using about 50m CPU and 64 megabytes of memory, with a limit of 128 megabytes. It runs without root privileges on a read-only root filesystem with dropped Linux capabilities.
Stations set hard time limits and concurrency caps. An agent that loops forever gets killed by its deadline, and a burst of tasks does not exceed API rate limits or starve the cluster. For runs that execute arbitrary code with bypass flags, a Station can set gVisor or Kata Containers as its runtime class, intercepting system calls in a sandboxed kernel so code cannot escape the pod.
Direct peers: CRD and controller, a pod per coding agent run, vendor CLIs inside#
A handful of projects share our core model.
| Project | Language and license | CRDs | Agents, run model, results | Maturity | How it differs from us |
|---|---|---|---|---|---|
| Kelos (formerly Axon) | Go, Apache-2.0 | Task, TaskPipeline, Session, Workspace, AgentConfig, TaskSpawner, SessionSpawner, WorkerPool | Claude Code, Codex, Gemini, OpenCode, Cursor, custom. Jobs, Pods, StatefulSets, reconnectable Sessions, and warm WorkerPool. Task.status carries branch, commits, PR, token usage. | ~344 stars, 1.9k commits, Helm and CLI | Adds pipelines and fan-out, triggers for GitHub, Jira, Linear, and cron, plus a web console and interactive sessions. |
| Ambient Code Platform (formerly vTeam) | Go operator, Python runner, Next.js, MIT | session CR (operator to Job) | Claude Code only. Job per session, results on the CR. CR status and web UI. | ~131 stars, no releases | Full platform with a REST backend, multi-agent personas, and a web UI. Heavier, Claude-only, and UI-first. |
| tiny | Go, MIT | Session, Question (no operator pod) | Claude Code, Codex. Deployment, tmux, PVC, with handoff from laptop. | 16 stars, weeks old | Personal interactive sessions rather than automated task runs. Uses the same credential-less git push pattern. |
| Sympozium | Go, MIT | AgentRun | External agent harnesses through an experimental AgentHarness adapter. Job per AgentRun, or a long-lived serving mode. Token usage and estimated cost on every AgentRun. | ~664 stars, v0.11.6, under active development | Coordination layer for multi-agent systems, with channels and shared memory, rather than discrete coding-task runs. |
| ai-agent-subsystem | D | AgentDefinition, Station, Agent, ModelCatalog | Claude Code, Codex, Gemini CLI, OpenCode. Job per Agent. status.output tail, phase, and timings directly on the Agent CR. | Active releases (v0.14.0) | Minimalist design where Kubernetes is the entire API. |
Adjacent tools across the stack#
Other projects operate in the same problem space at different layers.
Plural Agent Runtime is a commercial, proprietary runtime where CRD-configured agent runs execute in Docker on your cluster and open pull requests. It provides a similar workflow, but as a closed system tied to Plural.
In the CNCF, kagent is an Apache-2.0 open-source project introducing CRDs like Agent, AgentTemplate, ModelConfig, and RemoteMCPServer, with sessions stored in PostgreSQL. It targets long-lived chat agents with checkpointing and suspension rather than discrete, one-shot coding tasks that end in a commit or pull request.
At the container level, projects like the SIG Apps agent-sandbox (Apache-2.0) provide warm worker pools and isolation, which fits underneath the operator layer. Ark from McKinsey (Apache-2.0) defines generic agentic runtimes with teams and queries, but is not specific to software engineering workloads.
Other orchestrators like OpenHands (MIT), Engrams (Apache-2.0), agent-swarm, Superset, Claw Orchestrator, Bernstein, and NEEDLE run agents inside Firecracker microVMs, Docker containers, or git worktrees. They are not Kubernetes-native and do not provide a CRD model.
Takeaways from the comparison#
Kelos is the one to watch, sharing the same core model with more surface area.
When you look at how competitors approach this space, most of them combine execution and higher-level workflow orchestration into a single project, as the table shows. That approach turns the Kubernetes API server and etcd into a state machine for multi-step application workflows, which requires extra cluster dependencies like cert-manager.
We made a deliberate choice to separate execution from workflow orchestration. The ai-agent-subsystem is an execution runtime only. It does not know about pipelines, issue trackers, or GitHub events. It runs an agent in an isolated pod or dispatches a batch request, streams output events, and exits.
The orchestration of those stations belongs in a separate layer, which is where our Floor project fits in. Floor is a workflow engine built for pipelines of AI agents. It models work as assembly lines, where a graph of nodes describes the overall process and each step is worked by an agent at an ai-station, by a service worker, or by a human. Floor decides what step comes next, while cluster workers pull tasks and create plain Agent resources in Kubernetes.
This separation is the missing piece that keeps the cluster operator predictable and easy to secure. We will share more details on the design of the Floor project and how it coordinates assembly lines across stations in a future article.
Keeping the infrastructure simple#
Our focus is on keeping the operator simple to run. We do not run a separate database, a web console, or admission webhooks. Callers can use standard Kubernetes tools.
The installation uses a single manifest with digest-pinned images signed with cosign, with SLSA provenance and software bills of materials. Treating agents as plain Kubernetes resources gives us access controls and audit logs without building new infrastructure.
