Reproducible Embedded Builds, or Becoming the Agent's Compiling Slave

If you’re about to put an AI coding agent to work on your embedded codebase, there’s a decision waiting for you at the very first step, and most teams make it by accident. The decision is whether the agent can build the code on its own, or whether it has to keep coming back to you every time the build breaks — and a build that only works on one engineer’s machine breaks constantly. Get this wrong and the cost is not a one-time annoyance; it’s a recurring tax on your most expensive people, paid every time the agent hits the same wall, on the agent’s schedule rather than yours. Get it right and the same work is done once, up front, bounded, and never paid again. Stated plainly: either you do the bounded work now that lets the agent compile the code unsupervised, or you become the agent’s compiling slave — the person it comes back to, every time, to fix the build it can’t fix itself. There is no third branch where nobody does the work. “Do nothing” just picks the second option by default, and it’s the more expensive one.

This is the foundation piece of a larger idea I’ve written about separately — that embedded, built the right way, is a better place for an AI coding agent than web development is (see Why Embedded Is a Better Place for AI Agents Than Web Development). That whole argument rests on one thing being true first: the agent has to be able to build your code. This post is about making that true.

What is a reproducible, agent-ready build?

A reproducible build is one that produces the same working result from the same source on any machine set up to the documented spec, without depending on state that happens to live on one engineer’s laptop. An agent-ready feedback loop — the build, test, and verification infrastructure a codebase needs so an AI coding agent can compile, test, and iterate without a human re-checking every step by hand — starts with exactly that reproducible build, because nothing downstream can run until the code compiles. Put the two together and “agent-ready build” means something concrete and testable: a fresh environment, provisioned from a written spec, compiles your firmware to a correct binary on the first try, with no human in the loop. If that isn’t true today, it’s the first thing to fix, and everything else in the agent-ready loop waits on it.

Compiling is where the trouble shows up first

Compiling is the first thing an agent has to do to make any progress at all, which makes it the first place a shaky foundation becomes visible. No test runs, no refactor lands, no review means anything before the code builds — so a build that only works sometimes doesn’t stay contained to the build step. It poisons every step downstream of it. An agent that can’t trust the build can’t trust the compile-time checks (correctness checks the compiler evaluates before any test runs), can’t trust the test suite, can’t trust anything, because all of it is standing on a foundation that shifts between runs. This is also why the failure is easy to underestimate: on a good day the build works, you see it work, and you conclude it’s fine. The agent doesn’t get good days and bad days averaged out — it hits the bad-day build on task after task, and each time, the work stops until someone intervenes. The payoff of getting this right is that every later step — tests, mocks, the full agent loop — inherits a foundation it can rely on instead of one it has to keep working around.

What being the compiling slave actually costs

Here’s the failure mode in practice, because it’s worth seeing concretely before we talk about fixing it. The agent picks up a task, tries to build, and the build fails for a reason that has nothing to do with the task — a toolchain version mismatch, a missing system package, an environment variable only your shell has. The agent can’t fix that. It surfaces the error and waits. You context-switch off whatever you were doing, fix the environment by hand, and hand it back. Twenty minutes later, a different agent run — or the same one on a fresh machine — hits the same wall, because nothing about the underlying build changed. You fix it again.

Notice what’s actually being spent here. It isn’t the twenty minutes; it’s the interruption, the context-switch, and the fact that this repeats without bound for as long as the build stays fragile. That’s the real shape of the trade. The work of making the build reproducible is finite: you do it once, deliberately, and it’s done. The work of not making it reproducible is unbounded: the same manual fix, the same interruption, recurring every time an agent — or a new hire, or you on a fresh laptop — hits the same wall. You will resent it when it happens this way, and you’d be right to, but the choice was yours before it ever got to that point. The one-sentence version, the one worth quoting to whoever signs off the time: making the build agent-ready is bounded, one-time work; not doing it is unbounded, recurring work — and the second bill never stops arriving.

The three ways a build betrays an agent

Fragile builds fail in more or less three ways, and naming them makes the fix obvious rather than mysterious.

The first is toolchain drift — the compiler, the linker, the build tool, or a library is a different version than the one the code was written against, so a build that worked last month fails now, or works on your machine and fails on the continuous-integration (CI) runner — the clean, automated server that rebuilds every change from scratch. The second is the works-on-my-machine environment — the build quietly depends on something present only on one engineer’s laptop: an installed SDK (software development kit), a PATH entry, an environment variable, a file outside the repo. Nothing records that dependency, so no fresh machine reproduces it. The third is hidden host state — the build leaves artifacts behind or reads state from a previous run, so the first build and the second build don’t behave the same way, and “clean and rebuild” doesn’t actually get you back to a clean start.

All three share one root cause: the build depends on something that isn’t captured anywhere the agent can see or reproduce. So the fix is the same shape for all three — capture the environment explicitly, pin it, and make a fresh, empty environment the normal case rather than the exception. The payoff of fixing all three is a build whose only inputs are your source and a written spec, and therefore a build an agent can stand up from scratch without you.

Enumerating the fixes, from weakest to what I’d actually ship

There’s a real choice in how you capture that environment, so let me lay out the actual options rather than pretend there’s only one, then tell you which I’d ship and why.

The weakest option is a README that documents the setup — “install version X, set variable Y.” It’s better than nothing, but it’s advisory: it relies on every machine, and every agent run, faithfully re-reading and re-applying it, which is exactly the discipline a drifting environment already proved you don’t have. I reject this as the primary mechanism because it captures the knowledge without enforcing it. The next option is a setup script — setup.sh that installs and configures everything. Better, because it’s executable rather than advisory, but it still runs against whatever base system it finds, so it pins your steps without pinning the ground they run on. Two machines with different base images can run the identical script and diverge. The option I actually ship is a pinned container image plus a CI job that builds inside it — a container, meaning a fully specified, versioned operating environment described in a file checked into the repo, so the compiler, tools, and system packages are fixed to exact versions and rebuilt identically anywhere. The reason this one wins isn’t that containers are fashionable. It’s that it’s the only option on the list that closes all three failure modes at once: the pinned image kills toolchain drift, the in-file environment spec kills works-on-my-machine, and a fresh container per run kills hidden host state.

Because the container is described in a file, it doubles as the exact thing an agent needs — a declarative, reproducible environment it can stand up itself. Here’s the shape of that file, and the one property that matters most about it:

# Pin the base image by exact version — never ":latest", which drifts.
FROM debian:12.8-slim

# Pin the toolchain to exact versions, so every build uses the same compiler.
RUN apt-get update && apt-get install -y --no-install-recommends \
      gcc-arm-none-eabi=15:13.2.rel1-2 \
      cmake=3.25.1-1 \
      ninja-build=1.11.1-1 \
    && rm -rf /var/lib/apt/lists/*

The property that matters is that everything the build needs is named and versioned in a file under version control — not installed by hand, not assumed present, not carried in someone’s shell history. That’s what makes it reproducible: the inputs are the source tree and this file, and nothing else.

The CI job then does one job that matters here: it proves, on every commit, that the build still comes up green from nothing but the repo. CI for embedded firmware is often sold as “catch bugs early,” and it does, but its more important role in an agent-ready setup is quieter: it’s the standing, automated proof that the reproducible build is still reproducible, so drift gets caught the day it’s introduced instead of the day an agent trips over it. The payoff is that “does this build from scratch?” stops being a question anyone has to remember to ask — the pipeline asks it for you, on every change, forever.

One more thing the build must not do: touch the source tree

There’s a specific failure mode inside “hidden host state” that deserves its own attention, because it’s the one that quietly breaks the recovery move you’ll rely on most. When a build works but writes generated files back into the source tree — a header next to the source, an object file in a source directory — then “delete the build output and start clean” no longer gets you a clean start, because some of the output is tangled up in your source. For an agent (or several agents, or an agent and you) sharing a tree, that clean-restart move is the whole recovery strategy, so it has to actually work. Making it work means the build is genuinely out-of-tree — nothing it produces ever lands inside the source — and that turns out to be harder to achieve, and easier to fake, than it sounds. I’ve given it its own treatment in Out-of-Tree Builds in Embedded C++, Done for Real, because getting it right is what makes “delete the build directory and rebuild” a recovery move you can trust rather than a hope. The payoff of keeping the build strictly out-of-tree is that recovery from a broken build state becomes a single, reliable command instead of a manual hunt through your own source.

A bit of work now, or a lot of work from now on

Step back and the whole thing is one trade, stated the same way at the start and the end because it’s the point. Making the build reproducible and agent-runnable is bounded work: a pinned container, an in-file environment spec, a CI job that builds from clean, and the discipline to keep the build out of the source tree. You scope it, you do it, it’s finished. Skipping it doesn’t avoid the work — it converts it into unbounded work, the same manual fix repeating on the agent’s schedule for as long as the codebase stays fragile. In my experience the up-front version costs a few days for a codebase that isn’t already there, and the recurring version costs those same few days over and over, in twenty-minute interruptions to whoever is most expensive to interrupt. That’s the actual decision at this stage, and “we’ll deal with it later” is not a way of avoiding it — it’s a way of choosing the expensive branch without admitting you chose.

Where this leaves you

The choice at the first step is smaller and more concrete than the “should we adopt AI agents at all” conversation most teams are having, and it’s more decisive. You either spend the bounded few days that make the build reproducible and agent-runnable, or you spend an unbounded amount of your best engineers’ attention being the thing the agent comes back to whenever the build breaks. Everything else an agent-ready codebase needs — compile-time checks, tests it can run itself, the full unsupervised loop — is built on top of this and inherits whatever reliability this step has or lacks. Start here, do it once, and the rest of the loop has solid ground to stand on. Skip it, and you’ve picked the recurring bill without meaning to.