Reproducible Embedded Builds, or Becoming Your Agent's Synchronization Point

If you’re about to put an AI coding agent to work on your embedded codebase, there’s a decision waiting for you at the very first step, and most teams make it by accident. Whatever the agent is actually doing (implementing a feature, fixing a bug, running a test, checking whether a change grew the binary by ten kilobytes), the first step is always the same: build the code. So the decision that actually gets made by accident is whether the agent can take that first step on its own, or whether it has to stop and hand the whole task back to you every time the build breaks, and a build that only works on one engineer’s machine breaks constantly. Get this wrong and the cost is not a one-time annoyance; it’s a recurring tax on your most expensive people, paid every time the agent hits the same wall, on the agent’s schedule rather than yours. And it isn’t paid by your engineers alone: every minute the agent spends stalled on a build only you can fix is a minute it sits idle, producing nothing, which is its own cost on top of yours. Get it right and the same work is done once, up front, bounded, and never paid again. Stated plainly: either you do the bounded work now that lets the agent compile the code unsupervised, or you become its synchronization point — the human it has to wait on, every time, before it can even start the task you gave it. There is no third branch where nobody does the work. “Do nothing” just picks the second option by default, and it’s the more expensive one.

This is the foundation piece of a larger idea I’ve written about separately: that embedded, built the right way, is a better place for an AI coding agent than web development is (see Why Embedded Is a Better Place for AI Agents Than Web Development). That whole argument rests on one thing being true first: the agent has to be able to build your code. This post is about making that true.

What is a reproducible, agent-ready build?

A reproducible build is one that produces the same working result from the same source on any machine set up to the documented spec, without depending on state that happens to live on one engineer’s laptop. An agent-ready feedback loop (the build, test, and verification infrastructure a codebase needs so an AI coding agent can compile, test, and iterate without a human re-checking every step by hand) starts with exactly that reproducible build, because nothing downstream can run until the code compiles. Put the two together and “agent-ready build” means something concrete and testable: a fresh environment, provisioned from a written spec, compiles your firmware to a correct binary on the first try, with no human in the loop. If that isn’t true today, it’s the first thing to fix, and everything else in the agent-ready loop waits on it.

Compiling is where the trouble shows up first

Compiling is the first thing an agent has to do to make any progress at all, which makes it the first place a shaky foundation becomes visible. No test runs, no refactor lands, no review means anything before the code builds, so a build that only works sometimes doesn’t stay contained to the build step. It poisons every step downstream of it. An agent that can’t trust the build can’t trust the compile-time checks (correctness checks the compiler evaluates before any test runs), can’t trust the test suite, can’t trust anything, because all of it is standing on a foundation that shifts between runs. This is also why the failure is easy to underestimate. When you build by hand, your machine is already set up the way it needs to be, so the build succeeds far more often than it fails, and the rare failure barely registers against all the times it just worked. The agent gets no such average: it hits the same fragile step on every task, on every run, so what reads to you as an occasional rough edge is, to the agent, the normal case, and each time it hits it, the work stops until someone intervenes. The payoff of getting this right is that every later step (tests, mocks, the full agent loop) inherits a foundation it can rely on instead of one it has to keep working around.

What being your agent’s synchronization point actually costs

Here’s the failure mode in practice, because it’s worth seeing concretely before we talk about fixing it. The agent picks up a task, makes a first pass at the code, and tries to build it. That’s the first move in the loop it’ll run over and over until the task is actually done: build, test, look at the result, adjust, build again. This time the build fails for a reason that has nothing to do with the task itself: a toolchain version mismatch, a missing system package, an environment variable only your shell has. The agent can’t fix that, but it doesn’t just notice and stop: it tries. It spends time and tokens guessing at the missing package, patching a path, working around the version mismatch, often getting a different result on each attempt because nothing about the environment is pinned down. Only once it runs out of things to try does it surface the error and wait. You context-switch off whatever you were doing, fix the environment by hand, and hand it back. Twenty minutes later, a different agent run (or that same agent, restarted in a fresh sandbox that doesn’t carry your by-hand fix) hits the same wall, because nothing about the underlying build itself changed. You fix it again.

Notice what’s actually being spent here, and notice that it’s spent twice. On your side, it isn’t the twenty minutes; it’s the interruption, the context-switch, and the fact that this repeats without bound for as long as the build stays fragile. On the agent’s side, it’s every minute of the trying-and-failing before it gave up, plus every minute of idle waiting after, time in which it could be working and instead is doing nothing at all. That’s what a synchronization point is, and if you’ve built concurrent systems you already know the shape of the problem: two independent things (you and the agent) can only make progress by waiting on each other, so the whole system runs at the speed of whichever one is slower to respond, which in this setup is always you. That’s the real shape of the trade. The work of making the build reproducible is finite: you do it once, deliberately, and it’s done. The work of not making it reproducible is unbounded: the same wasted attempts, the same manual fix, the same interruption, recurring every time an agent (or a new hire, or you on a fresh laptop) hits the same wall; every one of those times, both sides pay. You will resent it when it happens this way, and you’d be right to, but the choice was yours before it ever got to that point. The one-sentence version, the one worth quoting to whoever signs off the time: making the build agent-ready is bounded, one-time work; not doing it is unbounded, recurring work, billed to both of you at once — and the bill never stops arriving.

The three ways a build betrays an agent

Fragile builds fail in more or less three ways, and naming them makes the fix obvious rather than mysterious.

The first is toolchain drift: the compiler, the linker, the build tool, or a library is a different version than the one the code was written against, so a build that worked last month fails now, or works on your machine and fails on the continuous-integration (CI) runner (the clean, automated server that rebuilds every change from scratch). The second is the works-on-my-machine environment, where the build quietly depends on something present only on one engineer’s laptop: an installed SDK (software development kit), a PATH entry, an environment variable, a file outside the repo. Nothing records that dependency, so no fresh machine reproduces it. The third is hidden host state: the build leaves artifacts behind or reads state from a previous run, so the first build and the second build don’t behave the same way, and “clean and rebuild” doesn’t actually get you back to a clean start.

All three share one root cause: the build depends on something that isn’t captured anywhere the agent can see or reproduce. So the fix is the same shape for all three: capture the environment explicitly, pin it, and make a fresh, empty environment the normal case rather than the exception. The payoff of fixing all three is a build whose only inputs are your source and a written spec, and therefore a build an agent can stand up from scratch without you.

Enumerating the fixes, from weakest to what I’d actually ship

There’s a real choice in how you capture that environment, so let me lay out the actual options rather than pretend there’s only one, then tell you which I’d ship and why.

The weakest option is a README that documents the setup: “install version X, set variable Y.” It’s better than nothing, but it fails in two ways that compound each other. First, nothing enforces it: a README is advisory, and relies on every machine, and every agent run, faithfully re-reading and re-applying it, which is exactly the discipline a drifting environment already proved you don’t have. Second, even the knowledge it captures doesn’t stay put: someone bumps a compiler version, adds a dependency, or works around a quirk on their own machine, and the README is now describing a setup nobody is actually running. This is the same problem engineers already live with in code comments. As Ron Jeffries put it, code never lies, comments do, and a README is just a comment that lives one directory up from the code it describes. I reject it as the primary mechanism because it neither enforces the knowledge it captures nor guarantees that knowledge stays true. The next option is a setup script: setup.sh that installs and configures everything. Better, because it’s executable rather than advisory, so it can’t drift silently the way a paragraph of prose can: it either runs and works, or it visibly fails. But it still runs against whatever base system it finds, so it pins your steps without pinning the ground they run on. Two machines with different base images can run the identical script and diverge. The option I actually ship is a pinned container image plus a CI job that builds inside it: a container, meaning a fully specified, versioned operating environment described in a file checked into the repo, so the compiler, tools, and system packages are fixed to exact versions and rebuilt identically anywhere. The reason this one wins isn’t that containers are fashionable. It’s that it’s the only option on the list that closes all three failure modes at once: the pinned image kills toolchain drift, the in-file environment spec kills works-on-my-machine, and a fresh container per run kills hidden host state.

There’s a second reason the container wins that has nothing to do with drift and everything to do with time: it’s checked into the repo, so it’s versioned alongside the code it builds. Update the README and you’re relying on every engineer to notice, re-read it, and change how they work, and you cannot assume perfect compliance across a team, let alone from an agent that only reads what it’s told to read. Update the Dockerfile and there is no separate compliance step: whichever commit you check out is built by whichever Dockerfile shipped with it, automatically, for everyone. That difference matters most exactly when you don’t want it to: you make a substantial change to the toolchain, and weeks later you need to cut a hotfix against an old release. Checking out that old commit gets you the old README just fine; git doesn’t lose files. The problem was never retrieval.

The problem is what following it actually requires. Someone has to notice the old README applies, then manually reconfigure their own machine to match it: uninstall the current compiler, install the old one back, and hope nothing else on that machine needed the newer version in the meantime. Then what? Revert that by hand once the hotfix ships, and hope the first attempt worked. If it didn’t, do the whole dance again. Or leave the old compiler installed for a few days “just in case,” which means that machine can’t be used for anything on the current version until someone remembers to switch it back.

This is also, incidentally, one of the real reasons teams stay on old compilers and old language standards years past the point where upgrading would help them: nobody trusts themselves to manage that back-and-forth reliably, so the safer-feeling choice is to never start the switch at all. That’s not a one-time cost either. It’s the price of every improvement the newer toolchain would have given you, paid indefinitely, for as long as the fear of the manual dance outlasts the will to fix it. Add it to the bill.

Checking out that old commit with a Dockerfile skips all of this: the build runs inside a fresh container built from that old, pinned spec, so nothing on the actual machine changes and there’s nothing to undo afterward. Pretending the README will get followed at that point isn’t optimism; it’s wishful thinking that costs you exactly when the stakes are highest.

Because the container is described in a file, it doubles as the exact thing an agent needs: a declarative, reproducible environment it can stand up itself. Here’s the shape of that file, and the one property that matters most about it:

# Pin the base image by exact version — never ":latest", which drifts.
FROM debian:12.8-slim

# Pin the toolchain to exact versions, so every build uses the same compiler.
RUN apt-get update && apt-get install -y --no-install-recommends \
      gcc-arm-none-eabi=15:13.2.rel1-2 \
      cmake=3.25.1-1 \
      ninja-build=1.11.1-1 \
    && rm -rf /var/lib/apt/lists/*

The property that matters is that everything the build needs is named and versioned in a file under version control, not installed by hand, not assumed present, not carried in someone’s shell history. That’s what makes it reproducible: the inputs are the source tree and this file, and nothing else.

The CI job then does one job that matters here: it proves, on every commit, that the build still comes up green from nothing but the repo. CI for embedded firmware is often sold as “catch bugs early,” and it does, but its more important role in an agent-ready setup is quieter: it’s the standing, automated proof that the reproducible build is still reproducible, so drift gets caught the day it’s introduced instead of the day an agent trips over it. The payoff is that “does this build from scratch?” stops being a question anyone has to remember to ask; the pipeline asks it for you, on every change, forever.

One more thing the build must not do: touch the source tree

There’s a specific failure mode inside “hidden host state” that deserves its own attention, because it’s the one that quietly breaks the recovery move you’ll rely on most. When a build works but writes generated files back into the source tree (a header next to the source, an object file in a source directory), then “delete the build output and start clean” no longer gets you a clean start, because some of the output is tangled up in your source. For an agent (or several agents, or an agent and you) sharing a tree, that clean-restart move is the whole recovery strategy, so it has to actually work. Making it work means the build is genuinely out-of-tree (nothing it produces ever lands inside the source), and that turns out to be harder to achieve, and easier to fake, than it sounds. Getting it right is what makes “delete the build directory and rebuild” a recovery move you can trust rather than a hope. The payoff of keeping the build strictly out-of-tree is that recovery from a broken build state becomes a single, reliable command instead of a manual hunt through your own source.

A bit of work now, or a lot of work from now on

Step back and the whole thing is one trade, stated the same way at the start and the end because it’s the point. Making the build reproducible and agent-runnable is bounded work: a pinned container, an in-file environment spec, a CI job that builds from clean, and the discipline to keep the build out of the source tree. You scope it, you do it, it’s finished. Skipping it doesn’t avoid the work; it converts it into unbounded work, the same manual fix repeating on the agent’s schedule for as long as the codebase stays fragile. In my experience, turning an already-sane build into a reproducible, agent-ready one costs a few days. How long it takes to get to “already sane” first depends entirely on how bad your current build is, and for some codebases that’s the much bigger number. Two things are worth knowing about that part, though. It goes faster than you’d expect if you approach it as building toward reproducibility from the start, because every step you take is also a step toward a build an agent can eventually work on, so agentic coding starts paying for itself before you’ve even finished. And that work has value on its own, independent of any agent: if your build is badly fragile today, your engineers are already burning enormous amounts of time fighting it by hand, agent or no agent, and fixing it creates real value regardless of whether you ever point an AI at the result. Either way, the recurring alternative costs those same days over and over, in interruptions that are expensive on their own and that also put a hard cap on how fast your agent can ever go, because it can only move as fast as the human it’s waiting on. That’s the actual decision at this stage, and “we’ll deal with it later” is not a way of avoiding it; it’s a way of choosing the expensive branch without admitting you chose.

Where this leaves you

The choice at the first step is smaller and more concrete than the “should we adopt AI agents at all” conversation most teams are having, and it’s more decisive. You either spend the bounded, if variable, stretch of time that makes the build reproducible and agent-runnable, or you spend an unbounded amount of your best engineers’ attention being the thing the agent has to wait on whenever the build breaks. Everything else an agent-ready codebase needs (compile-time checks, tests it can run itself, the full unsupervised loop) is built on top of this and inherits whatever reliability this step has or lacks. Start here, do it once, and the rest of the loop has solid ground to stand on. Skip it, and you’ve picked the recurring bill without meaning to.