Compile-Time Testing in C++: Prove It, Don't Just Test It

Last updated:

Compile-time testing in C++ means writing a unit test as a constexpr or consteval function and checking it with static_assert, so the test runs while the compiler builds the program, not afterward. Applied to a parser, it turns an entire class of defect, an out-of-bounds read or a narrowing-conversion undefined behavior, into something the compiler refuses to build at all, instead of something a sanitizer might catch on some input you happened to test. This post proves that against a real spec, not a toy example: a from-scratch CBOR (RFC 8949) decoder whose test suite runs inside the compiler itself, checked against the spec’s own 104 official test vectors and across every compiler it actually ships on.

The decoder itself is the supporting cast, not the point of this post. For the full API, the no-malloc variant, the policy choices, see the GitHub repo and its project page. What belongs here is the part that generalizes: how to write a parser whose test suite the compiler itself runs as part of compiling your program, and what that buys you that an ordinary test suite can’t.

I’ve watched more companies than I can count on both hands solve “how do we serialize our data” by inventing their own binary format on the spot, and every one of them painted themselves into the same corner, fast. Most were writing C++ that looked a lot like C, and the parsing code eventually grew a line like reinterpret_cast<MyStruct*>(binaryData): a pointer to raw bytes pulled off the wire, told to the compiler as if it already held a MyStruct. The compiler believes it. It’s also undefined behavior from the moment it compiles, and nothing stops it from appearing to work for years, right up until the sender’s padding or byte order stops matching what the receiver silently assumed. The bug that surfaces then doesn’t point back at this line; it points at whatever the corrupted bytes happened to touch next, months later, in a field incident with no repro on a bench. For a team funding embedded firmware work, that’s the expensive failure mode: a defect that only shows up on a real device, in the field, long after the code that caused it shipped, and exactly what compile-time testing is built to close.

Why Use a Binary Parser to Prove Compile-Time Testing?

CBOR, short for Concise Binary Object Representation, is a binary serialization format standardized as IETF RFC 8949. It encodes the same data model JSON does, numbers, strings, arrays, maps, into a compact byte stream that carries its own type and length information inline, so a decoder needs nothing but the bytes in front of it: no schema file, no out-of-band agreement about field order. That self-describing property is also exactly what makes it a brutal test of a parser’s correctness. Every value in the stream announces its own length, and that length comes from the sender, not from the receiver’s own assumptions, which means the decoder has to treat every length field as a number an attacker could have chosen on purpose. A binary parser over untrusted input is, structurally, the same shape as the reinterpret_cast disaster from the opening of this post: bytes in, trusted too far, out. If a technique can prove a parser like this free of out-of-bounds reads, it can prove the same thing about the register-mapped struct or the wire protocol your own firmware already has.

I didn’t reach for an existing CBOR library (QCBOR, tinycbor, zcbor are all mature, widely used C implementations) because of a quality judgment against them. It’s narrower than that: none of them were written to run as a constant expression. They’re built around pointer arithmetic over a mutable buffer, the ordinary shape of code meant to be compiled and executed, not evaluated by the compiler itself. Getting the compiler to prove a decode step free of undefined behavior, for a given input, at build time, means writing a decoder specifically shaped for that, and that rules out adopting one off the shelf no matter how well it’s engineered. Building this one from scratch, against the spec’s own 104 official test vectors from RFC 8949 Appendix A, is what gives the rest of this post a real target to point the technique at instead of an illustrative snippet.

What is compile-time testing?

Compile-time testing in C++ means writing a unit test as a constexpr function, or more strictly consteval (which forbids the function from ever running outside a compile-time context), and checking it with static_assert. The test runs while the compiler is still building the program, not as a separate step afterward. If the assertion fails, the compiler reports it as a build error, with the actual and expected values sitting right there in the diagnostic, and the defect it caught never makes it into a shippable binary. There’s nothing to link, flash, or remember to run afterward: if the firmware compiles, those checks passed, and they cost the target nothing, because none of the test code exists once compilation ends. This is also the technique behind the site’s broader compile-time testing coverage, applied here against a real spec instead of an illustrative snippet. The library doing the work in the examples below is compile-time-unit-testing, a single-header C++20 library I maintain; the technique doesn’t require that specific library, but it’s what makes the diagnostics in this post readable rather than a raw static_assert failure with no context.

That’s the mechanics. What makes it worth building a whole decoder around is what C++20 actually lets a constant expression do: std::span’s bounds checks, narrowing conversions, std::bit_cast, all of it is ill-formed to misuse inside a constant expression, full stop, the same way dividing by zero is. Feed a decoder real bytes at compile time and every one of those guarantees becomes something the compiler enforces on the exact input you handed it, not something you hope a sanitizer happens to exercise later. A consteval test only proves what you wrote down, though; it still needs a runtime counterpart for the inputs nobody thought to write, which is exactly the gap sanitizer-backed runtime testing closes, covered at more length in Sanitizers in Embedded C++ Belong in Your Host-Side Tests.

How Do You Prove a C++ Decode Function at Compile Time?

The simplest thing a CBOR decoder has to do is read a small unsigned integer, so that’s where the contract starts. CBOR packs the major type (what kind of value this is) and, for small values, the value itself into a single header byte: the top three bits say “this is an unsigned integer,” and if the bottom five bits are 0 through 23, they are the value, no further bytes needed. The postcondition for decoding that byte has three parts, not one: given a header in that range, return the exact value and the exact number of bytes consumed; given any other major type, refuse; given a marker this minimal version doesn’t handle yet, also refuse.

struct DecodedUint {
  std::uint64_t value;
  std::size_t bytes_consumed;
};

constexpr std::optional<DecodedUint> decode_uint(std::span<const std::uint8_t> bytes) {
  if (bytes.empty()) {
    return std::nullopt;
  }
  const std::uint8_t header = bytes[0];
  if ((header >> 5) != 0 || (header & 0x1f) > 23) {
    return std::nullopt;
  }
  return DecodedUint{.value = header, .bytes_consumed = 1};
}

Each of those three contract clauses gets its own test, spanning the low, middle, and high end of whatever range that clause is actually about:

consteval bool test_decode_uint_accepts_direct_values() {
  using namespace CompileTimeUnitTesting;
  constexpr DecodedUint sentinel{.value = 999, .bytes_consumed = 999};
  constexpr std::uint8_t zero[] = {0x00};
  constexpr std::uint8_t mid[] = {0x0b};
  constexpr std::uint8_t max[] = {0x17};

  expect_eq<decode_uint(zero).value_or(sentinel).value, std::uint64_t{0}>();
  expect_eq<decode_uint(mid).value_or(sentinel).value, std::uint64_t{11}>();
  expect_eq<decode_uint(max).value_or(sentinel).value, std::uint64_t{23}>();
  return true;
}
static_assert(test_decode_uint_accepts_direct_values());

Now break it on purpose, the same discipline I’d apply to any test suite: if a mutation to the real code doesn’t make a test fail, that test wasn’t checking anything. Change > 23 to >= 23, a subtle mutation that reads as a plausible rounding choice in a diff, and here’s the real compiler output against the boundary value it actually broke, 23 itself:

error: 'CompileTimeUnitTesting::detail::diagnostic(...) [with auto ...Vs = {false, true}]' called in a constant expression

That {false, true} is decode_uint(max).has_value() coming back false when the contract demanded true: the actual violated clause, not a downstream symptom of it. Put > 23 back, and the test compiles clean again. That’s the whole loop:

  1. Write the contract.
  2. Write the test.
  3. Break the contract on purpose.
  4. Confirm the test notices.

Scale that loop across every CBOR major type, text and byte strings, the three IEEE-754 float widths, arrays, maps, tagged values, chunked indefinite-length strings, and you get the real decoder: every one of the spec’s 104 official test vectors decoded, checked, and re-encoded canonically against the official bytes, inside consteval. I’m not walking through each of those here; the repo’s test suite is the place to see the rest, and it’s the same loop every time.

One real spec decision is worth naming before moving on, because it’s the kind of ambiguity a from-scratch decoder has to actually make, not inherit by accident from whatever a library author picked years ago. RFC 8949 section 5.3.1 explicitly leaves UTF-8 validation on text strings up to the decoder: “a decoder might or might not want to verify that the sequence of bytes in a UTF-8 string is actually valid UTF-8 and react appropriately.” That’s a policy, in the Alexandrescu sense: a type supplied as a template parameter, swapped in at compile time, with its own shape checked by a static_assert rather than hoped for. The default policy here skips the check, which the spec explicitly permits, and the project’s own README names the tradeoff rather than hiding it. The point isn’t the specific choice; it’s that the choice got made on purpose, with a compiler-enforced contract on its own shape, instead of being whatever the first if statement happened to do.

Why Test Compile-Time Code Across Every Compiler You Ship On?

Here’s the part of this project that turned out to matter more than any single decode function, and it’s the part most compile-time-testing writeups skip: a consteval test only proves what you claim for the compiler that ran it. This decoder is tested against fifteen versions of GCC and Clang, WebAssembly via Emscripten, and three embedded cross-compilers (an ARM Cortex-M toolchain, an Xtensa ESP32 toolchain, and a RISC-V toolchain), each one built and tested at both C++20 and C++23, because a library claiming a C++20 floor has to prove that against a real compiler, not just request a newer standard and let it silently cap down. Running that matrix on this project’s most recent change, migrating every refusal path from a plain std::optional to a richer error type carrying a reason and a byte offset, caught three regressions before any of them reached a shipped build.

Two of the three weren’t bugs in my code at all. std::isnan isn’t guaranteed to be usable inside a constant expression before C++23, and several current releases of Clang’s standard library don’t mark it that way even at C++23 — which only showed up once the matrix built this project at C++20 against those compilers instead of assuming a newer standard would paper over it. The fix was mechanical, a comparison-operator-only NaN check instead of the library function, but finding it required running the exact standard this project promises, on the exact compilers people use it with, not just the one on my own machine. The third regression was stranger: on one embedded ARM toolchain, moving a short string into a std::variant’s internal storage during constant evaluation tripped a failure in that toolchain’s own standard library, the compiler’s constexpr evaluator losing track of which internal representation the string was using. I didn’t take that on faith either. I reproduced the failure in a throwaway container against the cross-compiler itself, with a minimal case stripped down to a dozen lines, before trusting that the fix (restructuring the constructor to avoid the pattern that triggered it) worked — the same way I’d verify a fix against a field incident rather than assume the patch was right. The full write-up of all three, with the repro, is in Constexpr Testing Finds Compiler and Standard Library Bugs Without a Board.

That’s the honest shape of what “prove it, don’t just test it” costs once you take it seriously: the compiler evaluating your test is itself a piece of software, and it can be wrong. A single-toolchain CI configuration would have shipped straight through all three of these, silently — the one compiler it happened to use just didn’t have that particular gap. The payoff for a team funding this kind of work isn’t “zero bugs,” which no technique delivers; it’s catching the toolchain-dependent landmine that an ordinary test suite, run on one machine, structurally cannot see, before it ships to a customer running a different compiler than the one on your desk.

Where this goes next

A CBOR decoder proven this way is a complete, useful thing on its own, and also a deliberate first step. The next post in this series puts the same discipline under real pressure: parsing an actual IETF SUIT manifest, the format firmware OTA (over-the-air) updates use to describe what’s being installed and where, which is CBOR with a lot riding on every byte being exactly what it claims to be. That post isn’t written yet. This one is the foundation it needs: a decoder whose test suite the compiler itself runs, proven across every toolchain that will actually build the firmware consuming it, not just the one that happened to be open when the code was written.

Frequently asked questions

What is compile-time testing in C++?

Compile-time testing means writing a unit test as a constexpr or consteval function and asserting it with static_assert, so the check runs during compilation rather than as a separate test step afterward. A failing assertion is a compiler error, with the actual and expected values in the diagnostic, and none of the test code ends up in the compiled binary.

Can a compile-time test prove a decoder is free of undefined behavior, or only catch it when it happens?

For the exact inputs the test hands it, it proves it, not just catches it: an out-of-bounds read or an invalid narrowing conversion inside a consteval function is ill-formed, so the compiler refuses to build the program at all rather than running the bad code and hoping a sanitizer notices. That guarantee is total for a given input and the specific decode path it exercises; it says nothing about an input nobody wrote a test for.

Does compile-time testing replace fuzzing or runtime tests for a parser?

No. A consteval test only proves what you wrote down; fuzzing and runtime testing, ideally under sanitizers, exist specifically to find the inputs nobody thought to write down. A parser handling untrusted bytes off the wire needs both: compile-time proof for the contract you can state, and fuzzing for the one you can't yet.

Can the compiler's own constexpr evaluator have bugs that affect a compile-time test?

Yes, and this project hit two real examples in the same release: std::isnan not being usable inside a constant expression on several current Clang standard library releases even at C++23, and a genuine constexpr-evaluation bug in one embedded ARM toolchain's standard library when moving a short string into a std::variant. Both were confirmed with a minimal reproduction against the real compiler before being fixed, and both would have shipped silently under a single-toolchain CI configuration.