Why embedded tests flake and how you stop the flakes
Keywords: flaky firmware tests, deterministic embedded simulation, HIL CI/CD, race conditions, hardware in the loop.
Teams rerun failed tests
The time is 2:00 PM on a Friday. You push a three-line fix for an I2C edge case. You wait 45 minutes for the Hardware-in-the-Loop (HIL) pipeline to finish.
The result is a red checkmark. The error text is Error: Timeout waiting for sensor acknowledgment.
You check the logs. The logs do not show a fault in your I2C patch. You sigh. You click “Rebuild”. You walk away. The test passes 45 minutes later. This result is a flaky firmware test. A flaky firmware test is the largest morale problem in embedded engineering.
A software team can blame a noisy network for a flaky test. A software team can blame a faulty database state for a flaky test. An embedded team has many possible causes. One cause is a loose USB cable. One cause is a flash wear-level that wears. One cause is a slight variance in a power supply. The test script relies on sleep(100). On that day the script needs sleep(110). The host PC compiles other software in the background. That background work is the cause of the new delay.
The result is always the same. Developers lose trust in the continuous integration (CI) pipeline. Teams accept one response as normal practice. The response is “Oh, that test just fails sometimes, ignore it and rerun”. This practice lets critical bugs enter production.
Why firmware CI flakes
In embedded systems, flaky tests usually come from the difference between software logic and physical conditions.
The HIL limit
Tests on real HIL rigs are the accepted best method for final validation. These same tests are not suitable for CI. HIL setups are expensive. HIL setups are fragile. HIL setups are inherently non-deterministic. Your CI can run on a physical board in a server rack. Environmental instability can then change the test result. If another team puts a high load on the network, your test can fail. If a relay sticks, your test can fail. The cause is not your code.
Fast simulators change event time
Many teams use software simulators to avoid the HIL limit. QEMU is one simulator. Most standard simulators aim for speed. These simulators run instructions “as fast as possible”.
Your firmware can expect a specific sequence of DMA interrupts. The host CI server can suddenly start a heavy Docker build on another core. The event time then changes. The simulated hardware does not behave like the physical hardware. You trade hardware instability for host-load instability.
The vTaskDelay anti-pattern
You cannot predict event time in HIL tests. You cannot predict event time in standard simulators. Test engineers put hardcoded sleeps in many places in their test scripts. These sleeps include vTaskDelay and sleep(100). These sleeps are not a robust method.
Bit-accurate determinism
To stop flaky tests, you must separate simulated time from real-world time.
The LabWired engine is built for lockstep determinism. The engine is not built only to execute instructions quickly.
The simulation clock drives every part of the run. The CI runner can be a 64-core computer. The CI runner can be a slow laptop. The order of these events is mathematically identical on every run. The events are instruction execution, DMA transfers, and the time when each interrupt fires.
Deterministic headless runs
A standard LabWired CI test runs headless validation. A recent benchmark uses our uart-ok.yaml fixture. The benchmark output is below.
$ labwired test --script examples/ci/uart-ok.yaml
2026-02-23T20:32:23.113Z INFO labwired_core::system::builder: Loading system manifest: "ci-fixture-uart1.yaml"
2026-02-23T20:32:23.113Z INFO labwired_core::system::builder: Loading chip descriptor: "ci-fixture-cortex-m3-uart1.yaml"
2026-02-23T20:32:23.113Z INFO labwired_loader: ELF Entry Point: 0x9
OK
The test boots the firmware. The test maps the Cortex-M3 core. The test runs the target code. The test captures the UART OK string. The test exits. The full sequence is deterministic.
The engine does not lose speed to gain this precision. Our recent single-core throughput benchmarks reliably reach ~46.9 MIPS (Millions of Instructions Per Second).
[PERF] MIPS: 46.95 (Elapsed: 0.213s)
A strictly deterministic virtual hardware platform with this performance profile gives you these functions. Physical HIL cannot give you these functions.
- No
sleep(). The simulator synthesizes time. The simulator controls time. A test can advance the simulation clock by exactly 50 milliseconds. The test then knows the exact hardware state. - Time-Travel Debugging. If a test fails, you do not need to reproduce the failure. LabWired captures the execution trace. You can step backward in simulated time. You can see the exact race condition that caused the crash.
- Parallel runs. You cannot put 500 HIL rigs in a server closet. You can start 500 LabWired Docker containers. You can run your full validation suite in parallel. You can run that suite on every commit.
Do not rerun the pipeline
Flaky tests are not a physical law. Flaky tests come from the wrong tools for CI. HIL is for final sign-off. Modern development needs agility. You need an environment that always gives the same output for the same input.
If you do not want to manage flaky CI by hand, change the test platform. Use a platform that reports the true result.
Do you want to stop the flakes? Read the Getting Started Guide. See how LabWired compares with QEMU in LabWired vs QEMU.
