LabWired finds a CAN bug that occurs only on real hardware
Category: Whitepaper
A user of udslib reported a UDS bug. udslib is my small ISO-14229 stack. The bug did not reproduce in any test. The bug occurred only on a particular STM32F103 setup.
Single-frame UDS requests succeed. Multi-frame UDS requests do not succeed. The ECU sends no response. The PCAN trace from the user showed the sequence. The tester sent the FirstFrame. The ECU answered with a FlowControl. The tester sent the ConsecutiveFrame. Then no frame came back.
Caption: The bug as reported. 111h is the tester. 222h is the ECU. The FirstFrame is 10 0B 27 01 5A 11 22 33. The FlowControl from the ECU is 30 08 00…. The ConsecutiveFrame is 21 44 55 66 77 88…. Then no response comes.
The user had a theory. The theory was a length-calculation bug. That theory is a reasonable guess for multi-frame length math. The cause was different. The different cause is the part of interest. The bug does not reproduce in a normal test. The bug occurs only on a particular kind of real-hardware setup.
First, the bug does not reproduce
I did not touch the STM32.
I took the exact bytes from the user trace.
I replayed those bytes directly through the current udslib.
The bytes went into the real dispatch layer.
The replay worked.
The ECU reassembled the 11 bytes.
The ECU sent an answer.
I also did not have an STM32 on my bench.
I could not reproduce the bug on the bench.
I used LabWired for that reproduction.
The bug in LabWired
I built the real udslib firmware for the bxCAN controller on the F103. I ran that firmware in the LabWired simulator. I built a second copy. That copy includes the user code bug. I put the two copies side by side as two labs. Open the labs in the LabWired playground. Press Run.
Caption: LabWired steps through the firmware with lockstep instruction execution.
The actual bug: two different clocks
The root cause was an N_Cr timer clock-source mismatch.
The mismatch is between systick hardware ticks and a millisecond timer abstraction.
The timer abstraction is in the protocol stack.
The mismatch occurs under real peripheral timing.
Standard host-unit tests mock time with fake_time++.
The two clocks move in sync.
The bug never fires in those tests.
On real silicon, the hardware timer registers execute asynchronously.
The interrupt handlers execute asynchronously.
The bit-accurate simulation in LabWired executes the timer registers and the interrupt handlers asynchronously.
This execution exposes the race condition.
The fix on the silicon
I identified the cause in LabWired.
- I unified the tick source for ISO-TP timer management to one atomic tick provider.
- I committed the fix to
udslib(Issue #29). - The user tested again on the physical STM32F103 setup. Multi-frame SecurityAccess requests passed instantly. Multi-frame transfer requests passed instantly.
We reproduced the hardware-only bug in LabWired. That reproduction let us fix a complex CAN timing bug. We did not need physical probes on a physical board. We did not need logic analyzers on a physical board.
Author
Andrii Shylenko, Founder, LabWired.
- Email: contact@labwired.com
- GitHub: https://github.com/w1ne
- Calendar: Book a call
