A few years ago, we started work on the firmware for a company that builds powered respiratory-protection gear. These are the battery-powered blower units that push filtered air to someone working in a hazardous environment. The discrepancy their team had been chasing for days just wouldn’t go away, so we stepped in to assist. The board was reporting one current value during calibration and a higher value during actual operation. This made it look like a compensation bug, but it wasn’t. The real problem was that nobody on the team could be sure which code was actually on the chip they were looking at.
The symptom that looked like a sensor problem
This project was a classic case of one of the most common traps in embedded firmware debugging. Two numbers don’t agree when they should, and the engineering team blames the math. The firmware here had two modes. In calibration mode, the board presents the value programmed into the unit’s per-unit setup. In normal run mode, it displays the live, compensated current active in the system. When set up correctly and at the same operating point, the two modes should read almost the same current. Instead, the team found that run mode came in consistently higher than calibration mode by a small, but meaningful, margin. On a low-flow alarm, that margin is the difference between a right threshold and a wrong threshold.
The engineer on the client’s side had done the responsible thing before we jumped in. He swapped the blowers and the boards and then kept measuring. But unfortunately, the gap between the two numbers didn’t change. Consistent behavior across hardware is usually a good clue to the real underlying problem. Here, that clue pointed us away from a bad sensor or a flaky board. Instead, it pointed at the firmware.
Why the compensation math is the wrong thing to debug first
When two numbers in a project disagree, many engineer’s first instinct is to open the formulas to check for errors there. With all the inputs and outputs available, along with offsets and orders of operation, it is easy to place your mistrust in the wrong part of the project. Spending hours poring over the same arithmetic can be frustrating, and sometimes completely pointless.
The trouble with that first instinct so many have, is that it makes assumptions about the code. When you assume the code on the screen is the code doing the computing, it seems obvious to blame the formulas. But the problem was, the compensation math had always been correct when the team tested it in earlier builds. It felt natural to start there but re-deriving that code was the same as chasing a suspect with an alibi. It was a convincing distraction, but the real question was this. Is the binary that we are currently testing actually built using this source?
The real problem was an uncertain source of truth
What we discovered on this project were two teams working the same firmware, but with no shared version control between them. All the source files were sent via email. One team built the hex images, but the other team was in charge of flashing those same hex images. Over the course of several days, the source and the binary began to drift apart. This drift happened quietly, and slowly enough that it was hard to notice from inside the problem.
Unfortunately, the drift was not just cosmetic. It was a serious problem. One version smoothed the current with a moving average over a large sample window. Another used an exponential filter set to a handful of samples. Those two filters respond to a changing signal very differently, which means very different reading. When loading the different builds, the blower would display two different numbers, because the version of the code open on screen was not necessarily the version on the chip. This means that no matter what the engineers were reading and testing, they couldn’t be sure if it was accurately telling them about the code they thought it was.
Layered on top of that, the arithmetic in calibration and run mode was also different, which again created different values. The calibration logic applied a fixed offset and the compensation in one path, but not consistently in the other. This resulted in even perfectly matched builds showing disagreements between the two modes, because they were not actually computing the same quantity the same way.
What actually fixed it
The fix we settled on with this project was simple bookkeeping, not engineering brilliance. First, we identify the source and determine which source produced each hex, then we require everyone to use a single known build for the rest of the project. Second, we align the filter configuration so both modes ran the same smoothing with the same parameters. Third, we make calibration mode and run mode derive their number through one shared path, offset and compensation included. These three steps eliminated the possibility of a second copy of code that could begin to drift away from the true copy.
Once we had those three things settled, the two modes finally agreed, and the alarm threshold was able to function properly and stop crying wolf. Nothing about this project was overly complicated engineering. It was simply a discrepancy between the methods used to solve the problem. Without an outside perspective, it was easy to diagnose this as a math problem when it was really a “which build is this” problem.
How to keep a phantom bug from eating a week
The specific offset here does not matter to anyone else’ project. However, the pattern it reveals does matter across the board. A numeric discrepancy that survives every rebuild and every board swap is often actually a build or configuration mismatch that is wearing a math costume. The faster you suspect that, the less time you lose chasing a false lead. Anything that shortens the embedded debugging loop helps, but nothing shortens it like debugging the right thing in the first place.
A few habits can help make this bug a rare occurrence for your team. Make sure to prove the provenance of the binary before you try to debug the formula. you want to be certain that the hex on the chip came from the source you think it did, and ideally from a clean, reproducible build. When you put everyone working with that code on the same version control system, it eliminates any potential for drift, even if the project seems too short to need that safeguard. It also helps to make sure modes are calling the same function rather than two separate functions that are only identical until someone edits one or the other.
If your team has been circling a discrepancy that moves every time you re-flash, the answer may be upstream of the code you are staring at. Tell us what you are chasing and get in touch, and we will help you find the source of truth before you spend another week auditing math that was never wrong.
FAQ
How do you debug a value that differs between two firmware modes?
It helps to start by confirming both modes are computing everything the quantity the same way, ideally through one shared function. Then actually verify that the binary you are testing was built from the source you are reading. A mismatch between builds is exactly what produces these stable, repeatable gap errors. When you have established the provenance of the code, then you can start checking the math if the error hasn’t resolved itself already.
Why do calibration mode and run mode report different sensor values?
This is usually the result of differences in how each mode processes the reading. If a filter or offset is applied to one path but not the other, or you are running a different version of the firmware than you think you are, mismatches occur. Environmental compensation applied inconsistently is another common culprit. The easy fix is to make sure that both modes share one computation path, which eliminates most of these problems.
Is a sensor discrepancy a compensation bug or a build mismatch?
While either can cause it, if a discrepancy stays consistent across different boards and blowers, and survives rebuilds, that points more toward a build or configuration mismatch than a math error. Compensation bugs tend to move with temperature or pressure changes. So, before you assume the formula is wrong, take a quick second to confirm the hex on the chip matches your source.
How do you make sure the hex on the chip matches your source code?
If you always build from a clean checkout, keep the binary and its source under the same version control commit, and avoid hand-editing or emailing files between people mid-debug, you should avoid most of the drift that can happen in projects. Where possible, make sure to embed a build identifier or version string the firmware can report, so the running unit can tell you which build it is. That single readout will eliminate most of your “which version is this” confusion.
Why does shared version control matter for embedded firmware support?
When two teams pass source files and hex images back and forth without shared version control, the source and the binary drift apart. This turns the debugging process into nothing but guesswork. Keeping one shared repository across all teams helps maintain a single source of truth. This in turn makes builds reproducible and lets an outside engineer support the work without them having to reverse-engineering which file they are actually using. This discipline will help keep your team from wasting multiple days chasing a phantom bug on what should’ve been a short project.
