Measurement and evidence

An evaluation is only as good as its weakest honest step. A confident conclusion drawn from a sloppy measurement is worse than no conclusion, because it travels further and is trusted more.

The temptation in field work is to reach a verdict quickly — the device is good, the device is bad — and to treat the measurements as decoration on a judgment already made. We work the other way around. The measurement comes first, the conditions are recorded with it, and the conclusion is only allowed to be as strong as what was actually observed. Often that means the honest finding is narrower and less satisfying than the one we expected to write. That is the point.

What makes an observation evidence

• It is repeatable. A result seen once might be the device; it might be the day. A result seen again, under the same conditions, is the beginning of a finding.
• It carries its conditions. A number without the circumstances that produced it cannot be checked, compared, or trusted. The conditions are part of the evidence, not context around it.
• It distinguishes the device from everything else. If the user, the environment, or chance could have produced the same result, the observation is about the situation, not the device — and should be reported that way.
• It survives someone trying to break it. We look for the conditions under which our own result would fail before we publish it. A finding we have attacked and not overturned is one we can stand behind.

Honest uncertainty

Field measurement is noisy, and pretending otherwise is the most common way evaluations go wrong. A result is worth more when it comes with a plain statement of how sure we are and why. "This held across every trial" and "this held once and we could not reproduce it" are both useful; collapsing them into a single confident sentence is not. Where the evidence runs out, we say so, rather than extending the line past the last real data point.

Writing it so it can be checked

The final discipline is to record a finding so that someone who trusts neither us nor the manufacturer could follow the same steps and see the same thing — or show us where we were wrong. That means stating the method, the conditions, the number of trials, and the result, in language a non-specialist can follow. A finding that can only be taken on faith is not evidence. It is an opinion wearing evidence's clothes.

None of this requires elaborate instruments. It requires measuring before concluding, recording the conditions alongside the result, reporting uncertainty plainly, and writing the whole thing down so it can be checked. Do that consistently and the work earns the only kind of authority worth having — the kind that survives being examined.

Editor's noteThis essay was developed with the assistance of AI writing tools, then reviewed, revised, and approved by Technology Research Group. The judgments it expresses are ours, as is responsibility for any error.