I Reviewed for Five Conferences This Year. The Same Paper Keeps Arriving.
This week I accepted an invitation to join the program committee of IEEE BigData 2026, in Phoenix this December. That makes six venues in a year: AIES, SMC, CAI, ECCE, SciPy, and now BigData.
One boundary before anything else. Nothing here describes a submission I reviewed. Review is confidential and it stays that way. Every number below comes from my own work, which I published, and then had to correct in public two days later.
That is the honest reason I can write this post. The paper I keep declining is one I wrote myself.
The sentence I write most often
It is not "this is wrong." I almost never know that a result is wrong. Sitting with a PDF and no access to the authors' machine, I am not in a position to know. What I write instead is closer to this: I cannot tell whether this number would survive a change you did not make.
Authors expect the first complaint and prepare for it. They arrive with more baselines, more datasets, more ablations. The second complaint is not answered by any of that, and that is the one that decides the score.
A weak paper and an unfalsifiable paper look identical from the outside. Both report a win. Only one of them gives the reader a way to find out it is wrong.
I did this to myself for four months
I built a wheel-slip detector for a ROS 2 robot. It reached 0.869 accuracy and cut position error by 72 percent. That is a publishable-looking abstract, and I believed it.
The first crack was a control arm. A robot driving slower with no detector at all did better than my detector. Then a sharper measurement explained why: wheel encoders and an IMU cannot distinguish a robot stuck with its wheels spinning from a robot driving normally. Balanced accuracy 0.552, against 0.500 for a coin.
That was already a real finding, and I published it. Two days later I found the actual result, and it invalidated the study that produced it.
The robot had never been level. It was resting on its front bottom edge, pitched forward at 13.30 degrees, in every run I had ever done. Three independent channels agreed, and none is derived from the others:
| Channel | Reading |
|---|---|
| Ground-truth pose from the simulator | pitch +13.30° |
| IMU orientation | pitch +13.30° |
| IMU accelerometer, x axis | −2.25 m/s², which is −g sin(13.3°) |
The robot's own description predicted it in one line. Drive wheels on the centre axle, the only caster at the rear, nothing holding up the front half of the chassis. It tips until the front edge touches, at atan(0.033 / 0.1435) = 12.96 degrees. Wheel radius over half the chassis length. The measured 13.30 was geometry, not noise.
Add a front caster and the same world produces this:
| Robot | Distance travelled | Odometry reported | Error |
|---|---|---|---|
| Tipping 13.3° (all earlier work) | 6.44 m | 9.61 m | 49.4% |
| Level | 9.64 m | 9.64 m | 0.00% |
The slip was gone. Not reduced, gone. The effect I had spent four months measuring was being manufactured by the chassis ploughing the ground.
The controls were fine. They were also blind.
This is the part I now carry into every review.
My controls were not sloppy. Constant-speed baselines. Leave-one-terrain-out. A trade-off curve. A dominance test against measured arms rather than interpolated ones. Every one of them was run correctly, and not one of them could have caught this.
An ablation compares two arms against each other. Both of my arms ran on the same rig. When the fault lives in the substrate that every arm shares, comparisons between arms are blind to it by construction. This is not a discipline problem. More ablations would have produced more confident wrong numbers.
So when a submission answers a methodological doubt with "we ran an ablation," that is frequently not an answer. It tells me the arms differ. It tells me nothing about the ground they all stand on.
And the tell was in my own repository the whole time. The contact sensor reported "touching" in every single run, and I had written a comment attributing that to sensor semantics. The body was touching the ground. The sensor was reporting correctly. I had explained away my only direct evidence.
What I look for now
Five questions. I ask all of them, in this order, and the first one carries most of the weight.
What measurement, if it had come back differently, would have killed this result? If the paper contains no such measurement, the result is not supported, it is merely reported. Strong papers name this themselves. Most do not.
Which numbers were measured, and which were interpolated? A curve drawn through four measured points and a curve drawn through four assumed ones look the same in print. Only one of them is evidence.
Is there a condition where the proposed method should fail? Was it run? A method that wins everywhere has usually not been tested where it loses. My high-friction ramp exists for exactly this reason: if the robot cannot climb that either, the problem is torque, not slip, and nothing can be reported at all.
What did the apparatus itself report? Not the model, the rig. Sensor health, orientation, dropped frames, the fields nobody plots. My whole correction came out of a channel I had never looked at.
Does the paper survive its own limitations section? Sometimes a paper lists a limitation that, taken seriously, removes the contribution. Authors write these sections last and honestly, and then do not reread the claims above them.
If you are submitting
Hand the reviewer the shot that could kill your paper, then show that it missed. Name the condition under which your method should break, run it, and report what happened. A paper that does this is much harder to reject than one that reports only wins, because it has already done the reviewer's job in public.
The instinct runs the other way. Naming your weakest point feels like inviting the rejection. In practice it is the difference between a reviewer who cannot verify your claim and one who can. Only the second kind can argue for you.
I did not catch mine for four months. My controls were correct, I ran the experiments myself, and I still published a result that the apparatus had manufactured. That is not a failure of care, and I do not think reviewing is about catching people who were careless. It is about asking the one question the author was too close to ask.
The full correction, with the ramp experiments that recovered the study, and the original post it corrects.