The fit that can't tell

What you're seeing

The spectrum goes through the real XMM-Newton EPIC-pn response I used in the paper, with the paper's source model: an absorbed power law plus a blackbody, five free parameters and no background. The gain slider stretches the energy scale the source is read on, so at 3% every feature lands 3% lower in energy than it should. The fit assumes the detector is fine and gets redone from scratch every time you move anything. Each channel gets Poisson counts, and the same random numbers are kept as you move the slider, so the only thing that changes is the gain. Switch photon noise off and the data become the exact expected counts. The fit then sits on the truth at zero gain, but the checks have nothing to judge, so they stand down.

It hides because a power law looks exactly the same after you stretch the energy axis, only its normalisation changes, and a stretched blackbody is just a slightly cooler blackbody. Nearly all of the gain error gets soaked up by the source parameters. What's left sits in the absorption edges, which don't move with the source, and that's where the photon index gets pushed. The terrain shows this: at every point it is the best fit you can get with Γ and the power-law norm pinned there, and the height is √ΔC, so each glowing contour is about one more sigma. As the gain grows the valley keeps its depth and the ball slides along it, far less than one contour at 3%, which is why the close-up in the corner exists. A strong iron line does the opposite: the whole fit gets worse, and the terrain turns red. For the default source at 10,000 counts, even a 10% gain error makes the best fit worse by about 1 in the Cash statistic, which noise does on its own.

The three lights are simplified versions of the three schemes in the paper. The replay test is the posterior-predictive check, a chi-square and a cumulative-counts comparison against 400 replayed spectra. The reweighting test follows the importance-sampling check, which reweights the network's posterior against the exact Poisson likelihood. The toy has no network, so its rough first guess is a fit to the spectrum squeezed into 20 broad bands (the network also squeezes each spectrum into 20 numbers), and the light compares how many draws survive with 100 clean spectra of the same source. The evidence test stands in for nested sampling's evidence, compared with clean spectra at the same counts. Here it's the best-fit Cash statistic against what a clean spectrum with those counts should give. On 150 clean noisy spectra of the default source each light went red 0 to 2% of the time, and the 3% gain didn't change that.

For one source the push is a small slice of the error bar, which is why nothing complains. The catalogue shows the other half. In the paper the network's photon index came out biased by +0.018 ± 0.006 at 3% and about 1,000 counts, and +0.019 at 10,000 counts, which is only 0.03 to 0.11σ per source but goes the same way for the whole sample.

The marginalise switch refits with the gain as a sixth unknown, flat between 0.95 and 1.05, like the retrained network in the paper. In the toy the gain usually comes back nearly as wide as its prior, the error on Γ barely moves and the push stays. The paper saw the same thing: the retrained network gave the prior back on the gain, its σ(Γ) changed by a factor of 0.989 to 0.996, and the bias was +0.020 ± 0.006. Exact nested sampling did get some gain information out of three bright spectra, and the network caught at most 0.163 of it. So marginalising doesn't remove the bias. It's still the defence the paper ends on, because it costs nothing when you fit and none of the checks warns you anyway. Past 5% the true gain is outside that prior, and marginalising has no way of knowing.