8EB0_A confidently wrong novel

Per-residue accuracy vs. confidence

Reading along the protein chain: red is how far each residue sits from the experiment (Cα deviation in Å, higher = worse); green is local accuracy (lDDT×100); blue dotted is AlphaFold's own confidence (pLDDT). Stretches where confidence stays high but the red line is large are exactly where AlphaFold is confidently wrong.

Take-home: mean confidence pLDDT 85.77 vs. overall accuracy lDDT 0.81 and TM-score 0.47.

The metrics

Cα deviation: how far residue i sits from where the experiment places it, after superposing the whole chain. Δᵢ = |Pᵢ − (R·Qᵢ + t)| Å, with Pᵢ/Qᵢ the experimental/model Cα coordinates and R,t the best-fit rotation and translation.

per-residue lDDT: local accuracy at residue i without superposition — the fraction of i's neighbour distances (within 15 Å) the model preserves. lDDTᵢ = ¼ Σ_t 1[ |d_exp − d_model| < t ], t ∈ {0.5, 1, 2, 4} Å.

pLDDT: AlphaFold's confidence for residue i (0–100) — its own predicted lDDT, output by the network before seeing the experiment.

Is the confidence honest?

Each point is one residue: AlphaFold's predicted confidence (pLDDT, horizontal) against its actual accuracy (lDDT×100, vertical). Points on the dashed diagonal are perfectly calibrated; points well below it are overconfident — AlphaFold was surer than it should have been.

Take-home: pLDDT–lDDT correlation 0.4852 (near 1 = well calibrated; near or below 0 = confidence unrelated to, or opposite, real accuracy).

The metrics

pLDDT (x): AlphaFold's predicted per-residue confidence, 0–100. lDDT×100 (y): the accuracy actually achieved at that residue. Perfect calibration puts every point on the diagonal pLDDTᵢ = 100·lDDTᵢ.

Calibration correlation: the headline is the Pearson correlation of the two across all residues. r = cov(pLDDT, lDDT) / (σ_pLDDT · σ_lDDT) — near 1 means confidence tracks accuracy honestly; ≤ 0 means it does not.

Where the shape differs

The difference between the experimental and predicted residue–residue distance maps (Å). Bright regions mark pairs of residues whose separation AlphaFold got wrong — often a whole domain placed in the wrong position relative to the rest of the structure.

Take-home: mean distance-map difference 12.49 Å.

The metric

Distance-matrix difference: each cell is how much the separation of residues i and j differs between prediction and experiment. |Dᵢⱼ^exp − Dᵢⱼ^model|, where Dᵢⱼ = |rᵢ − rⱼ| is the distance between the two residues. Superposition-free, so a domain in the wrong place shows up as a bright off-diagonal block rather than being averaged away.

Did AlphaFold know it was wrong?

Left: AlphaFold's own predicted error (PAE, Å) for each residue pair. Right: the error we actually measured. Where the right panel is much brighter than the left, AlphaFold underestimated its own error.

Take-home: mean predicted error 13.97 Å vs. mean observed error 12.44 Å; 24.0% of residue pairs were more wrong than AlphaFold predicted.

The metrics

PAE (predicted): AlphaFold's Predicted Aligned Error — PAEᵢⱼ is the position error (Å) it expects for residue j when the structure is aligned on residue i, output by the network.

Observed error: the frame-invariant reality we measure for the same pair. Eᵢⱼ = | |rᵢ−rⱼ|_exp − |rᵢ−rⱼ|_model |. If the observed panel is far brighter than the predicted one, AlphaFold underestimated its own error — it was overconfident.