Analysis — where AlphaFold goes blind

How to read this

Each bar is a fold class or superfamily; length is the fraction of its analysed members that are confidently wrong (AlphaFold sure, fold wrong), with a 95% confidence interval. Bars are ranked by the lower bound, so a class doesn't top the chart on one lucky hit. These are AlphaFold's structural blind spots.

CW-rate + CI: the Wilson interval on the proportion. centre = (p + z²/2n)/(1 + z²/n), p = n_cw/n, z = 1.96.

Enrichment & p: the class's CW-rate over the overall CW-rate; the p-value is a one-sided Fisher exact test on the 2×2 [in-class/out-class × CW/not].

Folds & shared themes

CATH architecture (left): the striking result is that the large majority of confident failures have no CATH fold at all — they are amyloids, short peptides and disordered regions that AlphaFold renders as crisp but wrong structures (this fraction is stated in the chart title). The bars show only the classified minority, ranked by count and coloured by CATH class; each bar is a named architecture (e.g. Sandwich, Orthogonal Bundle), and the hover gives the most common specific fold within it.

Shared themes (right): the same set, tagged by cause — amyloid fibrils, large assemblies (AlphaFold DB models the isolated monomer), disordered regions, coiled-coils, and engineered/designed constructs. Flags combine literature/name keywords with structural signals (large Rg change, low secondary-structure agreement).

How to read the similarity heatmap

This compares the worst offenders to each other by sequence. Every structure is aligned against every other; each cell is their percent sequence identity (dark = unrelated, bright yellow = near-identical). The rows and columns are re-ordered so that similar sequences sit next to each other, which makes homology families appear as bright square blocks running down the diagonal.

The coloured boxes outline the families the algorithm found; each is labelled in the left margin by its most common protein name and size, e.g. transthyretin (5). A big bright block means AlphaFold's blind spot is really one family caught many times (many fibrils of the same protein); scattered singletons mean the failures are structurally diverse. Bright cells off the diagonal flag similarity between different families.

How it is built: pairwise global alignment (biotite, BLOSUM62) gives identity; distance d = 1 − identity; grouped by SciPy average-linkage hierarchical clustering and cut at 30% identity. families = fcluster(linkage(d), t=0.7). Capped at the top ~90 offenders by FRAUD.

Families & failure modes — the two views

The heatmap above splits the offenders by sequence (who is related to whom); this map splits them by failure mode (how AlphaFold got it wrong). It is a 2-D PCA of each structure's metric fingerprint (TM-score, lDDT, RMSD, radius-of-gyration change, secondary-structure agreement, contact overlap, PAE overconfidence), so nearby points failed in similar ways — global-fold-wrong vs domain-hinge vs local-melt. Point size = FRAUD; colour = CATH class; hover shows the PDB id.

What predicts a confident miss?

Confidently-wrong rate broken down by experimental method, resolution, chain length and sequence novelty. Cryo-EM (fibrils and assemblies) and low novelty (AlphaFold confidently reusing a wrong homolog) tend to enrich.