A frozen protein model reads hazard signal that alignment misses

Suppose you run the order desk at a company that synthesises proteins to order. A sequence arrives. You have to decide, before anything is made, whether it is a toxin or a virulence factor: something that helps a bacterium damage a host. Getting this wrong in the permissive direction is the failure everyone worries about. Getting it wrong in the strict direction has its own cost, because every false alarm is a customer whose legitimate order you have stalled and a human who has to adjudicate it.

The standard tool for this is alignment: line the submitted sequence up against a database of sequences already known to be dangerous, and look for a good match. Alignment is fast, interpretable, and very effective when the submitted sequence is a near copy of something in the database. It fails in a specific way. Proteins can keep a function while their sequence drifts far from any relative, so a genuinely dangerous protein can be too distant from every catalogued one for alignment to flag. In the held-out set used here, 96.62% of the dangerous proteins are ones alignment misses.

Protein language models are trained on hundreds of millions of natural protein sequences and are known to encode structure and function in their internal activations. That makes an obvious question: does what such a model already knows cover the gap that alignment leaves? The experiment behind this page freezes one of these models, reads out one internal layer, fits a simple linear classifier on top of it, and asks how well that classifier separates hazardous proteins from harmless ones the model has never seen, in families no training sequence belonged to.

It separates them well. And it is not a screen. Those two sentences are not in tension, they are measurements of two different things, and the distance between them is the point of this page.

What was measured

The model is ESM-C 6B, frozen: no fine-tuning, no gradient reaches it. The read-out is taken from one layer roughly three quarters of the way up the stack, averaged over the protein's residues, and a linear classifier is fit on that average. The classifier is the only thing trained.

The hazardous proteins are 6,498 bacterial virulence factors and toxins from curated databases. The controls are 107,923 other bacterial proteins drawn from the same 250 species, which matters: if the controls came from harmless organisms, any classifier could win by recognising the organism instead of the hazard. Hazardous proteins are 5.679% of the pool, so the task is deliberately imbalanced in the direction a real screen faces.

Two baselines stand in for what you would try before reaching for a foundation model. The first is amino-acid composition: count the twenty amino acids, add a few physicochemical summaries, fit a random forest. It has no idea what a protein does, but hazard classes have compositional tells, and a surprising amount of published performance in this area is reachable this way. The second is the alignment search, the incumbent tool.

The split is the part that decides whether any of these numbers mean anything. Proteins come in families of relatives, so splitting sequences at random puts near-copies of the same protein on both sides of the train/test boundary and lets a model score well by recognising something it has already seen. Here all 114,421 sequences were clustered at 30% sequence identity into 15,967 families, and whole families were assigned to one side or the other. The held-out side holds 4,791 families, 1,951 hazardous proteins and 32,447 controls, and shares no families and no identical sequences with the training side.

Ranking performance on the held-out split

Ranking performance is usually reported as AUROC: the chance that a randomly chosen hazardous protein scores above a randomly chosen control. The probe reaches 0.9283 (0.9156 to 0.9396), amino-acid composition 0.8196, and the alignment search 0.7096. The ordering holds when the arms are compared on identical resampled draws of families, so each pair is scored on the same proteins; the caption above gives those paired margins. Retraining the same probe on shuffled labels never gets above 0.5171, which is where a classifier with nothing to learn from should sit.

So there is real signal in the frozen representation, it is not an artefact of composition, and it is not the alignment result in disguise. On the specific gap that motivated the question, the 96.62% of held-out hazards that alignment misses, the probe keeps working.

At a strict false-alarm budget

AUROC asks about the whole ranking. A screen asks something much narrower: with a fixed and small budget of false alarms, how many hazards do you actually catch? Those two questions have very different answers here.

At the 0.1% budget the probe flags 194 proteins, 162 of them genuine hazards and 32 false alarms, so almost everything it flags is real. The failure is coverage rather than false alarms: 1,951 hazards exist in the held-out set, so the remaining 1,789 sit lower in the ranking, mixed among ordinary proteins. They are mixed in because the controls come from the same 250 pathogen species as the hazards, so the most hazard-like harmless proteins score in the same band as the hazards the model has weakest evidence about. Loosening the budget to a 10% false-alarm rate recovers 79.70% of hazards at the cost of 3,244 false alarms, which is triage with a human reviewing flags rather than an automatic gate.

On identical resampled draws of families, the probe's advantage at the 0.1% budget is +5.433 percentage points of recall over composition (0.8981 to 9.309) and +4.511 points over the alignment search (0.3937 to 8.887), both intervals excluding zero.

Being two to three times better than nearly nothing is still not a screen: roughly nine in ten dangerous proteins go through.

Two biases sit on that figure and they point in opposite directions, so it should be read as an estimate rather than a bound. The threshold that produces it was read off the same control proteins it was then evaluated on, which flatters it. Against that, the probe was trained with controls subsampled to a few per hazard, which is exactly the regime that hurts a classifier at the extreme tail of its own score distribution, so a version trained on the full imbalance might do better here. Neither effect has been separated out. A stricter operating point, 0.01%, cannot be reported at all: 32,447 controls put about three in that tail, too few to estimate from.

This is the most portable finding in the work, and it is worth stating in its general form: when data has group structure, a random split does not measure generalisation, it measures memorisation, and it rewards the methods that memorise most cheaply. The size of the effect is what makes it more than a methodological footnote. In the cross-benchmark setting below, letting shared families back into training inflates the probe's score by 0.1353 AUROC (0.1101 to 0.1630): 0.8322 instead of 0.6968. That single number is larger than the probe's entire margin over the composition baseline (0.1087). A published headline in this area that comes from a random split is therefore not comparable to one that does not, and the difference is bigger than the effect most such papers are reporting.

Transfer to independently built benchmarks

A result on one cohort, built by one team, is one result. So the probe was retrained on every other cohort available, with every sequence family present in the held-out cohort removed from the training pool (7,877 families removed in the hardest case), and then evaluated on the held-out cohort. Both baselines were refit on exactly the same pool, so all three arms see the same training data.

One comparison against a purpose-built published model is worth making carefully. ToxDL 2.0 is a model trained specifically for protein toxicity, and it reports its own performance on a holdout of proteins collected after the start of 2022. Refitting this probe on ToxDL 2.0's own training data, ten seeds, gives 0.9573 AUROC on that holdout, against the 0.945 the model's authors report. The interval on our number, 0.9385 to 0.9737, contains both. With 152 toxic proteins among 4,861 in that set, the right reading is that a linear probe on a frozen general-purpose protein model lands level with a purpose-built supervised model, not ahead of it.

What the probe reads

A classifier that works is not the same as an explanation, so the next question was which parts of a protein the read-out depends on. One protein made the question vivid.

Read on its own, that picture suggests something specific and interesting: that the model recognises a hazardous protein by how it gets delivered into a host cell rather than by what it does once inside. Delivery machinery is shared across many unrelated effectors, so a feature like that would transfer across families in exactly the way the results above show.

The aggregate association holds. Proteins carrying a delivery-related annotation in their first 30 residues draw more attention there than comparable proteins without one, matched on length and holding the region of the sequence fixed, and the effect is roughly three times larger for the 342 annotated hazardous proteins (+0.3608) than for the 4,191 annotated controls (+0.1257). Among the toxins of two other cohorts, annotated proteins draw a similar amount of extra attention (+0.2999 and +0.3533), though the hazard-versus-control comparison in those cohorts is underpowered.

Two checks on the probe that actually performs best, the plain averaging one, remove that reading. Masking the delivery region against a control region matched on sequence position changes its score by +0.02909, an interval straddling zero, so it does not depend on that region specifically. And the sharp two-residue map was not necessary: an attention probe retrained with 10,241 parameters instead of 1,312,257 spreads its attention out, dropping the median weight on its top residue from 83.33% to 6.893%, and matches accuracy at 0.9189 AUROC against the sharp probe's 0.9137.

So the striking single-residue picture was a property of a particular probe with capacity to be peaked, not a window onto how the representation encodes hazard. The best-performing probe has no region we could localise. That is a real negative result about the interpretation, and it leaves the aggregate delivery-annotation association standing as an association without a demonstrated mechanism.

Where this leaves things

A frozen general-purpose protein model carries hazard information that sequence alignment does not reach, the effect is not explained by amino-acid composition or by families leaking across the split, and it holds up on independently built benchmarks. Read as an instrument for ranking or triage, that is a solid result. Read as a screen, it fails: at a false-alarm budget a submission desk could actually run, it recovers 8.303% of hazards.

The shape of that failure suggests where to look next. The score ordering is good overall, which means the information is present; what is missing is enrichment right at the top of the ranking, which is a property of the score's calibration and of the tail of its distribution rather than of the representation. The concrete next test is to retrain the read-out on the real class imbalance instead of subsampled controls, and to set the threshold on controls that were never scored during evaluation. If the tail improves under those conditions, the gap is a calibration problem. If it does not, the representation genuinely does not separate the hardest hazards from the most hazard-like harmless proteins, and no amount of thresholding will fix it.

Limitations

• Ranking well and screening well are separate properties. Only the first is established here, and the second is where a deployment decision would live.
• These numbers are not comparable to a published leaderboard. The task was made harder on purpose: controls from the same organisms as the hazards, realistic imbalance rather than balanced classes, a stricter identity cutoff between train and test, and no architecture search or ensembling.
• The model's pretraining data cannot be audited for exposure to these proteins. The family-disjoint split controls what the probe saw during training, not what the underlying model saw during pretraining, and that bounds every representation-based arm here equally.
• Hazard labels are curatorial artefacts. Whether a protein is catalogued as a virulence factor correlates with how well studied its organism is. Drawing controls from the same species bounds this, and does not remove it.
• Everything here is natural sequences. This tests whether distant natural relatives of known hazards can be detected, not whether deliberately redesigned sequences can be, which is the harder threat model.
• One benchmark of the four supports no directional conclusion, and its labels had to be reconstructed from counts published in a paper rather than read from a label field, so its result should not be read as evidence about virulence-factor cohorts generally.