What a finetune can hide
You download a model. It comes with a name, a card, a few benchmark numbers, and a one-paragraph account of what its makers did to it. Then you wire it into something that matters, and you trust that account — because there is nothing else to trust.
That account is a claim. It is not evidence. The distance between the two is the whole reason this house exists.
A description is not a witness
A base model is a known quantity in the weak sense that many people have looked at it. A finetune is a private act performed on those weights: someone took the base, showed it data you cannot see, and shipped the result. What changed is now inside the numbers, and the numbers do not narrate themselves.
Most of what a finetune changes is exactly what its makers meant to change — a tone, a format, a domain skill. But the same operation that teaches a helpful habit can install a quiet one, and nothing in the file marks the difference. The model behaves. Until the one input that was trained to matter arrives, and then it behaves differently.
The failure mode is not a model that is obviously broken. It is a model that is obviously fine — right up until the moment it was shaped to stop being fine.
You cannot close this gap by reading the card, and you cannot close it by sampling outputs, because the behavior you are worried about is the behavior that only shows on inputs you would never think to try.
The three shapes of the problem
Hidden behavior in a finetune tends to take one of three shapes, and it is worth naming them plainly.
- A disposition it will not admit to. The model leans a way its card does not mention — a bias, a refusal pattern, a loyalty. It answers your probes cleanly and keeps the lean for the cases you did not probe.
- A trigger someone planted. A specific, unlikely input flips the model into a different mode. The rest of the time it is indistinguishable from the honest version, which is the point.
- A drift no one decided. No adversary, just training that moved the model somewhere its makers did not intend and did not notice, because the eval suite did not ask.
Each of these is invisible to the tools most teams already run. An eval harness measures the behavior you thought to measure. Observability watches production, after the fact, on the traffic you happened to get. Neither is built to make a model give up an account of itself it was shaped not to give.
Holding on until the true shape shows
The old name for this house is a story about a shape-shifter who knows the truth and will not surrender it — until he is held through every false form he can take. Let go early and you keep only the shape he happened to be wearing: a lie that escaped.
That is the discipline, made literal for a model:
- Detect — get the true account out. What the finetune actually did, including what it was shaped to hide, surfaced rather than sampled.
- Attest — set that account down as evidence, not assertion. Every finding travels with a replayable witness: a reproduction you can run to make the effect fire with ordinary tooling. A finding you cannot reproduce is not a finding.
- Excise — where something is in the model that should not be, cut it and attest that only that changed. The correction carries its own proof.
The order is not decorative. Detection without attestation is a rumor. Attestation without the option to excise is a diagnosis with no treatment. The point is to move from "the card says it's fine" to "here is what it does, here is how to see it for yourself, and here is the model with the bad part removed."
The part that earns the trust
There is a fourth move, and it is the one that makes the other three worth anything: when the grip cannot be kept, the instrument says so.
Where it cannot prove an effect, it does not guess to look thorough. It abstains — visibly, with a reason — and the abstention is recorded as plainly as any finding. This is not humility for its own sake. A tool that fabricates findings to look complete is worse than no tool, because now you trust a number that was invented. The abstain list is a feature, not a footnote.
That is why what slipped goes up with the same prominence as what was caught, in trust and the open record. A record that prints only its wins is a brochure; one that prints what was held and what got away is the only kind you can lean on.
Why write this down
Because the gap between a claim and a witness is not a Protora problem — it is everyone's, the moment they ship on top of weights someone else shaped. This journal is where the house writes about that gap: the discipline behind the record, the findings worth explaining, and the craft of proving a thing about a model instead of asserting it.
The instrument is one answer. Reading it should not require taking our word for it either — which is rather the whole idea.
