Two hundred lesions. Every one biopsied. Thirty of them melanomas. That is the test set Magnus Lynch, a consultant dermatologist at Guy's Hospital, gave to GPT-6 Astra, Claude Fable 5.1 and Inception v3, a convolutional network Google published in 2015 that he fine-tuned on public dermatology images for about twenty minutes. The old network named the right diagnosis 62% of the time. The frontier models managed 42%. That gap is what drew 575 points on Hacker News.
It's the less interesting number. The row that matters sits further down Lynch's table: how many melanomas each system would have sent home. And it says something the accuracy figure doesn't. The CNN's advantage isn't that it knows more dermatology. It's that its line between discharge and biopsy is a number someone set, can see and can move. The LLMs have a line too. Nobody set it.
Start with the uncomfortable part. At its preset threshold the CNN discharged 76 of the 100 harmless lesions, and 18 of the 30 melanomas with them. Astra sent home three melanomas. Fable sent four. Read that row alone and the old network is the dangerous one.
Then look at the same row with the CNN's threshold matched to the LLMs. Now it sends home two melanomas, or one, against their three and four. Same network, same weights, same 200 images. The only thing that changed was where the line sits, and that line is a parameter: one number in a config file.
The LLMs' line lives somewhere else, in training and in the wording of the prompt. "Each LLM had a favourite answer," Lynch writes. Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas; Fable called 53 actinic keratosis when 20 were. Then he changed the question. Given only the clinical photograph and asked, as a patient, whether they needed to see a doctor, both models said yes almost every time, harmless lesions included. Ask as a clinician and they discharge three or four melanomas in thirty. Ask as a patient and they refer nearly everything. The threshold moved because the phrasing did.
The threshold moved because the phrasing did.
The fair objection is that a dial is only as good as the hand on it. The CNN's preset sent 18 melanomas home, and somebody chose that preset. A default that bad would be a scandal in a clinic. But it's a scandal you can find. The threshold is one number, the trade it makes is one row of a table, and a clinic can pick the point it's willing to defend, so many benign biopsies to catch one more melanoma, and write that choice down. Lynch is careful about scale: thirty melanomas give wide error bars, and the regulated products built on newer architectures do considerably better. None of that touches the mechanism. He fixed the CNN's worst row by changing one value, and he published the cases, prompts, every model reply and the scoring code so anyone can check that he did. There is no equivalent value to change in Astra. The nearest thing is rewording the prompt, and the patient test shows how far that swings the answer without anyone deciding it should.
Medicine already knows how to argue about a cutoff. It does it for PSA levels, for mammogram recalls, for every screening test with a sensitivity and a specificity. That argument needs a number to argue about. The twenty-minute model has one. The frontier models have a favourite answer.