The LoRA looked excellent on the training-style prompts. Faces locked. Details popped. Clients were impressed by the first grid. Three days later a new prompt set arrived and the same adapter produced plastic skin, collapsed profiles, and wardrobe that refused to change. Classic overfitting, diagnosed late because the early tests were too kind.
This is the failure file for that adapter. I am writing it as a timeline because that is how the problem actually unfolded, and because the sequence itself is the lesson.
Day 0: Training Finished, Grids Looked Clean
I trained a character LoRA on a set of 32 images that already satisfied the identity card. Training stopped at a checkpoint that produced sharp, consistent faces on the training distribution. I ran a quick in-distribution grid, liked what I saw, and published a temporary strength of 0.7. No out-of-distribution probes. No zero-weight control. The early grids were seductive enough that I skipped the rest of the protocol.
The recipe card went into the shared shelf with a short note: “Strong identity lock, recommended 0.65–0.75.”
Day 1–2: The First Client Brief Exposed the Crack
The client needed the character in a new wardrobe and under hard outdoor light. Half the frames lost the scar. The jawline drifted. Skin took on a plastic quality that had never appeared in the training-style tests. I lowered the strength to 0.5 and the identity improved, but the detail the client had liked disappeared. Raising it again brought the artifacts back.
That evening I ran the full probe set I should have run on Day 0. The pass rate collapsed above 0.4 on any prompt that stepped outside the training distribution. The adapter had learned the training outfits and lighting far more thoroughly than the underlying identity.
Day 3: Confirmation and the Decision Not to Patch
I confirmed the overfit with three additional tests:
Zero-weight control still produced a usable base-model face, proving the identity was not solely dependent on the LoRA.
Strength sweep from 0.0 to 0.8 showed a clear cliff between 0.4 and 0.5 on out-of-distribution prompts.
Dataset audit revealed near-duplicates and a heavy concentration of three-quarter poses under soft indoor light.
I refused three tempting shortcuts: raising strength and hoping, adding more training steps, and publishing a note that said “use lower strength for hard prompts” without fixing the root. Those moves would have left the next operator with the same trap.
Day 4–5: Clean, Retrain, Re-Probe
I removed the near-duplicates and the most pose-concentrated images, bringing the set down to 24 cleaner examples. I retrained with an earlier stop. The new adapter cleared the original out-of-distribution probe set at 0.45–0.55. The usable range was wider, the plastic-skin artifact was gone, and the scar held under the client’s outdoor lighting.
The recipe card was re-issued with the new range, a note about the original failure, and a requirement that any future strength change must be accompanied by a fresh out-of-distribution sweep.
What the Timeline Made Visible
The seductive phase lasted less than 48 hours. The cost of recovery was one focused day and a modest amount of compute. The cost of not catching it would have been client revisions, lost trust, and a shared shelf that taught the wrong lesson to the next person who picked up the card.
I now show new operators the original seductive grid next to the later collapse and the recovered probe table. The visual sequence is more persuasive than any abstract warning. Once they have seen how cleanly an overfit can hide inside pretty in-distribution results, they become far more willing to run the full sweep before publishing a strength.
The prevention habits that survived this failure are simple and non-negotiable:
No strength recommendation is published without a completed out-of-distribution sweep that includes 0.0.
Dataset unique-count and pose-distribution tables are required artifacts before training starts.
Any new adapter spends its first day on the internal probe set only; client-facing grids come after the pass rate is known.
Run the test. Save the settings. Show the failure. The overfit LoRA is not a rare accident. It is the default outcome of testing only on the prompts that already look good.
The timeline also changed how I talk about success. An adapter that looks excellent on Day 0 is no longer celebrated until it has survived at least one out-of-distribution probe set and one hand-off to a second operator. Early aesthetic reward is treated as a risk signal rather than a green light. That cultural shift inside the kitchen has been more valuable than any single technical fix.
I keep the original seductive grid, the collapsed OOD results, and the recovered probe table in a single folder labeled with the failure date. New operators are shown the folder before they publish their first strength recommendation. The folder is more persuasive than any checklist I could write. Seeing the sequence once is usually enough to make the full sweep feel non-negotiable.
The cost of recovery was one focused day and a modest amount of compute. The cost of not catching the problem would have been client revisions, lost trust, and a shared shelf that taught the wrong lesson to the next person who picked up the card. Paying the recovery cost early remains almost always cheaper than paying it after the work has left the kitchen.
No comments yet.