The first LoRA I trained looked perfect on the prompt I used during testing. Same character, same lighting family, same camera distance. I saved the file, named it something optimistic, and assumed I had a reusable identity tool.
Then I changed the scene.
New background, new pose request, slightly different outfit wording—and the identity collapsed. Sometimes the face drifted. Sometimes the style overpowered the prompt. Sometimes the output looked like a distant cousin of the training set and nothing else. The adapter had not learned a character. It had memorized a narrow corridor.
This Failure File is about that corridor, how I found it, and what I changed before training the next version.
I am sharing the failure in detail because the successful screenshot is still the default currency of AI sharing. The failed generalization is the part that actually improves the next training run. Run the test. Save the settings. Show the failure. This post is the “show the failure” half. The settings from that run still sit in the notebook so I cannot romanticize what went wrong.
What “Worked Once” Actually Looked Like
The winning test prompt was close to the average training image: similar crop, similar expression range, similar lighting direction. On that island, the LoRA was confident. Off that island, it was guessing.
I had done the classic beginner mistake: evaluated the adapter on data that looked like the training distribution, then declared victory.
The Clues I Ignored
Looking back at the notes, the warnings were already there:
Strong results only on close-up or medium shots that matched the dataset framing
Immediate quality drop when I asked for full-body poses
Style leakage into unrelated prompts even at moderate weights
Almost no useful signal when I tested a deliberately out-of-distribution scene
I treated those as “edge cases” instead of core measurements. Edge cases that appear every time you leave the training comfort zone are not edge cases.
The most seductive clue was the first success. Once I had one strong image, I started explaining away the weak ones. That is the opposite of measurement. Measurement would have counted the weak ones as data, not as noise to be minimized in the highlight reel.
Symptom | What I told myself | What it actually meant |
|---|---|---|
Perfect on training-like prompts | “Identity is solved” | Memorization, not generalization |
Weak on full-body requests | “Need better posing prompts” | Dataset lacked pose diversity |
Style overpowering base prompt | “Weight is a bit high” | Adapter dominated instead of cooperating |
Failure on new environments | “One-off bad seed” | Concept never left the training corridor |
How I Confirmed the Failure
I built a small probe set: five prompts that stayed close to training, five that changed pose, five that changed environment, and three that mixed identity with a different style request. Same seeds across base-only, moderate LoRA weight, and higher weight.
The pattern was consistent. Close prompts: strong. Everything else: fragile or actively harmful. That was enough evidence to stop using the file in real work.
The probe set took less time than the original training run. That ratio matters. If evaluation is more expensive than training, people skip evaluation. Keeping the probe lightweight made it sustainable. I still use a version of that structure for every new adapter.
What I Changed Before Retraining
Dataset diversity — More poses, more distances, more lighting directions, fewer near-duplicates.
Hard negatives in evaluation — Every future LoRA gets an out-of-distribution probe set before I call it usable.
Lower default weight exploration — I now sweep weights instead of jumping to 1.0.
Honest labeling — If an adapter only works in a narrow band, the notes say so.
None of these changes required new hardware. They required refusing to be impressed by a single good island of results.
I also started keeping a short “known weak zones” list for every adapter. If a LoRA is acceptable for close-up product shots but unreliable for full-body editorial frames, that limitation belongs in the workflow notes. Silent limitations become other people’s surprises.
Result / What Worked / What Failed / Would I Use It Again?
Result: The first LoRA was a useful teacher and a bad production tool.
What Worked: It proved the training pipeline could complete and produce a visible effect.
What Failed: Generalization, pose range, and prompt cooperation outside the training corridor.
Would I Use It Again? No—not that checkpoint. The lessons, yes.
The next LoRA I trained with a more diverse set and a mandatory probe evaluation was less flattering on day one and more useful on day ten. That trade is the one I now optimize for.
Run the test. Save the settings. Show the failure.
A LoRA that works once is a demo. A LoRA that survives a probe set is a tool. The difference is not talent. It is whether you measure beyond the flattering prompt. If you are in the middle of a similar first-adapter disappointment, the failure is not evidence that LoRA is useless. It is evidence that the evaluation was too kind. Make the evaluation harder; the next version usually gets better. That is the entire point of keeping Failure Files next to the successful demos in this kitchen.
No comments yet.