How to Decide Whether to Add More Training Data to a LoRA

A strict two-question decision gate for adding training data to a LoRA: name the specific underrepresented attribute, confirm clean examples exist, and re-test the original failure mode before keeping the new images.

Adding images feels productive. It is also one of the easiest ways to introduce near-duplicates, caption drift, and longer training times that encourage overfitting. The default in this kitchen is therefore “do not add.” Addition is the exception that must earn its place.

This post is the decision gate I actually use. It is deliberately narrow.

The Gate: Two Questions, Written Down

Before any new image enters the training folder I force myself to answer two questions in a plain-text note:

Exactly which attribute or pose is underrepresented, and how do I know the current set fails on it?
Do I already possess clean, identity-card-compliant examples of that attribute that do not introduce new near-duplicates or caption noise?

If either answer is vague, the addition is postponed. The act of writing the answers has stopped more hopeful data dumps than any technical filter.

Two-question gate checklist for deciding whether to add LoRA training data

How the Gate Handles Common Situations

Situation	Gate answer	Action
Character fails on extreme low-angle shots; set has zero low-angle examples	Gap is named; clean examples exist	Add only the clean low-angle images, then re-probe
Character fails on extreme low-angle shots; base model also fails the same angles	Gap is named but examples cannot teach a transferable trait	Document as known limitation; do not add
“More variety would be safer” with no specific failure	Gap is not named	Do not add
New lighting style required for a client; current set is locked to soft indoor light	Gap is named; clean examples can be shot or sourced	Add, caption carefully, re-probe original failure mode
Training loss still dropping and grids look fine	No failure has been observed	Do not add; stop training instead

The table is not exhaustive. It is a pattern. Named gap plus clean examples equals possible addition. Everything else defaults to no.

Probe pass rate before and after justified training data addition

What Happens After a Justified Addition

I never retrain on the enlarged set and assume success. I re-run the identical probe set that failed before. If the specific failure mode improves and no new failure modes appear, the addition is kept. If the original failure remains, the new images were the wrong medicine and are removed again.

Even after a justified addition I still run the full strength sweep before publishing a new recommendation. More data can shift the usable strength range. Assuming the old range still applies is another quiet way to re-introduce overfit.

Why “More Is Safer” Is Usually Wrong

In the early days of LoRA work I believed a larger set was automatically more robust. The opposite proved true for small character adapters. Larger sets increased the chance of subtle near-duplicates and made it harder to keep captions focused. The model learned the average of the noise rather than the intended identity. Smaller, deliberately balanced sets have produced wider usable strength ranges and fewer surprise failures on out-of-distribution prompts. Size is not a proxy for quality; distribution and cleanliness are.

A recent borderline case illustrates the discipline. A character LoRA failed on extreme low-angle shots. The existing set had zero low-angle examples. I could have added ten quickly. Instead I first checked whether the identity card still held at that angle on the base model alone. It did not; the face structure itself was unstable at low angles even without the LoRA. Adding data would have trained the adapter to compensate for a base-model weakness rather than teaching a transferable trait. I documented the angle as a known limitation and left the set unchanged. The decision felt conservative at the time and correct a week later when a different model update improved the base low-angle behavior.

Run the test. Save the settings. Show the failure. More data is a tool, not a superstition. Use it only when the gap is visible and the examples are already good enough to teach the right lesson.

The gate also interacts cleanly with strength testing. Even after a justified data addition I still run the full strength sweep before publishing a new recommendation. More data can shift the usable strength range. Assuming the old range still applies is another quiet way to re-introduce overfit. The gate and the sweep are sequential tools, not alternatives.

I have found that writing the two answers in a shared note (even if only I will ever read it) forces a level of honesty that mental review does not. Vague answers become obvious on the page. That small friction is the entire point of the gate. Without it, the path of least resistance is always to add a few more images and hope.

The pre-addition checklist I keep next to the two questions is short on purpose:

Confirm the new images already satisfy the identity card under the lighting and camera distance I plan to use at inference.
Confirm they do not introduce new background objects or pose concentrations that the existing set lacks.
Confirm captions stay focused on the intended identity and do not drift into scene description.
Plan the exact probe set that will be re-run after training to verify the original failure mode improved.

If any item on the checklist fails, the images go back to the source rather than into the trainer. The checklist takes ten minutes and has prevented more wasted training runs than any learning-rate schedule I have tried. Adding data is easy. Adding the right data, and then proving it helped, is the actual work.