How to train a character LoRA: a practical start-to-finish guide

Choose the base before anything else

A LoRA does not replace its base model. It steers it. The adapter learns efficiently because the base already knows faces, cameras, fabrics, lighting, and composition. That also means the base model’s anatomy, prompting language, license, and visual bias come along for the ride.

Start by generating ten untrained images close to the work you want to make. If the base cannot produce the age range, photographic style, body framing, or skin rendering you need, training identity into it will not repair the foundation. Choose another base now, before paying a GPU to memorize your workarounds.

Do not assume adapters transfer between families. A LoRA trained for FLUX.2 Klein is not a Z-Image or SDXL LoRA. Store the exact base model ID, revision, hash, license, training tool commit, and configuration beside every output.

Hardware and software prerequisites

FLUX.2 Klein 4B is small by current standards, not small in ordinary-laptop terms. A modern NVIDIA GPU with 24GB of VRAM is the comfortable local starting point for this guide. Memory-saving options can reduce that requirement, but they usually trade speed, setup simplicity, or optimizer choices. If your machine cannot hold a stable training run, rent a GPU for a few hours instead of spending a weekend fighting out-of-memory errors.

Before opening the training UI, prepare:

• A current NVIDIA driver and a CUDA/PyTorch combination supported by the training tool.
• A clean Python environment dedicated to AI Toolkit or Diffusers.
• Enough fast disk for model weights, cached latents, previews, and several checkpoints.
• The official Base checkpoint and any access agreement required by its host.
• A dataset folder, a separate validation file, and a project folder for configs and notes.

Install from the current AI Toolkit repository and start with its FLUX.2 Klein example, because dependency and configuration names change faster than editorial guides. Save the repository commit and your package lock or environment export. “It worked last month” is not a reproducibility strategy.

Run one official sample configuration before modifying it. That separates installation problems from dataset problems. If a known-good example cannot train and produce a sample, your captions are not the issue yet.

Build a small, deliberate dataset

For a first character LoRA, 20–30 excellent images is a useful working set. Fifteen can be enough for a simple, distinctive identity. Forty may help when the subject needs more hairstyles, angles, expressions, and full-body coverage. Those are starting ranges, not a law: twenty near-duplicates contain less information than twelve deliberate photographs.

Use only images of a fictional character or a consenting adult for whom you have clear training rights. Remove minors, private photographs, leaked material, watermarked images, and photos you merely found through search. Permission to view or repost an image is not automatically permission to train a model on a person’s identity.

A balanced character set usually needs:

• Close, medium, three-quarter, and full-body framing.
• Front, three-quarter, and side angles, plus a modest high/low camera variation.
• Neutral, smiling, serious, and mid-expression faces.
• Several locations, light directions, and camera distances.
• Different outfits and hairstyles unless one is intentionally part of the character.
• Clean hands and body geometry in the frames where they are visible.

Keep five useful conditions out of training. They can be prompts, reference images, or both. A holdout is how you learn whether the LoRA understands the identity instead of recalling the dataset. The dedicated dataset guide goes deeper on shot lists, culling, aspect buckets, and captions.

Write captions and choose a trigger

Give the identity a rare, readable trigger such as rfgmara. Avoid a common word, a celebrity name, or a token that the base already associates with a visual concept. The trigger is a handle for the learned identity; it is not security and it does not make an unauthorized dataset ethical.

Caption what you want to remain controllable. If every image contains a black leather jacket but none of the captions mention it, the trainer has no reason to separate jacket from identity. The same applies to a studio background, heavy makeup, direct flash, or one hairstyle.

A useful caption is plain and factual:

photo of rfgmara, adult woman, medium shot, looking to the side,
short wavy brown hair, blue cotton shirt, overcast window light,
neutral apartment interior

Do not caption stable facial geometry as if you need to prompt it every time. The adapter should learn the identity. Do caption transient attributes: clothes, glasses, expression, pose, framing, light, background, and photographic treatment. Automatic captioning can produce a first draft, but inspect every file. Captioners confidently invent eye color, age, ethnicity, camera gear, and emotions.

Use one caption format consistently. Place a same-name text file beside each image if that is the dataset convention selected in AI Toolkit. Confirm the tool is actually reading those sidecars in the training log; a perfect caption folder ignored by the loader is an uncaptioned dataset.

A FLUX.2 Klein starting configuration

Start from the current official FLUX.2 Klein Base example in AI Toolkit. The table below is a conservative first experiment for a 20–30 image character set, not a preset that wins every subject. Keep the official model-specific optimizer, scheduler, and target modules unless the repository says otherwise.

Effective batch size, gradient accumulation, caching, and optimizer state have a large memory impact. Adjust those with the tool’s current memory guidance rather than copying a config from a different architecture. Change one meaningful variable per run and name the experiment accordingly: klein4b-r16-lr5e5-v01 is more useful thanfinal_final_good.

Run, sample, and save checkpoints

Before the full run, train for enough steps to create one preview and one checkpoint. Confirm that loss is finite, images and captions are being counted, samples use the expected seed, and output files land on persistent storage. This ten-minute check is the cheapest failure you will ever buy on a cloud GPU.

Use four to six fixed sample prompts throughout training. Keep seeds, dimensions, guidance, sampler, and negative prompt stable. Include one easy portrait, one side angle, one full-body scene, one different hairstyle or outfit, one difficult light, and one prompt that does not contain common training backgrounds.

Watch the previews as a sequence, not individually. The useful checkpoint is often the first one that carries identity into unseen conditions while still obeying the prompt. Past that point, resemblance may look stronger because the adapter is pulling every scene toward the training set.

Save four things together after every experiment:

• The LoRA checkpoint and exact base-model identifier.
• The complete training configuration and tool commit.
• The dataset manifest or hashes, without redistributing private source images.
• The fixed validation prompts, seeds, images, and a short decision note.

Validate instead of admiring one cherry-pick

A good LoRA should add identity and give control back to the prompt. Test at several adapter strengths—such as 0.6, 0.8, and 1.0—using the same seeds. If the LoRA only resembles the subject at an extreme strength that destroys clothing, lighting, or composition, training has not really succeeded.

Compare checkpoints in a contact sheet with labels hidden, then reveal them after scoring. Keep the earliest checkpoint that meets the brief. Training longer because the loss is still moving is how a useful adapter becomes a costume it cannot take off.

Training a Z-Image LoRA instead

If Z-Image is your target ecosystem, use the official Diffusers train_dreambooth_lora_z_image.py workflow and the non-distilled Z-Image Base checkpoint documented with it. Do not assume the FLUX.2 target modules, text-encoder behavior, or memory flags map across by changing a model path.

1. Install the Diffusers example dependencies from the Z-Image DreamBooth README.
2. Run accelerate config for the actual GPU or cloud instance.
3. Point the script at Z-Image Base and your captioned instance-data directory.
4. Begin from the README’s current learning-rate, precision, and memory recipe.
5. Add fixed validation prompts and save intermediate LoRA checkpoints.
6. Test the adapter with the same Z-Image pipeline and prompt language used in production.

The data principles do carry across: varied images, explicit transient captions, holdout conditions, and early checkpoints. The architecture-specific training knobs do not. Read the model card and training script at the pinned revision you actually run, especially before commercial use.

Export and test in ComfyUI

Export the selected adapter as a .safetensors file with a name that includes the base family and version. Copy it into the ComfyUI LoRA model folder expected by your installation, refresh model paths, and load it after the matching base model. A LoRA appearing in the menu does not prove compatibility.

Build a clean acceptance graph before adding production tricks:

1. Matching base checkpoint, text encoders, and VAE.
2. One LoRA loader with separate, visible strength controls where supported.
3. Fixed sampler settings appropriate to that base.
4. Six validation prompts and seeds saved in the workflow.
5. No face detailer, style LoRA, ControlNet, upscale, or post-processing.

Generate a contact sheet at several weights, then add your normal production nodes one at a time. If identity collapses only after a style LoRA or detailer is restored, retraining the character is the wrong fix. You have found a workflow conflict.

Preserve the clean graph with the adapter. ComfyUI embeds workflow metadata in supported image outputs, which makes a known-good sample far more useful than a loose file named “best settings.”

Diagnose failures, consent, and rights

More steps are the last fix, not the first. Dataset repair has the highest leverage: remove duplicates, rebalance angles, correct captions, and replace low-quality or filtered images. Then change one training parameter.

Finally, document consent and permitted uses beside the model. Train fictional characters you own or adults who explicitly agreed to this use. Do not create identity adapters from minors, private people, former partners, leaked images, or a public figure scraped from the web. Do not use the output to impersonate a person, bypass platform rules, or imply their endorsement. A technically successful LoRA can still be a project you had no right to make.