12.5.1 DreamBooth Personalization

Overview

Stable Diffusion can render almost anything you can describe in English, but it can’t render your specific thing. Ask it for “an African fabric pattern” and you’ll get a plausible-looking African fabric pattern — drawn from the statistical average of what the model saw during training, not from any specific source. There is no prompt you can write to get the pattern you have on disk, because the model has never seen it.

DreamBooth [1] is the technique that fixes this. Pick a rare token (the original paper used sks, others use ohwx or [V]), show the model 5–10 images of your subject paired with prompts that include that token, fine-tune for a few hundred steps with a special loss term that prevents the rest of the model from collapsing, and you end up with a model that responds to sks <class name> by rendering your subject in whatever context the rest of the prompt describes. This lesson does it for African fabric patterns — same dataset as Module 12.1.2 and 12.3.1, now with text control.

The training itself fine-tunes the U-Net inside Stable Diffusion. Doing this naively would update every weight in a 1B-parameter model and require a server-class GPU; LoRA [2] reduces it to a few thousand low-rank parameters in the attention layers, which is small enough to ship as a 3 MB .safetensors file and train on a consumer card in 15–30 minutes.

Learning objectives

1. Explain why a vanilla text-to-image model cannot reference a specific subject and what DreamBooth’s token-binding mechanism adds to fix it.
2. Describe the prior-preservation loss and predict what happens to general capabilities if you train without it.
3. Distinguish Textual Inversion (embedding-only) from LoRA (attention-adapter) fine-tuning and pick the right tool per task.
4. Tune classifier-free guidance scale and predict how it interacts with personalised generation.

Quick start — load the LoRA, generate with sks

Pull the pre-trained LoRA from the GitHub Release, layer it on top of Stable Diffusion v1.5, and use the sks token in any prompt.

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")
pipe.load_lora_weights("models/fabric_lora")     # folder with both downloaded files

prompt = "a beautiful dress made of sks african fabric pattern, fashion photography"
image = pipe(prompt, guidance_scale=7.5).images[0]
image.save("personalized_fabric.png")

The same prompt without the sks token returns a generic African-fabric dress — pleasant, but not the specific pattern. The token does the steering.

Core concepts

Concept 1 — Why “personalised” is structurally different from “fine-tuned”

Fine-tuning a generative model on a new dataset is an old idea: take a pre-trained network, continue training on your data, hope the output distribution shifts toward what you want. It works, mostly, but it has two failure modes that make it the wrong tool for personalisation. First, it requires a lot of data — fine-tuning Stable Diffusion in the usual sense wants thousands of paired image-caption examples, not ten. Second, it catastrophically forgets: training on 1,000 images of fabrics teaches the model your fabrics, but at the cost of degrading every other concept it knew.

DreamBooth solves both problems with two ideas working together. First, token binding: pick a token that is rare in the text encoder’s vocabulary (so it doesn’t already mean something), use it as the subject anchor in every training prompt (a sks african fabric pattern), and let the cross-attention layers learn to associate that token with your visual concept. This is data-efficient because you’re not teaching the model what fabric is — only what this specific fabric looks like, indexed by a token name. Five to ten reference images is enough.

Second, prior preservation: prevent the catastrophic forgetting by sneaking generic class images into the training mix. We’ll get to that in Concept 3. The combined effect is a fine-tune that adds your subject without subtracting anything else — exactly what “personalisation” should mean.

Concept 2 — Same U-Net, same noise prediction, plus text conditioning

DreamBooth doesn’t change the diffusion architecture from Lesson 12.3.1 — it inherits the U-Net, the noise schedule, the predict-noise-at-timestep-t objective. The new piece is text conditioning, threaded through cross-attention layers.

The text encoder is CLIP [3] — same model used in Stable Diffusion’s release configuration. A prompt is tokenized, embedded, and reduced to a sequence of vectors that are fed into the U-Net via cross-attention at multiple resolution levels. The noise-prediction loss now takes a third argument:

L = E_{x, c, ε, t} [ ‖ε - ε_θ(x_t, t, c)‖² ]

where c is the CLIP embedding of the prompt. During training, every training image is paired with a prompt template (a sks african fabric pattern) so the token-to-concept association builds through gradient updates on the cross-attention paths.

Stable Diffusion is itself a latent diffusion model [4] — the noise is added to a compressed latent representation rather than to raw pixels, which is what makes 512×512 generation tractable. DreamBooth fine-tunes on top of this latent diffusion. The mechanics that interest you for personalisation are entirely in the attention layers; the encoder/decoder pair that produces the latent representation stays frozen.

Concept 3 — Prior preservation: the trick that lets you train on 10 images

Fine-tune Stable Diffusion on 10 fabric images for 800 steps without any safeguards and a measurable thing happens: the model gets very good at generating your fabric, and noticeably worse at generating everything else. Ask for “a golden retriever” after such a fine-tune and you may get a fabric-textured golden retriever, or a golden retriever in the same colour palette as your training fabrics, or simply a degraded golden retriever. This is the catastrophic-forgetting problem and it scales with training time.

The DreamBooth fix is structural. Before fine-tuning, generate 100–200 class-level images using the original pre-trained model with a generic class prompt (a fabric pattern). During fine-tuning, mix these generated class images into every training batch alongside your subject images. The loss becomes two terms:

# Instance loss — learn the specific subject from your 10 images
instance_loss = F.mse_loss(noise_pred_instance, noise_instance)

# Prior loss — keep generic class behaviour from the original model
prior_loss = F.mse_loss(noise_pred_prior, noise_prior)

# Combined loss; lambda balances the two (typically ~1.0)
total_loss = instance_loss + lambda_prior * prior_loss

The instance loss pulls the model toward your subject. The prior loss tethers the model to its original behaviour on the broader class. Without the prior term, the only signal the model receives is “your 10 images are the truth” — and it eventually concludes that all fabric is your fabric. With the prior term, the model has to satisfy both constraints, and the only way to do that is to learn the subject as a token-conditioned specialisation that lives alongside the original class capability rather than replacing it.

Concept 4 — Textual Inversion vs LoRA: two ways to be parameter-efficient

The naive DreamBooth implementation fine-tunes all of Stable Diffusion’s U-Net — about 860M parameters. That works but it’s expensive in compute (need a beefy GPU), storage (need to ship hundreds of MB per personalised model), and risk (you can break things you didn’t mean to). Two lightweight alternatives dominate practice.

Textual Inversion [5] freezes the entire model and trains only the embedding vector for a new placeholder token. The token becomes a vector in CLIP embedding space optimised so that, when fed through the frozen U-Net, it produces something that looks like your training images. Output is tiny (~3 KB — literally one vector), training is slow but stable, and the technique is excellent for styles but limited for subjects — there’s only so much information you can pack into a single embedding vector.

LoRA [2] freezes the model weights and adds small low-rank adapter matrices A and B (with rank(AB) ≪ rank(W)) inside the attention layers. Trained output is the adapter matrices themselves — typically 3–50 MB depending on rank. Quality is better than Textual Inversion for subjects because the adapters have more parameters and can modify attention behaviour layer by layer. Two practical advantages: LoRAs compose (you can load multiple at once, e.g. a subject LoRA plus a style LoRA), and they’re stackable across base models (a LoRA trained against SD 1.5 often transfers usably to SD 1.5 variants).

The default for this lesson is LoRA at rank 4 — small enough to train on a 12 GB consumer GPU, large enough to capture the African fabric patterns convincingly.

Exercises

python exercise1_generate.py

MODIFY II.

Three explorations: guidance, style mix, seed variation

exercise2_explore.py — three explorations in one script

bash Copy

python exercise2_explore.py

Goal 1 — Guidance scale comparison. Same prompt at guidance_scale ∈ {1.5, 3.0, 7.5, 12.0, 20.0}. Saves a side-by-side grid.

Goal 2 — Style transfer grid. Render the learned fabric pattern in nine artistic styles (“watercolour”, “oil painting”, “stained glass”, “pixel art”, …).

Goal 3 — Seed variation. Hold the prompt constant, vary the random seed across 6 values. Useful for assessing consistency vs diversity.

What to expect — guidance scale

At guidance_scale = 1.5 you’ll see the model’s prior dominate — generic African fabric without strong sks influence. At 7.5 (default) the prompt drives generation but the model retains compositional naturalness. At 12+ the prompt becomes near-literal: every detail you mentioned is enforced, colours saturate, lighting becomes flat. Most users settle in the 6–10 range; the upper end is for cases where the model won’t listen.

What to expect — style grid

The motifs and colour palette of the learned pattern persist across styles, while the rendering medium changes — “watercolour” softens edges, “stained glass” introduces black outlines, “pixel art” quantises the colours. This is evidence the LoRA has captured the pattern as a style concept, not as a single canonical image.

What to expect — seed variation

Composition changes substantially across seeds: the dress is cut differently, the lighting differs, the pose changes. The pattern style stays consistent because it’s anchored to the sks token, not the random initial noise. If you see the pattern itself shifting between seeds, you’ve under-trained the LoRA (try more steps).

CREATE III.

Train your own personalised LoRA (15-30 min on GPU)

This is the full training pipeline. Two scripts cover the two training approaches; LoRA is the recommended starting point.

exercise3b_train_lora.py — LoRA fine-tuning (recommended) exercise3a_train_textual_inversion.py — Textual Inversion (smaller output, longer training)

Step 1 — Reference images. Place 5–10 images of your subject in training_images/. Quality and variety matter more than quantity: include different lighting, angles, and (for fabrics) folds. All resized to 512×512.

Step 2 — Configuration. The defaults are tuned for African fabric patterns; adjust for your subject:

Hyperparameter	Default	Note
Base model	runwayml/stable-diffusion-v1-5	Best balance of quality and resource use
Instance prompt	a sks african fabric pattern	Replace african fabric pattern with your class noun
Class prompt	a fabric pattern	Used for prior-preservation generation
LoRA rank	4	Trade-off: higher rank = more capacity, more risk of overfitting
Learning rate	1e-4	AdamW; lower if loss diverges
Training steps	800	More for complex subjects, less for simple ones
Prior preservation	enabled, 100 class images	Disable only if you’ve verified the resulting model degrades cleanly

Step 3 — Train.

bash Copy

python exercise3b_train_lora.py

The script generates the 100 class images first (~5 min), then trains the LoRA for 800 steps (~15–30 min on RTX 5070Ti). Sample images get saved every 100 steps in training_progress_lora/ so you can watch the subject emerge.

Issue triage

CUDA OOM — lower TRAIN_BATCH_SIZE from 2 to 1 and increase GRADIENT_ACCUMULATION to 4 to compensate.

Generated images look generic, not your style — the sks token didn’t bind. Check that your training prompts include the token, then try more training steps (1000–1500).

Generated images are repetitive/over-fit to training images — too many training steps for too few reference images. Try fewer steps or more reference images.

Training takes >2 hours — almost certainly running on CPU. Verify with torch.cuda.is_available(). If False, install CUDA-enabled PyTorch.

Loss decreases but quality plateaus — your LoRA rank may be too low. Try rank=8 or rank=16 at the cost of larger output files.

Make it your own

After training, swap the subject — feed in 10 photos of your own object (a sculpture, a mug, a pet), update the class prompt to match (a sculpture, a coffee mug, a dog), and watch the same pipeline learn it. The data-efficiency is the magic; the rest is mechanical.

Summary

Common pitfalls

• Forgetting the subject token in generation prompts. Without sks, you get the model’s prior; the LoRA does nothing.
• Skipping prior preservation. With enough fine-tuning steps you’ll see general concepts degrade visibly.
• Training too many steps on too few images. The output becomes brittle — every generation looks suspiciously like one specific training image.
• Setting guidance_scale too high in pursuit of “stronger” personalisation. Above ~12 you trade naturalness for oversaturated rigidity.
• Mismatched base model. A LoRA trained against SD 1.5 won’t load correctly into SD 2.x or SDXL — the attention architecture differs.

References