Seurat's Le Chahut, close enough to see the encoding. Step back and it becomes a dancer. The VAE is the step back, and it is the only part of your pipeline that ever touches an actual pixel.
Seurat's Le Chahut, close enough to see the encoding. Step back and it becomes a dancer. The VAE is the step back, and it is the only part of your pipeline that ever touches an actual pixel.
Everything the diffusion model does happens in a compressed code. The decoder turns that code into an image, and it is where washed colours, plastic skin and black outputs come from.
You get a generation back and the pose is right, the outfit is right, the lighting you asked for is there, and the skin looks like a vinyl doll. Colours sit a half step grey. Fine hair goes to mush. So you go change the checkpoint, because the checkpoint is what people blame.
The checkpoint may have had nothing to do with it. The last piece of software that touched that picture was the decoder, and most people running these tools have never chosen one on purpose.
Latent diffusion does not denoise pixels. That was the whole idea behind it, because denoising a 1024 by 1024 image directly is enormously expensive. Instead an autoencoder compresses the image into a small numerical grid, the diffusion model does all of its work inside that grid, and a decoder expands the grid back out to pixels at the end.
The original Stable Diffusion autoencoder is called kl-f8. The f8 means it shrinks each side of the image by a factor of eight, so a 1024 by 1024 picture becomes a 128 by 128 grid. That grid holds 4 channels, which works out to roughly 48 times lossy compression. SDXL later retrained the same architecture from scratch, so it is a different set of weights doing the identical job at the identical ratio. Every SD1.5 and SDXL image you have ever made came out through that funnel.
Forty eight to one is a lot to throw away. It is enough for the model to compose a scene. It is not enough to keep a fine texture perfectly intact, which is why eyelashes, fabric weave and text have been historically painful in this generation of models.
Channel count is the single most useful number to know about any VAE, because it tells you how much of the original image survives the round trip.
| Autoencoder | Latent channels | Compression | Notes |
|---|---|---|---|
| SD kl-f8 (1.x, 2.x) | 4 | 48x | Carries a bright spot artifact that gets worse at higher input resolutions |
| SDXL VAE (0.9 and 1.0) | 4 | 48x | Same architecture, retrained from scratch, no bright spot |
| SD3 VAE | 16 | 12x | Adds a shift factor alongside the usual scaling factor, drops the quant conv layers |
| FLUX.1 VAE | 16 | 12x | Same config as SD3, higher decoding quality |
| FLUX.2 VAE | 32 | 6x | Smallest compression factor of any of them, normalisation baked into the checkpoint |
Going from 4 channels to 16 is why Flux and SD3 hold small text, fine lace and individual hairs so much better than SDXL ever did at the same resolution. It has very little to do with the transformer and a great deal to do with the funnel being four times wider.
This is also the one number you cannot change on your own. The diffusion model was trained against a specific latent space. A 16 channel VAE bolted onto a 4 channel model does not produce a better image, it produces noise, because the two halves are speaking different languages.
If you have ever run SDXL and received a solid black image or a screen full of coloured garbage, this is almost certainly what happened.
The original SDXL VAE produces NaNs in fp16 because its internal activation values are too large for half precision to hold. The numbers overflow, the overflow propagates, and what comes out the other end is not an image.
A community finetune called sdxl-vae-fp16-fix solves it by keeping the final output the same while scaling down the weights and biases inside the network so nothing overflows. It is 83.7 million parameters, MIT licensed, and running roughly 300,797 downloads a month at the time of writing, which tells you how common the problem is.
Quality cost is close to nothing. Measured against the original running in fp32, the fixed version scores 0.056 on LPIPS against 0.055, and 0.73 on SSIM against 0.73. If you are on SDXL in half precision and you have not swapped this in, do it before you change anything else.
One other decoder is worth knowing by name, because most SD1.5 workflows quietly load it: vae-ft-mse-840000-ema-pruned. The number in the filename is the training step count.
Stability finetuned the original kl-f8 decoder twice. The first pass, ft-EMA, kept the original loss configuration of L1 plus LPIPS. The second, ft-MSE, resumed from that checkpoint and ran another 280,000 steps to reach a total of 840,001, this time with a loss weighted toward mean squared error at MSE plus 0.1 times LPIPS. The training set was a one to one mix of LAION-Aesthetics and an unreleased LAION-Humans subset, chosen specifically to make faces reconstruct better.
Measured on COCO 2017, ft-MSE against the original: rFID 4.70 against 4.99, PSNR 24.5 against 23.4, SSIM 0.71 against 0.69, PSIM 0.92 against 1.01. Every one of those is an improvement, and the entire cost is loading one extra file. The licence is MIT.
There is a real trade between the two finetunes. ft-MSE gives smoother output. ft-EMA keeps more high frequency texture. If you shoot portraits where skin should read as skin and not as porcelain, try ft-EMA before you assume the smoother one is the better one.
Four practical rules come out of all of this.
Match the decoder to the architecture, always. SD1.5 VAEs go on SD1.5 models, SDXL VAEs go on SDXL models, and Flux ships its own. There is no cross architecture upgrade path.
Most checkpoints have a VAE baked in already, and it is frequently a bad one that the merger inherited from somewhere and never thought about. If your colours look grey and desaturated across every prompt and every seed, that baked decoder is the first suspect, not your prompt.
If you get a black image on SDXL, it is the fp16 overflow. Load the fixed VAE or force the decoder to run in fp32.
And when you want to know whether a VAE is doing anything for you, test it the only way that proves anything: fix the seed, fix the prompt, generate once, then decode the identical latent through both decoders and put the two results side by side at 100 percent zoom. Look at hair, eyelashes, fabric and any text. That is where the difference lives, and it is invisible at thumbnail size.
The VAE is the smallest lever in the entire stack, and it gets blamed for a lot of things it did not do.
A decoder cannot fix composition. It cannot fix anatomy. It cannot fix a prompt that was vague about lighting. If your image is wrong in a way you could describe in words to another person, the VAE is not your problem and swapping it will waste an afternoon. The visible effects are narrow: colour cast, fine texture, and the specific failure modes described above.
More channels is not free either. A 16 or 32 channel latent costs more memory and more compute for every step of every generation, and on modest hardware that trade is genuinely arguable. The reason people running Flux need more VRAM than people running SDXL is not entirely the transformer.
And there is a real risk in treating VAE swapping as tuning. Every extra encode and decode cycle degrades an image slightly, so a workflow that round trips through the decoder four times to apply four different tweaks will look worse than one that does the work in latent space and decodes once. The decoder is a door, not a filter. Walk through it as few times as you can.