A text encoder is a stack of layers too. Skip enough of them and you stop reading the whole onion.
A text encoder is a stack of layers too. Skip enough of them and you stop reading the whole onion.
A layer-skipping trick every 2023 anime tutorial swears by was tuned for a text encoder most current checkpoints don't train around anymore.
Somebody in a Discord told you to set Clip Skip to 2 for "better anime results," so you did, once, years ago, and never touched it again. Every checkpoint since has inherited that number. Some of them looked great. Your newer SDXL and Flux downloads have looked a little flatter, a little less obedient to the prompt, and you chalked it up to the checkpoint being mid.
Your checkpoint was probably fine. The setting you dragged over from your old SD 1.5 folder was built for a model those newer files don't share a text encoder lineage with.
CLIP, the text encoder that turns your prompt into numbers the image model can use, is not one flat calculation. It is a stack of transformer layers, each one building a more specific reading of the text on top of the layer before it. Early layers catch broad concepts, later layers refine them into something closer to a finished interpretation. Clip skip decides how many of those final layers your prompt actually passes through before that reading gets handed off to the image model.
In ComfyUI this is the CLIPSetLastLayer node, and its one input is stop_at_clip_layer, a value from -24 to -1. Minus one is the default and means every layer runs. Drop it further and you are cutting the encoder off earlier, feeding the image model a coarser, less fully-processed version of your prompt. Automatic1111's Clip Skip field is the same idea counted from the other direction: 1 means nothing is skipped, 2 means the last layer is dropped. On an SD 1.x encoder, which runs 12 layers, a Clip Skip of 2 means your prompt's final read stops at layer 10 instead of layer 12.
That number did not come out of nowhere, and it is not superstition. It traces to a specific model: the NovelAI Stable Diffusion 1.5 finetune that leaked in 2022. That checkpoint was trained with Clip Skip 2 baked into its process, and it became the ancestor of an enormous share of the anime and booru-style checkpoints that followed. Every finetune built on top of it, or merged with something that was built on top of it, inherited a text encoder that had learned to expect a slightly earlier stopping point. Set Clip Skip to 2 on one of those models and you are matching the render to how it was actually trained. That is a real, earned recommendation, not folklore.
Trouble is, that recommendation outlived the lineage it was written for. It got copied into model cards, tutorials, and default configs long after most new checkpoints stopped sharing that ancestry, and it kept getting passed down as a general anime setting rather than a NovelAI-specific one.
SDXL was trained at Clip Skip 1, meaning the full text encoder stack was used during training rather than a truncated one, and that value was effectively locked in by the training process itself. Push it to 2 on a base SDXL checkpoint and you are asking the model to work from a prompt reading it never learned to expect, which is exactly the flatter, less-responsive behavior a lot of people write off as "the checkpoint just isn't that good." Illustrious, one of the more widely used SDXL-based anime checkpoints, is a clean example of the generational split: it was trained at Clip Skip 1, so the old "2 for anime" habit that still circulates in some tutorials and model cards actively works against a model built specifically for anime-style output.
Flux complicates things further, in a way that makes the setting even less useful to fiddle with. Flux does not lean on a single CLIP encoder to carry your whole prompt the way SD 1.5 and SDXL do. It reads your text with two encoders in parallel: T5-XXL handles the actual sequence of tokens and carries most of what your prompt means, while the CLIP-L encoder only contributes a single pooled, global embedding rather than a token-by-token reading. A layer-skipping dial aimed at CLIP-L is adjusting the encoder that was never doing the heavy lifting in the first place, which is why its effect on a Flux render is described as negligible rather than absent. If a Flux generation looks off, the text encoder depth setting is close to the bottom of the list of places to look.
| Checkpoint lineage | Trained clip skip | What to set |
|---|---|---|
| NovelAI-derived SD 1.5 anime/booru finetunes | 2 | 2 (or -2 in ComfyUI). This is the one case where the old advice is correct. |
| Base SDXL and SDXL-based finetunes (e.g. Illustrious) | 1 | 1 (or -1 in ComfyUI). Leave it alone. |
| Flux Dev and Flux-based checkpoints | 1, and largely moot | 1 (or -1). CLIP-L is pooled-only, so this dial barely reaches the render either way. |
There is a second layer to this, and it is the kind of thing that makes two people compare notes and get confused. Not every interface even applies Clip Skip to newer architectures. Some tools historically left SDXL and other post-1.5 models alone regardless of what the slider said, on the assumption that the setting only meant something for the older encoder generation, while other tools apply it uniformly across every model family you load. That means the exact same slider position can do nothing on one setup and actively degrade output on another, and the only way to know which situation you are in is to test a checkpoint you know well at both values and actually compare the pair, rather than trusting that the number carries over between programs.
If you train or use LoRAs, the same caution applies one level down. A LoRA is trained against whatever clip skip value was active during its own training run, and that value does not always match the base checkpoint's. When a LoRA you trust starts looking subtly wrong, checking its training config for the clip skip it was built at is worth doing before you touch anything else.
Here is the honest counterpoint: for a lot of everyday generation, Clip Skip is a small dial, not a decisive one. On a checkpoint that matches its trained value, moving it by one step tends to shift mood and specificity rather than break the image outright, and plenty of usable renders come out on either setting. It is also not a universal quality knob, there is no version of it that makes a model "smarter." It only ever changes how much of the text encoder's reasoning is allowed to reach the image model, and whether that helps or hurts depends entirely on what the checkpoint in front of you was trained to expect. Treat mismatches as a small tax on prompt fidelity worth correcting, not a catastrophe worth chasing across every generation you've ever made.
None of this costs anything to fix. Know which family your checkpoint actually descends from, set the number to match, and stop letting a setting from a 2022 leak quietly steer models that were never trained around it.