怎么选择模型？
Canny models
Use the Canny ControlNet to copy the composition of an image.
The Canny preprocessor detects edges in the control image. The Canny control model then conditions the denoising process to generate images with those edges.
diffusers_xl_canny_full (推荐, 速度比较慢, 但效果最好.)
diffusers_xl_canny_mid
diffusers_xl_canny_small
kohya_controllllite_xl_canny_anime
kohya_controllllite_xl_canny
sai_xl_canny_128lora
sai_xl_canny_256lora
t2i-adapter_xl_canny
t2i-adapter_diffusers_xl_canny
Use diffusers_xl_canny_full if you are okay with its large size and lower speed.
Use kohya_controllllite_xl_canny if you need a small and faster model and can accept a slight change in style.
Use sai_xl_canny_128lora for a reasonable file size while changing the style less.
The control weight parameter is critical to generating good images. Most models need it to be lower than 1.
Depth models
Use the ControlNet Depth model to copy the composition of an image. The usage is similar to Canny but the result is different.
Here are the depth models we are going to study.
- diffusers_xl_depth_full
- diffusers_xl_depth_mid
- diffusers_xl_depth_small
- kohya_controllllite_xl_depth_anime
- kohya_controllllite_xl_depth
- sai_xl_depth_128lora
- sai_xl_depth_256lora
- sargezt_xl_depth
- sargezt_xl_depth_faid_vidit
- sargezt_xl_depth_zeed
- t2i-adapter_diffusers_xl_depth_midas
- t2i-adapter_diffusers_xl_depth_zoe
A depth control model uses a depth map to condition a Stable Diffusion model to generate an image that follows the depth information.A depth map can be extracted from an image using a preprocessor or created from scratch.
diffusers_xl_depth, sai_xl_depth, and t2i-adapter_diffusers_xl_depth models perform well despite their size differences. All are safe choices.
Recolor models
Use the recolor models to color an back-and-white photo.
- sai_xl_recolor_128lora
- sai_xl_recolor_256lora
Both the 128 and 256 Recolor Control-Lora work well.
Use the recolor_luminance preprocessor because it produces a brighter image matching human perception.
Be careful in crafting the prompt and the negative prompt. It can have a big effect on recoloring. Use these SDXL style prompts as your starting point.
You don’t need to use a refiner.
Blur models
Use the Blur model to recover a blurry image.
- kohya_controllllite_xl_blur_anime
- kohya_controllllite_xl_blur
Alternatively, you can use blur_gaussian preprocessor to blur a clear image for testing.
Of course, some image details are lost in the blur, so you should not expect to recover the same image.
IP-adapter
The Image Prompt Adapter (IP-adapter) lets you use an image prompt like MidJourney. Let’s use the original example from the ControlNet extension to illustrate what it does.
OpenPose models
The OpenPose ControlNet model is for copying a human pose but the outfit, background and anything else.
Here are the OpenPose models available.
- kohya_controllllite_xl_openpose_anime
- kohya_controllllite_xl_openpose_anime_v2
- t2i-adapter_xl_openpose
- t2i-adapter_diffusers_xl_openpose
- thibaud_xl_openpose
- thibaud_xl_openpose_256lora
[Major Update] sd-webui-controlnet 1.1.400 – Official writeup of SDXL ControlNet models for WebUI.
Stabilityai/control-lora – An overview of Stability AI’s Control LoRA models.
kohya-ss/controlnet-lllite – Model Card of ControlNet-LLLite.
tencent-ailab/IP-Adapter – GitHub page of the Image Prompt adapter.
