Understanding Prompting for Stable diffusion in ComfyUI
Vložit
- čas přidán 6. 06. 2024
- A deep dive into how Clip Text Encode works in ComfyUI for Stable Diffusion, analyzing tokens, conditioning, and prompt engineering best practices.
In this video, we take a deep dive into how the ClipTextEncode node works in ComfyUI for Stable Diffusion. We look at what happens behind the scenes when text gets tokenized, converted into a condition tensor, and passed to the diffusion model.
First, we use Python debugging to inspect the tokens - we see how the text gets split into start tokens, word tokens, and stop tokens in batches of 77. This helps us understand optimal prompt lengths.
Next, we examine the condition tensor, seeing how it is a multidimensional representation of the text data. We look at the default 32-bit float precision and how it impacts conditioning.
[CORRECTION]
There is a small correction about the 32-bits float precision and it's well explained by Reddit user u/adhd_ceo in this post here: bit.ly/3tvkzw9
Finally, we explore techniques like ConditioningConcat, ConditioningAverage, and ConditioningCombine to isolate prompt elements and improve image generation. We see how separating color elements into different text encodes can help reduce color bleeding issues.
Overall, this video gives you a deeper understanding of how text conditioning works in Stable Diffusion inside of ComfyUI Web Interface so you can craft better prompts and use conditioning nodes more effectively.
I appreciate if you can like and share the video if it was helpful.
Subscribe for more content soon!
[SUPPORT THE CHANNEL]
Patreon: bit.ly/44js1Xx
Paypal: bit.ly/45lJsIg
[Resources]
Blog Post: / 95377687
[SOCIAL MEDIA]
CZcams Channel: bit.ly/47OterT
Twitter X: bit.ly/3ReP9D3
[PREVIOUS VIDEOS]
ComfyUI End of Year Updates: • End of Year ComfyUI Up...
Custom Nodes: • Create Your Own Custom...
SDXL Turbo Gradio App: • How to Use My SDXL Tur...
SDXL Turbo: • How to Use SDXL Turbo ...
Python API for ComfyUI: • Building a Python API ...
Introduction to Gradio: • Introduction to Python...
Timestamps:
00:00:00 Introduction
00:01:11 Loading the default workflow
00:02:05 Adding a breakpoint
00:03:40 Using the Python Debugger
00:05:28 Analyzing the CLIP object
00:08:22 Analyzing Tokens
00:09:52 Prompt and word weights
00:12:12 Tokens of a batch of 77
00:13:07 Long Text Prompt
00:15:36 The WebUI Error Message
00:17:20 Conditioning Tensors
00:18:51 Torch.float32 [Correction: Check Reddit link in video description]
00:19:39 ConditioningConcat
00:20:14 ConditioningAverage
00:20:27 ConditioningCombine
00:21:46 Examples
00:22:46 Example of well trained model
00:23:40 Bleeding of colors
00:24:57 ConditioningConcat
00:26:04 ConditioningAverage
00:26:45 ConditioningCombine
00:28:22 Conclusion
00:29:18 Thank you for watching
00:29:31 I will see you in the next one
Tags:
stable diffusion, ComfyUI, cliptextencode, stablediffusion, clip text encode, python debugger, tokens, conditioning, prompt engineering, conditioning tensors, conditioningconcat, conditioningcombine, conditioningaverage
Hashtags:
#stablediffusion #cliptextencode #python #tokens #conditioning #promptengineering #comfyui
I thought it was great, especially because you also explain how the AI deals with the prompts. I'd love to see more
Thank you, @SaschaFuchs! I appreciate your positive feedback. I'll definitely aim to create similar content in the future.
This channel is perfect for me, I could watch for hours.
Thank you, @HurdRandy! I appreciate your kind words.
That is an amazing tutorial and approach to learn SD, well done!
Thank you very much, @lhovav! Appreciate the support.
Thank you for sharing this in-depth technical exploration which unveals the secrete happened behind UI.
Thank you, @WalidDingsdale! I'm glad you found the technical details helpful.
This was awesome! Watched to the last second🎉 Great pace and explanation of CLIP node. Goes beyond comfyui to really show what and how prompts work. Looking forward to more in depth videos like this.
Thank you, @salmanmunawar1! I'm glad the pace was right and explanation helpful. I'll definitely continue to create similar content in the future. Thank you for the positive feedback.
I got into AI imaging about three weeks back with fooocus, A1111, koyha ss then invokeAi and comfyui. All have similar concepts under the hood. There is so much terminology you come across it leads to a lot of trial and error, mystery and unexpected results, it’s frustrating. From this video I now know how prompts are parsed. What causes color bleeding, or what are tokens…there is a lot of information here without any filler or repetitive content. Worth multiple re-watches :) I hope you continue to be demystify the image generation process. Looking forward to your next video lesson.
Yes, exploring and experiencing these stable diffusion user interfaces is great, as each provides its own benefits. I appreciate the support.
I can see you doubled up your subscription since my last visit. Well done! And a Happy New Year.
Thank you, @banchew3983! Your continued support is greatly appreciated! Thank you and Happy New Year!
Very nice primer on prompting, thank you
Thank you, @glibsonoran, for you continuous support!
Excellent video! Thanks very much.
Glad you liked it, @henrischomacker6097!
Exactly what I was looking for, thank you.
Thank you, @sidewinder979! I'm glad the content matched what you were looking for.
Very nice deepdive on clip text, thank you very much, I think I get clip text a little better now :D
Thank you, @ferniclestix! I'm glad the video was helpful. Appreciate all your hard work bringing rich knowledge to the community.
Keep it up, always interesting/educational videos.
Thank you, @Utoko! I'm glad this video was helpful. I was worried it was too technical.
@@CodeCraftersCorner I am sure it is for many people, that being said that is why I enjoy your channel a lot, where you get real understanding and not just the effect.
Hope your channel can grow and enough people which value it. gl
Thank you so much, @Utoko! Your kind words and support mean a lot. Appreciate your well wishes for the channel's growth.
Thank you for this video. This is great information.
Thank you for watching, @gabrieldaley5881! Glad the video was helpful.
float32 does NOT hold 32 significant digits! The number signifies the amount of *bits* used to store the whole float: the sign, mantissa *and* exponent. Since mantissa is defined as 23 bits, this gives only 6 to 9 significant digits to go around.
Thank you for the clarification, @MoraFermi! I appreciate your input on clarifying the precision of float32. There's a correction in the video description to ensure accurate information.
Awesome tutorial, thanks for this!
You're very welcome!
Exactly what i needed, thanks a lot
Thank you, @yvann_mp4! Happy to help.
really good explanation, thx!
Earned a subscriber this video is super clean thank you.
Thanks for the sub @astr010!
great explanation, althhough got very technical at some points but in conclusion I came to understand how prompting works for stable diffusion and why on civit ai there are different prompt guidelines given beneath diff models and why I never get good results with using just prompts :D
Thank you, @PallaviChauhan91! I'm glad you found the explanation helpful, even with the technical details.
A highly enlightening look under the ComfyUI hood. Much appreciated! Now I'm off to see if there happen to be any "token count" nodes out there or if I'm going to have to write one myself, just to make life easier.
Thank you, @kpr2! I'm glad you found the video interesting. Good luck with the token counter.
You got my attention now you have my curiosity.. Thank you
Thank you, @SezginAhmetKaratas! Glad you found the content interesting!
Super useful bravo
Thank you, @romanbulatnikov7157! Glad the video was helpful.
damn that video was tight. bless you man.
Thank you, @Painfroi!
Somewhere I learned to use the word " BREAK " along the way so my prompt looks like this " 1girl, 26 years Old, white beach hat, BREAK blonde, BREAK green eyes," and this works. Great explanation !!
Thank you, @randymonteith1660! I'm glad you found the explanation helpful and appreciate your input the the word "BREAK". I'll look into it and do some testing.
Isn't break a A1111 thing to separate conds into isolated chunks? There is probably a node for it in comfy, just like combine is a replacement for AND.
This explanation got me curious on what would happen if I average hat with the color, etc, and then concat all. Gotta test that 1.
Update: the word BREAK did not work in ComfyUI for me. Thanks @finald1316, for clarifying that it's for A1111. I tried a few times and thought i was using the word the wrong way. I am not aware of such a node. If I come across one, I will update.
The ConditioningAverage can be used to merge two or more concepts like making a hybrid horse + crocodile. Definitely give it a try.
This information is pure gold, and I very much appreciate your clear and concise style! 🙏🏻
Thank you, @ArielTavori! I'm glad to hear that you found the information valuable.
great!! i loved it
Thank you! Glad you liked it.
Thank you!!
You're welcome, @francaleu7777! Thank you for watching!
Very useful and well explained! I now feel like I have a MUCH better understanding of how prompting works, though I also understood it depends a lot on how individual models were trained. Makes me wish there was a way to query a model to know what kind of prompts it prefers.
Thank you, @Ken1171Designs! You're right, individual models can have their own preferences. I like the idea of querying a model for its preferred prompts. I appreciate your feedback. Thank you for watching.
@@CodeCraftersCorner The way I handle this is to keep a copy of the example prompts for the model, so I use them as a starting point. However, that alone is not enough. I also copy the recommended settings to get results similar to the example ones. This is a lot of work, so automated the task with a Python scraper, but they changed the site to use dynamic pages, making the process harder. That's why I wish there was a way to simply query the models to know the kind of prompt and the recommended params. I wonder if that could be added to the models as metadata?
It might be possible but I am not sure how that will affect the training process. The way I handle it is to manually set the settings / prompts a first time and save the workflow in JSON format. The next time, I simply load the workflow and continue from there. It is not perfect but at least I am only entering the settings once.
@@CodeCraftersCorner The problem with JSON is that we can only save 1 set of prompts and params that fit 1 model. The moment we change models, we may need a completely different set.
When I mentioned adding this info to the model as metadata, I meant adding it after the training as extra info, similar to how PNGs have reserved space for metadata that is used for all sorts of things. I know models may not have such things, but I leave it as suggestion to add a metadata section to AI models for this purpose. I am sure people will find many uses for it. :)
Good suggestion! For now, I am using the Note node to hold any additional prompts or information before saving the workflow.
@5:30 chapter: From my understanding, setting the last clip layer to -1 is _not_ removing the last layer. The issue is that there are like three ways to represent the issue that this is trying to resolve, and it's different between ComfyUI and A1111, which are themselves different from what someone sitting down and looking at the problem from a pure UI perspective would probably do. ComfyUI is using the value as a direct index.
With the caveat that this is from memory, what happens is that there's an array of layers - we'll say an array of [A, B, C, D] as a simple illustration - and we want to make it so that the system disregards a certain number of layers at the end.
In most programming languages (including Python), arrays are indexed starting at 0. That is, array index 0 will get you A, then array index 1 will get you B, index 2 will get you C, and index 3 will get you D. We want to drop D, but we don't want to require that the user know how many layers there are. In Python, you can do this by using negative numbers to index from the _end_ of the array instead. So -1 is D, -2 is C, -3 is B, and -4 is A.
All well and good, but the node is specifying what the last layer is, not the first layer getting dropped (or the number of layers getting dropped). So if you set it to -1, you're saying that the last layer that the system will use is -1 (D), which means you're not actually dropping any layers. What you need to do is set it to -2 (C), so that C is the last layer used, and D is discarded, if you want to drop the last layer.
Thank you, @David-id6jw for providing a detailed explanation and sharing with the community.
super, thanks🥰
Thank you very much, @murphylanga147! Appreciate the support!
Thanks!
very good
Thank you, @alencg2518!
thanks!
Thank you for watching!
This is very useful. I had a lot of troubles with clothes, I was asking black, white or any other colors and it persists to be green, now with this combine maybe I can fix it
Thank you, @KodandocomFaria! I'm glad to hear that the video was helpful for you. When experimenting with different conditioning nodes, consider using weights in your prompts. It can make a significant difference in addressing color consistency issues. From my testing, it is not 100% guarantee as each model is trained differently. Best of luck!
this is a great video with very well explained concepts. i finally understood a little of what´s behind prompting. Thank you! now i see old Automatic 1111 had a limit of 75 tokens per prompt. so in thath case it never got to generate a batch of 77 tokens right?
Thank you @fernandomasotto! I am not sure how Automatic1111 was doing it. It just means that batches of 77 tokens are passed.
thank you for your great effort to explain ist, amazing. I just tried your prompt and added "blue jeans, black shirt, red sneaker". Randomly it ignored one or two of the elements. I don't understand why.
Thank you for trying out the prompt, @muc2810! Different models can react differently to prompts. Also, the same model can be trained more on a particular term. For example, if during training, the model was exposed to blue jeans more than red jeans, the model will react more to blue jeans than red ones. It's best to experiment by adjusting or removing certain terms and see the influence on the final output. Play with the prompt weights and try various conditioning nodes. You can also try inpainting, image manipulation, post processing to optimize the results.
Is there a way to use PDB to prevent mat1 and mat2 not matching, tensor numbers not same ,or need half got float etc.? Somehow correct the size errors before they error out. I also keep seeing the clip missing: ['clip_l.logit_scale', 'clip_l.transformer.text_projection.weight'] error while loading comfyui.
Hello @offmybach! The error is more like a warning. The content of some checkpoints are different which shows the warnings but using it should work just fine. There an issue log here: bit.ly/43yQnhk
what is it that you use to mark and highlight everything on your screen?
Thank you for watching, @edwardwilliams2564! I use ZoomIt from Microsoft to annotate on the screen. It's a small utility which allows zooming and drawing on screen. It does not have a highlighter mode but you can use the free draw option to get a similar effect.
Thank you
Happy to help!
Can you explain how the 'clip set last layer' node works?
Yes, @mehradbayat9665! In neural networks, there are input and hidden layers which are responsible for receiving input data and converting it into vector representations which machine can understand. Let's say, the input data is text. The text gets pass to the input layer which outputs a representation of the data. This is then passed on to hidden layers (can be many). Let's say, we have 3 hidden layers. If we do not set the clip set last layer, the data will go from input layer to all 3 hidden layers and then gets outputted. If we set the clip set last layer to -2, the last hidden layer will be skipped (input to hidden layer 1 and 2 only). Hidden layers perform computations using weights and biases. In general, less hidden layers means less intricate detail about the input data is captured. It's best to experiment as each model will give different results.
@@CodeCraftersCorner Does clip training perform back-propagation? If the -2 layer is set, is this assumed?
From what I can see in the code base, it does not explicitly show backpropagation. the clip_layer method is specifically setting the layer index. If you want to take a look at the code base, you can go to ComfyUI portable folder > ComfyUI > comfy > sd.py > lines 88 to 151. There's the CLIP class and the method clip_layer.
By using Combine average with weight 1.0 you just gave a weight of nearly zero to another part of prompt. This is why green eyes don't influence the picture. No green eyes so no green color bleeding.
To me this video doesn't look correct, but I will recheck on my own. I'm sceptical still
Thank you, @lalayblog, for catching that mistake! You're absolutely right about the weight of 1.0 affecting only one prompt. I was so focused on the "white beach hat" that I did not realize I forgot to change the weight. I have rerun the experiment with the same checkpoint, prompts, seed, cfg and sampler with a weight of 0.0, 0.25, 0.5, 0.75 and 1.0. I have posted the results here bit.ly/3H0doPy.
I learned so much, thank you for all your skill and hard work making this. I got to say, in the SD 1.5 images, she looks more like 40 years old. I wonder if that is because you use the word "Lady" As an American I would only use Lady with older women. I don't know what bias the Models have however conversationally, Girl would be up to 25 or so, Women is older, and lady or old lady even older. I would doubt many on this list would work en.wikipedia.org/wiki/Category:Slang_terms_for_women
Thank you, @othoapproto9603! I appreciate your kind words and feedback. You raise an interesting point and your observations on the words influencing the models are valid. I must give this a try. It's fascinating how the language choices can impact AI's interpretation. Thank you for the link!
girl in most models in a teenager in sd/sdxl. so you should avoid it if you don't really want a very young woman.
Thanks for the input, @gameplayfirst6548! I appreciate your advice on the models' ages.