ELLA - A Powerful Adapter for Complex Stable Diffusion Prompts
Vložit
- čas přidán 10. 04. 2024
- Diffusion models have made incredible strides in text-to-image generation, but they still struggle with dense, complex prompts that involve multiple objects, detailed attributes, and intricate relationships.
Enter ELLA - the Efficient Large Language Model Adapter that's poised to revolutionize how diffusion models handle sophisticated prompts. This ingenious adapter equips text-to-image models with the power of large language models, without the need to train either the U-Net or the language model itself.
In this video, I dive into using ELLA in ComfyUI, and explore how it tackles the limitations of current text encoders like CLIP. Prepare to be amazed as Ishowcase the superior performance of ELLA compared to CLIP conditioning, unlocking a new level of sophistication in text-to-image generation. If you've ever struggled to craft the perfect prompt, this video is a must-watch!
(* oops! Was playing with scheduling in the video and so forgot to switch back to sgm unfiform, but it really doesn’t make much difference. It also turns out they are no longer going to release the SDXL weights.)
However! For SDXL, check this video out!
Pixart Sigma - Like ELLA but for SDXL+ Resolutions in ComfyUI!
• Pixart Sigma - Like EL...
Want to support the channel?
/ nerdyrodent
Links:
ella-diffusion.github.io/
github.com/TencentQQGYLab/ELLA
github.com/ExponentialML/Comf...
huggingface.co/QQGYLab/ELLA/b...
huggingface.co/google/flan-t5...
huggingface.co/Kijai/flan-t5-...
See also: github.com/TencentQQGYLab/Com...
== More Stable Diffusion Related Stuff! ==
* Installing Anaconda for MS Windows Beginners - • Anaconda - Python Inst...
* Installing ComfyUI - • How to Install ComfyUI...
* ComfyUI Workflow Creation Essentials For Beginners - • ComfyUI Workflow Creat...
* Faster Stable Diffusions with the LCM LoRA - • LCM LoRA = Speedy Stab...
* Make an Animated, Talking Avatar - • Create your own animat...
* One Image Gets You a Consistent Character in ANY pose - • Reposer = Consistent S... - Věda a technologie
Finally...I don't need SD3 anymore if this is the case. Who needs to download more SD3 models when this is doing prompt adherence that well...because my disk space is suffering these days.. :D
I took that back. It seems that I need SD3 after all. I cannot get the style I want because of limitation in prompting (like nerdy said in the video). And the prompt adherence is very well only with the raw prompt without including styling prompt. So it is basically useless.
@@Cingku plus they don't release sdxl
SD3 has a wayyy better VAE too. That's the main thing I'm looking forward to
I see small improvements with ELLA but it is not even close to sdxl examples on their github page which they wont release :(
edit: oh wait, seems like I used outdated extension, Will try again
edit: still very tiny improvement
I've been looking forward to this.
That is quite the improvement!
Thanks man for the flow, I am one of your Patreon's now! 😀
Legend!
Absolutely amazing!
Epic!! Brilliant tutorial as always, thanks Nerdy :)
6:38 Love your sense of humor, Nerdy Rodent :D (pretty sure I would have made similar joke either lol).
Also it's amazing that such a simple additional tool on top of the already known one, can make such a huge difference and it shows that these models are actually more powerful that we thought they are. Really impressive new nerdy tech news as of recent 3-6 months to me :)
I can’t wait to see what we get two papers down the line!
Nerdy's always on time 👍
Thank goodness I can write my mini-novel prompts! This is quite a game changer.
This really seems to be getting closer to the ability of the private imagegen models to follow the prompt better.
FINALLY! Stable diffusion is at prompt adherence level of natural language as DALLE!!
Cool video. Much love ❤❤❤❤. Cool video. Am waiting for next video 😊😊😊😊
Thank you so much 🤗
How well does it output text and does it work with artist name like art by Picasso ? Also can it run with 8GB of Vram like normal SDXL models can ? Thanks for the great video, it like getting a sneak peak at SD3 capability.
Very impressive, I need to try this out as soon as possible! :) What's the sigma node doing and what kind of impact does it have? Should it use the same sample and scheduler as the KSampler?
Would love to see this with a control net depth map
I installed all the T5 files linked but keep getting a "T5Tokenizer requires the SentencePiece library but it was not found in your environment." error. I re-downloaded the spiece.model file which I assume is the file in question but still get the error. It could be something I'm doing since I'm new to comfy (still learning how to navigate it).
Edit: managed to fix it somehow, installed and reinstalled ella, also pip installed sentencepiece and installed the other ELLA add-on in comfy, so if anyone has a similar issue one of those might help (can't be more specific as I genuinely have no idea why it happened or how it was fixed)
I wonder if the SentencePiece Library has to be installed prior to installing the customNode? I am having the same issue but didn't try removing Ella's node and reinstalling it, So I will give that a go now.
What worked for me was uninstalling it and then using the Experimental Pip installer to install it from the ComfyUI manager. No clue why that worked.
perfect, just got your patron too
Legend!
How does it too for low steps if we want to combine with LCM Lora?
Unfortunately, this group has stated definitively that they are NOT releasing their SDXL weights. I'm waiting to see what the RPG Dungeon Master group comes up with.
We will also have to see what license they have on that 🫤
Hey thanks for your videos ;) you are awesome! ...
Did you tried it with animediff ? :) do you think it would work? :) maybe even some control-nets would help it with understanding and it can color them correctly :)
t5 model is showing as unidentified even after putting it in right folder , any help , i also placed it in ella-embedd folder
Is there a huggingface space where I can try this out?
A shame it was tencent who came up with this, they stated that sdxl wont be made publicly available, they also didn't release the training process for 1.5. There is a community effort to reverse engineer the training process, hopefully they'll pull it off.
Shame they probably aren't ever going to release the SDXL weights (only the SD 1.5 version)
They have stated definitively that they are NOT releasing the SDXL weights.
Ah yes, I see they said after I’d made the video 😞 Still, maybe someone will do it in the future!
Any reason for not releasing the weight for SDXL?
@@prettyawesomeperson2188 Business reasons... ie they want to make money with the good stuff.
I see it might not be so profitable to release SDXL version of this technique. 1. It produces the anime style of semi-realistic picture alone (flan-t5-encoder-only gives anime). You need to combine it's output conditioning with clip conditioning.
2. SDXL prompt adherence comparable with Ella on sd1.5. I can't expect Ella adherence will be significantly better for SDXL (didn't see claims on that yet).
Ok cool
But, how good is it with cartoonish or anime style?
I'm getting this error "RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'" on a GTX 1660S, could the limited VRAM be the cause?
My guess would be due to the older GPU 😕
@@NerdyRodent I figured that would be it, thank you anyway, I'm sure the requirements would be halved in the future
@@shirwan The GTX1660s is obsolete for AI due to the scarce 6GB, I recommend the RTX3060 12GB which is very economical due to the launch of the RTX4060
now I want chips
NO SDXL release and NO TRAINING CODE either.
I just don't even bother working with such things, as it is just a dead end.
edit: feels like they just released it to create hype for something else that they wanna monetize
Flan-t5-encoder-only-bf16 useless alone because it produces anime style only.
So I agree that without fine tuning to realism it is useless.
I still dont understand HOW ELLA does what it does?
Just tried it. The prompt adherence is good (not as good as SD3), but the quality of SD1.5 is terrible. The SDXL version would probably be a lot better.
How about using it as first pass and then a SDXL model as second pass? The prompt adherence would help a lot.
@@itycagameplays Might be worthwhile.
@@itycagameplays I've been doing AI images for a long time and the idea of using the SD1.5 model to create hundreds of quick images and improve the one I like in SDXL didn't occur to me, thanks man 🤣
🐀 ❤🤘
💜
Can we be seeing this anywhere in automatic1111 or forge sd , i don't want to switch to comfy ui as i am already comfortable with forge
combine this with FreeU and Self-Attention Guidance
Doesn't OpenAI already do this with DallE3?
CHISP!
Omnomnom!
🫶
💜
YOU USED 2 DIFFERENT SCHEDULERS, now you have to redo everything dude))))
Can you please do a more clear instruction on the path. ComfyUI is complicated as it is without giving how you set all this spaghetti up.
So this only works with sd 1.5 models?
So, this thing is profitable for sd1.5 mostly. The only advantage for SDXL would be support for different languages but for the price of huge space consumption on SSD.
@@lalayblog It's working for me with sd1.5 but I prefer sdxl and it just crashes with sdxl.