MULTIPLE CHARACTERS In ONE IMAGE WITH CONTROLNET & LATENT COUPLE! THIS IS SO FUN!
Vložit
- čas přidán 2. 03. 2023
- Recently a brand new extension for Stable Diffusion was released called Latent Couple, which allows you to determine specific zones of an image and attribute it a prompt, making it possible to generate multiple characters in different styles in 1 style generation, no Inpainting required at all! And when combining the extension to ControlNET and the Composable LoRA extension you can generate multiple different characters in different styles and in different positions in one single image generation! This is so cool and so powerful! So in this video, I will show you how to install and use the Couple Latent extension and how to use it in combination with ControlNet and LatentCoupleHelper to get the best results possible! So let's go!
Did you manage to generate multiple characters in 1 image? Let me know in the comments!
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
SOCIAL MEDIA LINKS!
✨ Support my work on Patreon: / aitrepreneur
⚔️ Join the Discord server: bit.ly/aitdiscord
🧠 My Second Channel THE MAKER LAIR: bit.ly/themakerlair
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Runpod: bit.ly/runpodAi
Latent Couple Extension: github.com/opparco/stable-dif...
Composable Lora: github.com/opparco/stable-dif...
Latent Couple Helper : github.com/Zuntan03/LatentCou...
divisions=1:1,1:3,1:3,1:3 positions=0:0,0:0,0:1,0:2 weights=0.2,0.8,0.8,0.8 end at step=50
All ControlNet Videos: • ControlNet
My previous ControlNet video: • GET PERFECT HANDS With...
NEXT-GEN MULTI-CONTROLNET INPAINTING: • NEXT-GEN MULTI-CONTROL...
CHARACTER TURNAROUND In Stable Diffusion: • CHARACTER TURNAROUND I...
EASY POSING FOR CONTROLNET : • EASY POSING FOR CONTRO...
3D Posing With ControlNet: • 3D POSING For PERFECT ...
My first ControlNet video: • NEXT-GEN NEW IMG2IMG I...
Special thanks to Royal Emperor:
- Merlin Kauffman
- Totoro
Thank you so much for your support on Patreon! You are truly a glory to behold! Your generosity is immense, and it means the world to me. Thank you for helping me keep the lights on and the content flowing. Thank you very much!
#stablediffusion #controlnet #aiart #stablediffusiontutorial
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
WATCH MY MOST POPULAR VIDEOS:
RECOMMENDED WATCHING - My "Stable Diffusion" Playlist:
►► bit.ly/stablediffusion
RECOMMENDED WATCHING - My "Tutorial" Playlist:
►► bit.ly/TuTPlaylist
Disclosure: Bear in mind that some of the links in this post are affiliate links and if you go through them to make a purchase I will earn a commission. Keep in mind that I link these companies and their products because of their quality and not because of the commission I receive from your purchases. The decision is yours, and whether or not you decide to buy something is completely up to you.
Its crazy how fast Stable DIffusion outscales all Paid services. Thats the power of the people
Literally every day I master one aspect of controlnet and another appears. This space ceases to amaze me with the daily advances
Master?? That's a bit of an exaggeration, no?
I get paid to do both prompt engineering and Training Models for bigger companies for concept art, so I have to learn each extension and maximize its usage. It can only be considered an exaggeration based on how fast the technology updates. For example, this video hasn’t been out for 24 hours, and there is already something called Multidiffusion Region Control which is an extra add on to this that allows you to sketch a mask in each section rather than using rectangular boxes/ratios. I spent the past 6 hours learning how it works, and what limitations are and I’m in a lot of communities getting updates about this stuff. Anyway, if that was your focus in my comment, you missed the point of my awe in how fast all of this stuff keeps updating
@@depthbyvfx9604 which communities if you don’t mind? I’d like to master as much as I can and maybe in the future get a job in this
This is just insane, each day there is something new, when is this even gonna stop 😳Thanks as always for the tutorials!
try taking personal photography and throwing it into a program like this with img2img, it's mindblowing what can be generated. dude I took some personal photography that took days to shoot along the erie canal. threw them into img2img and was blown away with what it came up with with the right prompts and checkpoints.
Wow, this is really fantastic! You could essentially create comic book panels with ease this way, with a separate prompt for each panel. Thanks for yet another great video!
Maybe yeah, would be interesting to try!
That's an interesting idea in general. Like I wonder if anyone has tried the ControlNet with straight lines, but instead of using it for a room/building, lay out panels for a comic book prompt? Separate from Latent Couple, I wonder if that would work? Or I guess canny probably would if not.
The only problem with this is that you maybe get up to 300 dpi with this technique, which is absolutely horrible resolution for a manga panel. You can probably get around it with upscaling and inpainting, but then you hit a wall if your story contains original characters (because of consistency).
This could be solved with loras, but I feel like you'd have to train a lot of them.
I really can't wait for elite to come out as an extension, as it might be able to solve the consistency problem.
@@pladselsker8340 Hey, what is that elite extension?
You're probably still best served by generating each panel separately for now. Say you do six generations of each panel- you get to mix and match the best for each panel. If you try to generate them all at once, each generation will take significantly longer and the odds of all the panels being exactly what you want are honestly pretty low. There's definitely interesting potential, though- and definitely worth experimenting. It seems particularly promising with a style where one panel blends seamlessly into the next instead of having gutters.
OK, so I just spent like two hours getting this to work only to produce horrible abominations, and here are two very important tips:
1) *If you are generating monsters fusing together:* The first subprompt (before the first AND) will be applied to the whole image (if you use divisions like in this video). This means that if you leave the settings like they are in this video and enter "a man AND a woman", you will smear the man over the whole image and then denoise the woman into the left side of the image, and do nothing for the right side (you entered only 2 subprompts). _You need as many subprompts as there are divisions. And you need to pay attention to the ratios of the divisions._
2) *If you are generating a person who is cut in half and generating the other person on their other half:* Increase the width of the image. None of the models I tested were able to properly generate two full persons like this with the default 512 width. I assume that's because the models were trained on 512 images, meaning they try to adjust the size of a person to 512. If you halve this 512 width, you are not letting the model complete a full human.
+1: You can leverage latent coupling with img2img and controlnets to better guide generation to what you are trying to achieve.
but the best part is making horrible abominations depending how you look at it 😅
yeah it doesnt work for me neither, it just mashes the loras into one...
I had problem 2) solved it by increasing the width of my image to 1024 while keeping the height at 512. Now it produces two different characters as prompted. Thank you!
I just keep generating one person who is a mix of both characteristics... I have Latent Couple enabled, but not sure what's going on. Any advice?
@@Gins. If you divide the picture to 1x100%, 1x50 left side, and 1x50 right side then you need to enter something like this:
forest background AND black man AND white man
You also need to set the aspect ratio to be wider. This will create a wide picture of a black man on the left and a white man on the right, with a forest in the background.
It’s scary how fast this entire space is improving
I was still working on learning blender posing
It's amazing!
@@F5alconsHouse I think ill just skip this and download mdoels from other users xD that will safe a lot of time and i never used blender so yeah
The word you're looking for is "exciting."
This video is literally ALREADY out of date, now you can draw colored masks in whatever shapes you want instead of being stuck with rectangles.
HELLO HUMANS! Thank you for watching & do NOT forget to LIKE and SUBSCRIBE For More Ai Updates. Thx
please can you post which version of stable diffusion, wed UI, lora.. your using as I just get errors on lora and Dream booth. ? or could you give us and install order with version's used so we can replicate your install.
What if I installed the latest latent couple with composable lora, and no latent couple section appeared in txt2img? Only the composable Lora did
@@anastasiaklyuch2746 same here, any solutions to this problem yet?
@@anastasiaklyuch2746 never mind I got it, Open cmd in sd root dir then paste this line below
git apply --ignore-whitespace extensions/stable-diffusion-webui-two-shot/cfg_denoised_callback-ea9bd9fc.patch
restart sd not just UI
@@zeeshanzaffar1435 It worked! Thank you, my heroic technomancer!
This was such a good explanation of how this works. I've seen other tutorials on this before but none that actually explained it like yours did. I am going to have a lot of fun with this now.
HOLY CRAP!!! Another HUGE advance. Now I can say a character has blue eyes without everything else being blue. Plus ALL the other things I can do. Fantastic!!!
Yup it's really cool!
Thank you for the video! Exactly what I have been looking for a long time! 😊
The possibilities are insane ! Thx for the tutorial.
Glad you like it!
Getting ready for more insane content!
omg just yesterday I thought to myself that something like this would be amazing. And there it is already. How could I miss this video =D ?
Instant sub. I needed this.
You are awesome! Thats exactly what I needed! Thank you so much!
Beautifully explained!
Wow ... my mind continues to get blown with how fast powerful tools are being created. Makes me really curious about there we will be by the end of the year!
Man, your videos are Incredibles. Thank you!!
Thanks !!! I can see those extension will be helpful for doing Comic panel , more precise t- shirt - merch and even book cover mockup direct inside SD and A1111. Awesome !
This is so cool! thanks for sharing!
What a time to be alive!!!! Thanks for this 😍
Don't forget to squeeze your paper!
@@Aitrepreneur
Wow so incredible ❤❤
Tried around with this a few days ago and was surprised by how well it worked. Did not know about the Latent Couple Helper though. Makes things a lot easier.
Such power 🤯
Finally got this to work. Some checkpoints really don't like to render more than one person without face issues (that not even higres nor fix faces will fix), amplified by using loras.
Different checkpoint model and reduced lora strength, fixed it. Good video :)
I hope in future it will respect the reduced height more. Especially people will just be cut off or have their heads cut off, lol. Somehow SD still tries to put a full person in there, instead of shrinking it. to respect the area.
I'll definitely need to play around with this tomorrow
Thanks for this video.
This + Composable LoRA could make really great images.
Absolutely!
Wow! So powerful.
thank you~! many inspiration !
Thanks for the tutorial man .... 🔥🔥
No problem 👍
just got into all this ai art generation stuff and this was very useful, have subcribr
suuuuuuuuper cool again!!!!
The Stable Diffusion space is evolving faster than I can keep up with it. Or so fast that I can't learn the previous new features and get good at them before something else comes out.
Thing is once something comes out, and I watch an older video of it to catch up, there's been another video that has updated the previous feature. I kind of wish it would slow down a little bit, I still got to go to work in a few hours. 😆
Yeah I feel you :)
Heck I'm still learning about merging checkpoints.
Oh man, I really understand you. I don't have time to learn in practice one function, 3 weeks later there's another, better one coming out. Or even UI elements can become slightly different or move somewhere else in some cases 😁
@@SkyGeekWave Yeah, at this rate, by the time I catch up to where it is now, Stable Diffusion may have already replace the president.
it's growing very fast. I remember not even understanding how to make multiple characters generate and thought it was impossible
Really cool. I'm gonna need a minute to digest this information. It blows my mind how far this technology has come in such a short time.
true!
ikr i didnt even finish last tutorial
unfortunately the installation doesn't work for me. Compsable Lora tab is there, latent couple is not. latent couple is also no longer selectable in the extentions, via URL installation I get an error message that it already exists. everything is up to date. the folder is in extention folder as it should. I restarted the web ui and also in the browser. any ideas?
I did a git pull origin master and restart the webui and it appeared.
This is cool... that said, I feel like this at the back end of a painted rgb mask would be the next step.
i am going to try this.... definitely xD
Very useful, thank you very much ! By the way, is there something similar for comfyui ?
Nice!
Holy diffusion
This is huge. Thanks for all your hard work. I never regret having the bell on.
Awesome! Thank you! ;)
There's a pull request on the repo that allows you to draw masks (with multiple colours) instead of messing around with calculating partitions.
I tried it. It basically solves composition problems almost entirely.
Can you share repo with me, thank you
Hey, thanks a lot for your videos they are really top notch, I have request, can you please make a video on upscaling in stable diffusion, It would great since there is no video online which explains everything on upscaling.
We are almost there.
The ultimate version Of Stable Diffusion is almost Here.
It will be a Blender Addon that will combine the recently released Blender Skeleton for MULTI-CONTROLNET that our AI overlord talked about.
Combined with the next version of this which will allow us to assign a Prompt, Hypernetworks and Multi-Controlnets to each Skeleton and or "Control Meshs" and the Background.
And once Text To 3D, AI Animation and Images to 3D are also inevitably implemented as Blender Addons The fusion of the 2D and 3D Workflows will be Complete.
And with it The full democratization of animation.
It Will be Glorious and at the rate we are going It will be here Sooner than we realize.
Blender is a program the vast majority of people won't be using. Those same types of tools will be included with WebUi etc. -- In my opinion.
hey great explanations !, but how did you manage to make the characters interact with each other ? so that they touch or look at each other ?
Yes, you made a video on this, and the explanation was super easy to follow, as expected, thanks so much!
Would you ever do a video on the merge block weighted gui extension? The extension allows for more in-depth control in merging models than the stock 'Checkpoint Merge' ui thats in the base Automatic 1111 and it has pretty great potential.
I saw it, I need to try it out first
Thx!
Nice work! Do you think it would be possible to take s picture of a room and clear out all the existing furniture with a prompt that only an empty room remains? And then add custom furnitures in custom places with the latent couple.
Its up and running for auto now?? epic!
Really cool. Don't have to do a huge inpaint area on a completed image.
looking forward to an installation tutorial on comfy UI! I'm a node nerd
Really very good explained!! Congratulations! You are always able to make hard things as simple as possible! Thanks a lot
You are welcome!
Amazing, more control!
I feel like you could already do this with controlNet and photoshop though, but I'm not saying no to a workflow improvement :3
Great video again, keep the streak going man!
I think the tools are advancing but getting quite complex in a way so hope some people can streamline controll net and coupls soon, but this is barely year 1! so there is alot of progress ahead
...wait for Multidiffusion. Veeeery promising!
I agree (this new tool makes sense to me, but only because I did table-based layouts for websites back in the day lol). But I bet it'll get there pretty soon. Leonardo and Mage (and probably other sites) seem like they're trying to wrap up functionality in a more user-friendly package. This is a perfect case where a good UI would help. Like you drag over a region of the image and it pops open a prompt and strength slider. You even could build in a full 3d posing tool with the hands/feet without having to go into another app and exporting/importing stuff. And of course would have a big library of predefined poses, etc.
Another great video! Is there something like this but uses layers like Photoshop?
Do I really need to fit all the prompts for one character in one line then enter, put "AND", then do the other character prompts? or it doesn't matter as long as I put the "AND" word to separate them?
Hello Ai Overlord K, has this extension been replaced by the "Regional Prompter manipulations"? Thanks!
Would be super cool if the LoRA part comes earlier or if there was a chaper division in the video time bar.
This is evolving faster that I can learn it :(
Can the negative prompt be used the same way?
Embeddings seem to be general and is npt prossible to restrict to the specific image zones
Do you depent on Nerdy Rodent in any way?
I shouldn't have watched this before lunch. The pizza made me hungry.
Now I can make a harem book cover!
i watched someone else explain this and i really didnt understand it but this makes way more sense. now i get it. thanks you.
Happy it helped ;)
5:05 _it knows_
I live by the Avenue Q philosophy of the internet.
Do you have to also enable the Composable Lora extension? I didn't see you enable it in the video. Thanks
Been trying to make a half human face with half venom face without any luck, I hope this will finally make it work!
INSANEEEEEEE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Can you add different seeds to go with the prompts? That would produce ultimate consistency
there is Ebsynth Utility too in my extension, is this automatically used ebsynth in our img2img? if it yes, that is awesome
Would be interested to of seen you do an image where characters are interacting. Making dynamic fights is a huge headache right now for me.
Just discovered Stable Diffusion a few days ago which I thought was insane but this is fucking crazy
Can this easily be used with the masking instead of the numbered zones?
Is there any way to use this or any similar tool like Latent Couple Helper in Mac OS?? Thanks in advance for the good quality content
Do you enable the composable lora, didnt work on my setup. Already followed your step all the way, but there's a frame skip you didnt explain anything before generating image.
Question, since you can use AND for the positive prompt, can you also use AND in the negative prompt to give discrete negative details to the individual zones?
I believe someone else mentioned it is a common negative prompt unfortunately
14:33 I have two remaining questions after watching the video. What about LoRAs trained on concepts or characters? Can I put one specific character in one side of the image and another one in the other? I ask because I've tried numerous times without any success, I'm getting the worst imaginable results. Secondly, I suppose this doesn't support textual inversion at all? I have some very clean, very well trained character embeddings but none of the cool new stuff seems to waste time on text inversion anymore. I wonder why? They are still pretty damn powerful, nothing has really changed in that regard. And they are like 1/250th the size. Is there maybe a way to convert an embedding to a LoRA so I can make use utmost use of what I already have? I'm begging you, if you have any information for me, please share. You would not believe how much I've dug for an answer.
I will sum up for ease:
- What about character LoRAs? Can we group together two, three, five specific characters in one prompt/image? I asked because my attempts failed.
- Are text embeddings out of fashion? Why is no new tech supporting them? I see them on the same level as LoRAs
- Can I port my good textual inversions to a LoRA somehow without going through the process of training all over again, lol?
I tried to use Controlnet and Latent couple and it kept fusing my two LORA characters together into a badly rendered hybrid one. Still trying to figure that out myself.
If anyone knows how to use Latent couple in combination with separate LORAs any help would be appreciated.
🤯
man this is crazy. tnx a lot.
Can you please tell how did you get clip skip slider on the txt2img tab?
I have installed with this tutorial, follow step by step. And even so, my Latent Couple dont work, its enabled and i have even tried with this model and settings, looks like its not doing anything with the images. Anyone else have this problem?
seems to be broken
Wanted to ask, how do you make a New Line at the Prompt Text box ? I pressed Enter, Alt Enter, Shift Enter, it doesnt create a new line.
I installed the extension but it does not appear on the UI
Oh boy, does he know?
so the regions are like layers ?
NICE TUTORIAL. Latent cople gives me an error when pressing Visualize. To me works Regional Prompter extension.
Question please. what if I want them to be kissing for example? That division would make them separately and ruin the composition
Nice and looks powerful, but why couldn't they implement photoshop style square region selection tool, like img2img has.
I tried with Chilloutmix model, but it doesn't create 2 characters
I can't seem to get this to work, enabling the extension and generating an image just results in one character with aspects of both prompts merged into one character
The problem is the size of images, if you want to have 2 subjects consider a width of 1024, if you want 3.. consider a width of 1536 and so on because each character takes 512 pixels
So we will come to the place, where we can plan and generate graphic Novels, with consistant Lora /Textual Embedding Characters pretty fast.
Your tutorials are so good.
5:05 "weird position but pretty cool" 🤣DEAD
Well, already you can pose two or more people (in any place you want) and replace them by inpaint? anyway, it's great new tool. Thanks for a video.
hmmm I installed it, but it just won't show in the text2image tab... any ideas?
hey bro, i installed the extensions, yet the latent couple doesnt appear in my SD, do you know anywhere i can seek for trubleshoot?
PD. i see it as installed in the extensions tab so it should be installed
I had problems with runpod, fresh installation of SD 1.5
The latent couple did not load. But after running git pull from the console in runpod, it did work. Maybe this could be of use to others.
Thank you. It solved my problem too after I applied git pull. All extensions now is working.
i keep getting merged characters although I've defined the right zones and included AND. how do i ensure they're separate characters?