CONSISTENT VID2VID WITH ANIMATEDIFF AND COMFYUI
Vložit
- čas přidán 3. 06. 2024
- Get 4 FREE MONTHS of NordVPN: nordvpn.com/enigmatic
Topaz Labs Affiliate: topazlabs.com/ref/2377/
ComfyUI and AnimateDiff Tutorial on consistency in VID2VID.
HOW TO SUPPORT MY CHANNEL
-Support me by joining my Patreon: / enigmatic_e
_________________________________________________________________________
SOCIAL MEDIA
-Join my discord: / discord
-Twitch: / 8bit_e
-Instagram: / enigmatic_e
-Tik Tok: / enigmatic_e
-Twitter: / 8bit_e
- Business Contact: esolomedia@gmail.com
________________________________________________________________________
My PC Specs
GPU: RTX 4090
CPU: 13th Gen Intel(R) Core(TM) i9-13900KF
MEMORY: CORSAIR VENGEANCE 64 GB
Stabilized Models: huggingface.co/manshoety/AD_S...
My Workflow:
mega.nz/file/uFxXCDQJ#eN_laa1...
IP-ADAPTER MODELS:
huggingface.co/h94/IP-Adapter...
CLIP VISION MODELS:
huggingface.co/openai/clip-vi...
huggingface.co/comfyanonymous...
Folders
Clip model goes in the Comfyui/models/clip_vision folder
IPAdapter model goes in the comfyui/custom_nodes/comfyui_ipadapter_plus/models folder
0:00 Intro
0:36 Nord VPN
1:16 Worflow Setup
5:16 FreeU
06:59 IPadapter
10:59 Face Restore
11:45 Controlnets
13:45 Generate
17:21 Upscaling - Zábava
![ComfyUI EP06: Using ControlNet in ComfyUI to Control Results as Desired [Stable Diffusion]](http://i.ytimg.com/vi/B6qilXLls0o/mqdefault.jpg)
Get 4 FREE MONTHS of NordVPN: nordvpn.com/enigmatic
dude! this is exactly what I have been looking for! I love the way you explain and the stuff you are covering. Thank you so much for sharing this valuable info. Subscribed right away!
Thanks for making the time to explain and share your workflow, dude! Super appreciate it!
Great stuff, super helpful, finally got to where I wanted in ComfyUI and vid2vid thanks to you!
Results look great. So many softwares, so little time. Workflow looks nice. Looks like I'll need to deep dive on this at some point.
Dude, you’re the best! Glad to see you have more sponsors and advertisers :) All the best to you and your channel 💪
🙏🏽🙏🏽🙏🏽
Wow great work flow! Thanks for sharing!
Your videos are so helpful. Thank you for sharing your knowledge with us. Gracias!
De nada 👍🏽
You can also skip the LoRas by selecting and CTRL-B, it'll send the signal without the LoRa Loader processing its execution step
Awesome video, thanks so much!
Great tutorial, thank you
I cannot wait to start playing around with this. Putting together a PC build for this reason.
Hell yeah, this is amazing!
Thanks dude, you rule.
Best tute out.
amazing stuff!
i love you. thank you so much
great work man, thank you
👍🏽 no Problem.
Awesome great Video,..learned a lot 🙂
I liked MDMZ ... till I found this channel. Lovely work. Keep it up.
Amazing, thanks a lot.
No problem!
What is the folder where the FaceRestoreModelLoader node models are placed?
Great video! Could you also share the input video and images? Thanks a lot.
Is there a way to see a frame of Ksampler progress before ending? I mean, in A1111 you could enter to output and see batch progress before ending. Thanks!
thank you
What's the learning curve for comfy ui or someone you recommend that teaches comfy ui
thx
where do you run your workflows? do you use your own desktop with what GPUs? or services like runpod etc.?
I use my own gpu.
In the video, your OpenPose preprocessor (titled "DWPose Estimation") has more options than in the workflow, where it's called "OpenPose Pose Recognition", and doesn't have the bbox_detector and post_estimator options. Did you get that preprocessor from a custom node?
Does the workflow not have DWPose?
@@enigmatic_e It does, but it's not the same node I think. The one in the video has more options.
@@abovethevoid653 mmm not sure why it’s different. I wonder if it’s an updated or outdated version maybe.
By the way, I see that everyone places the width and height in external nodes? I change the values directly in the upscale node, does that alter anything?
I would just try it the way you have it setup and see how it looks.
When I try to open your workflow I get a bunch of missing nodes, how do I install/add those on a mac? :/ thanks
ControlNetLoaderAdvanced
CheckpointLoaderSimpleWithNoiseSelect
OpenposePreprocessor
VHS_VideoCombine
LeReS-DepthMapPreprocessor
ADE_AnimateDiffLoaderV1Advanced
IPAdapterModelLoader
IPAdapterApply
PrepImageForClipVision
HEDPreprocessor
FaceRestoreModelLoader
FaceRestoreCFWithModel
VHS_LoadVideo
Integer
amazing
Hey, I get this error and i've been trying my best to troubleshoot to no avail :
Error occurred when executing KSampler:
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Could you explain why you set resolution to 704 in the node of HED Soft-Edge Lines?
I might have been playing around with parameters, it makes the hed resolution higher but can’t remember if it makes much of a difference.
Hello thanks for de video, I have been fighting with this for several days, I removed the background of the character leaving a black background, then I created HED and Openpose, both were perfectly backgroundless, I also added ip2p, when creating the video the character appears perfect but the background is dark with stripes and lights which have nothing to do with the prompt, I want to do it outdoors with sunlight but it is dark like a dark room. (If I bypass animatediff it makes the exact image at the prompt I'm writing.)
You will always have stuff generate in the background. You need to use some external software to remove the background with like a mask or rotoscoping.
Where do you get the safetensors model for CLIP Vision on your IPAdapter? I can't seem to get it to work. Thanks for this video! It's helped a ton.
I just included it in the description. Sorry about that.
Ahhh! You're the best. Thank you!@@enigmatic_e
@@enigmatic_e Have you gotten this before with IPAdapter? I'm thinking my issue isn't CLIP Vision related now, but thank you so much for adding the file to the description!
Error occurred when executing IPAdapterApply:
Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 1280]).
@@SadPanda449 Hey got the same problem, did you find the solution? Thanks
@@enigmatic_e I'm getting the same error as SadPanda... Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
Any way to fix this EniE?
Quick note - I installed the IP Adapter models directly from Comfy, and it put it in the wrong folder. I found the folder and moved the contents over, then it worked.
Overall - phenomenal work, I am running the flow right now. Does anyone know what CRF does? it doesn't seem to be affecting anything.
🙏🏽
I've found that when you're trying to replace a subject with a specific style (vid2vid), using Controlnet ip2p helps with the style transfer, whether it's from IP Adapter, or LORA.
I have a question, about something I haven't tried yet: it would seem that you can use IP Adapter to influence an image's background. Would that be possible in vid2vid?
Is there a way to process 2-3 minute videos? like anything more than 150-200 frames crashes my comfyui. Like is there a way to do in batches maybe? Im already using low aspect ratio and everythhing.
have you tried using lcm?
set the load image cap to 150 on first run then on second set the load image cap to 300 run skip the first 150 then skip first 300 skip ect or break your images up in multiple folders and run each folder
Chingon. I'm noticing more and more people diving into comfyui.
Yeah I feel it’s the best thing at the moment.
Only rich people with expensive GPUs are diving into it.
Part 2 please.
subscribed
Thanks for the tutorial but How can I find "apply IPadapter". I tried for load but I couldn't find it.
You can replace "Apply IPadapter" with "IPadapter Advance" node.
Which model is that on Load Clip model? "model.safetensors" !? where to get it?
You could probably google it. I think that’s I found a few by searching load clip models for comfyui or something like that.
Anyone know a controlnet to prevent colorshift on something like clothing with vid2vid?
I have that error :
Error occurred when executing IPAdapterApplyFaceID:
InsightFace: No face detected.
Any solutions please ? :)
there is a problem these days, when you update the ipadapter, so this workflow doesn`t work anymore with the workflow, do you know how to fix or get the new workflow with updated ipadapter?
never mind!! i follow some of the comments here and found the answer, works great, nice tuto, thanks a lot
I’ll try to upload an updated version today
My animatediff loader is not working, it doesn´t recognize the mm-Stabilized_high.pth that is in the proper folder...
There might be another folder you have to put it. Do you have more than 1 animated diff folder in your custom node folder?
Does animate diff work with SDXL sizes?
it does but sdxl doesnt give the best results at the moment.
any new workflow?
❤
Best in Bizz
I appreciate that. 🙏🏽🙏🏽
How was the 3D video you are uploading created?
www.mixamo.com/
@@enigmatic_e thanks
I am so confused, I am getting this error: Error occurred when executing IPAdapterApply:
Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 768]). Any idea on how to fix this?
Hmm are you missing model for ipadapter?
@@enigmatic_e had the wrong model! its solved now. ty
I have the same issue, i have the model shown in the video. Do you think the CLIP model is causing this? I cant find that one, so I am using clipg
I have the same problem for both models, investigating...
same here
will this work with 11gb of vram?
What PC Specs do you use?
I have an RTX 4090. Sorry I will make sure to put that information on my videos from now on. Thank you!
@@enigmatic_e thank you
Could you adjust your videos volume to be even lower so that only ants can hear it? THanks
I’m recording at a standard volume for video. I try to peak at -6db averaging -12db
Swear u gotta be a genius to understand this stuff lol
Can I do this on stable diffusion
Do you mean in automatic 1111 the webui? If so I would say that it's more limited than comfyui. ComfyUI allows for way more customization.
@@enigmatic_e okay I'll try on ComfyUI
nice, thanks.
👍🏽👍🏽
as young people say... first! _o/
😅
🎉🎉🎉
That UI isn't looking so comfy anymore. How the hell are people keeping track of all these nodes 0.0
😂😂 I guess it takes some getting used to.
Node editor spaghetti is my kryptonite, I commend anyone able to keep track. You also can't argue with the results. Probably the best consistency I have seen yet. Good stuff!@@enigmatic_e
The workflow sharing and reuse is comfy AF tho!!
Its called continuously learning. If it sounds like hard work its because it is 😅…
Is even not that a big workflow 🤷🏿♀️
When it gets to ControlNet, it seems that the DWpose Estimation node is not available? I also get this message:
Error occurred when executing LeReS-DepthMapPreprocessor:
LeresDetector.from_pretrained() missing 1 required positional argument: 'pretrained_model_or_path'
File "C:\----COMFY-UI-APPS+FILES\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\----COMFY-UI-APPS+FILES\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\----COMFY-UI-APPS+FILES\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\----COMFY-UI-APPS+FILES\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux
ode_wrappers\leres.py", line 21, in execute
model = LeresDetector.from_pretrained().to(model_management.get_torch_device())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
anyone got an error at the KSampler part ?
Error occurred when executing KSampler:
Unknown context_schedule 'uniform'.
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI
odes.py", line 1375, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI
odes.py", line 1345, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 346, in motion_sample
latents = wrap_function_to_inject_xformers_bug_info(orig_comfy_sample)(model, noise, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\utils_model.py", line 360, in wrapped_function
return function_to_wrap(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 100, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 713, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 618, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 557, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 154, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch
n\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch
n\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 281, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch
n\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch
n\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 271, in forward
return self.apply_model(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 268, in apply_model
out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 385, in evolved_sampling_function
cond_pred, uncond_pred = sliding_calc_cond_uncond_batch(model, cond, uncond_, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 461, in sliding_calc_cond_uncond_batch
context_windows = get_context_windows(ADGS.params.full_length, ADGS.params.context_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\context.py", line 296, in get_context_windows
raise ValueError(f"Unknown context_schedule '{opts.context_schedule}'.")
I've got the same error (both on macOS and windows)
Have you found how to solve it by any chance..?
Same here! Any luck figuring it out?
fixed it and updated link, check description
fixed it and updated link, check description
thanks for fiixing it bro ! @@enigmatic_e