Auto Lipsync in Blender with Parrot Lipsync
Vložit
- čas přidán 24. 07. 2024
- In this video I demonstrate my new addon for Blender that lets you automatically generate lip sync key frames from spoken audio tracks.
This addon uses OpenAI's Whisper technology to do the translation, so it requires an internet connection and some additional setup on your end. But once that's done you can quickly generate lipsync tracks in most languages for your animations (but currently not east Asian languages like Chinese, Japanese or Korean).
Get it on Blender Market:
blendermarket.com/products/pa...
Get it on Gumroad:
blackears.gumroad.com/l/parro...
Get it on Github:
github.com/blackears/parrotLi...
Help support my videos!
Ko-fi://ko-fi.com/Y8Y43J6OB
Patreon: / markmckay
Ffmpeg:
ffmpeg.org//
0:00 Intro
0:37 Installation
2:33 Lipsyncing to an Action
11:51 Lipsync to NLA Strips
14:08 Lipsync for 2D and Grease Pencil
27:06 Editing the Phoneme Table
31:48 Outro
32:17 Twinkletoes - Animation
32:40 I'll tell you how the Sun Rose - Animation
33:36 The Raven - Animation - Krátké a kreslené filmy
Woaaah, this is gonna save me a lot of time for a music video I've been planning to do. Thanks!
can you help me, did you worked on this ? i have an issue with installation the zip file into my blender system, it says installed but i cant finde them. thanks
Same issue
Did you find any solution of this ?
This looks really promising. Looks like the raw whisper output may need some smoothing or something though - alot to twichiness
Whisper does not give out exact times and sometimes is bad at predicting the beginning of words, so some clean up is needed. But it gets most of the syllables in the right place. Hopefully it will improve in future updates.
im trying to do this with some dinosaur models ive got but i dont know how to make facial mouth expressions for it
You need to make a good mouth rig and then shape the mouth. Rigging can get pretty advanced, but you can get a decent one by using the face controls with Rigify. You can even subdivide the mouth bones in rigify to get extra controls.
I put together a basic mouth animation video:
czcams.com/video/htSUzvC3drI/video.html
You can also look at this. It uses a shape key technique which is harder to set up but which can give you better results:
czcams.com/video/tmAUQVpvgKU/video.html
I've a question, i've the rig done with rigify right, does it automaticly detect the bones around the mouth and setup the keywords/letters?
I mean, for example when i do lipsync, i do it with shapekeys, so i make each shape of the mouth needed, do i need to do this aswell with this addon or will it automaticly generate those poses to be able to animate the mouth for the lipsync?
Appreciate the answer!
Parrot will use the timing info from the dialog tp automatically copy pose actions into a final action track. So all the pose keys you use need to affect the same object that will have the final action.
Blender can be a bit confusing with this. You can think of an action an just being a list of properties that are keyed to change over time. The action itself contains no info about which particular object has those properties. That is determined by assigning an action to the object. So, for example, you can create an action for a rotation property and then assign that action to as many objects as you want. Those objects will then all rotate with the same animation. One drawback with this method is that an action cannot animate the properties of multiple objects.
I'm assuming that you want to have your body rig animate your shape keys. The easiest way to do this is to create extra bones for each of your mouth positions and then set drivers on them that drive your shape keys. You can then create the pose library using your driver mouth bones. Another method you could try is adding custom properties to your rig and drive your shape keys that way.
As long as Parrot can find your source pose actions, it can put them together into a lip sync animation action. If you haven't watched it already, the 2D lip sync section shows how to get Parrot to work with using poses not in the pose library.
I just purchased the app last 3 days I dunno is there is any option of lowering the speed of mouth motion, apart that the addon is really great
I've been thinking about that myself. Best solution I've come up with is just to make the mouth positions under exaggerated. You might even want to leave out some of the less important syllables. The addon is just key framing every syllable, so it really comes down to the mouth shapes. Maybe someone wit more lip syncing experience could give some good advice.
@@GalaxyPedlar I have some experience. A person's mouth doesn't form every phenom of every word that they say. Try this test: Rest your jaw on your fist and start speaking, so that your head goes up instead of jaw goes down. Feel how often your jaw opens, how long it stays open, etc. We move it less than the number of syllables we make and sometimes even less than the number of words we say! If we animate movement for every syllable, it looks like a flapping mouth.
I think syllable dropping would be the way to do it.
Purchased the add on and the first try it worked fine. Everytime i run it now the output is messed up. Where the soundfile is say 300 frames, the addon puts it in 20 frames. Pretty useless even if i try to scale a strip down. Is there something im missing, it bakes a 10 second file in like 1 second output :(
If you go to the Github page, you can post an issue. Please include a file that demonstrates the problem you're having. There might be something wrong with this particular file - have you tried it on a new file? Are there any error messages printed out?
@@GalaxyPedlar Thanks for the reply, I reckon it glitched when i tried it in another file this problem didnt occur on "the same" model. Happy with the add on, really speeds things up. Might have a go at editing the list though. I feel there are too many that cause the mouth to move unnaturally fast. But thats each to their own and also depends on how realistic you want to go.
@@Simskin2 I'm looking into ways to improve that. One thing that seems to work is reducing the expression of the mouth poses
@@GalaxyPedlar Yeah thats what I thought, first gonna have some fun with it, and perhaps you already made some templates in the meantime. I guess animations/photorealistic/cartoon/2D prefer a different amount. Anyhows it works and saves so much time. Carry on!
@@GalaxyPedlar We don't want smaller movements, rather, fewer movements. One can go back and delete frames, and also experiment with not registering some phonemes.
Hi SIr, maybe you can help me shortly on this ? I managed to install the ffmpeg and the python syntax codes. Now i faced an issue where the zip file is not found ? I click install ( Blender 3.6 or 4.0) navigate to the zip folder and double click the file. the list on the left stay the same and the info in the bottom line says this modul () is isntalled...but i cant find it on the list, do you know what i di d wrong ? would be awesome if you can help me on this. looks like a operating system issue ? i use blender 4.0 on a 64x system... thanks sir
if the addon is not to see in the List after installed it, means there is somethign wrong with the steps before ? so wrong ffmpeg codec pack or wrong python codes, right ? need help please. i dont understand where i need to look at to fix this. ciao sir this is awesome addon !
in the windows folder app data. user. blender foundation, in both folder 3.6 and 4.0 is the addon folder parrotlipsync...wht the hell is going on that blender cant see it in the system ?! oh my god
i tried run the py script into blender scripting envoirnment, but only i can see is a IF string that navigates to blender 2.92 path with i dont have if its important. i dont know. i think its just a default syntax right. so im finished try to bring my character alife, hope you will read this and can help me. would be nice for us 3D artist. thanks
This addon requires Blender 4.0. The setup is more complicated than it is for a typical addon. It sounds like you're having trouble installing the addon itself. You should be able to click the Install button from the addons page and browse to where you downloaded the Parrot Lipsync addon. Click on it and it should appear in the list of available addons. If there are too many addons, you can type the word parrot in the search area and that should trim down the list. If you still can't see it, try opening the console window by selecting Window/Toggle System Console from the main Blender menu. That will open a window that may print out an error message describing what is going wrong.
@@GalaxyPedlar so happy you replied :D thank you sir. the info says: Modules Installed () from 'C:\\Users\\"NAME"\\Downloads\\parrot2024\\parrotLipsync-master.zip' into 'C:\\Users\\"NAME"\\AppData\\Roaming\\Blender Foundation\\Blender\\4.0\\scripts\\addons'
awesome is this better than rhubarb?
Please answer ASAP, I really want to buy it. :)
1.Can it be used on multiple characters? To make some kind of dialogue in the same blender file?
2. I'm working in grease pencil. Does it work if the character isn't rigged?
3. Do you have a platform where we can chat faster?
At the moment, if you want to use it on multiple characters, you will need to create two copies of the vocal track and change the starting and ending offsets to match each character's dialog. You would then run Parrot twice on each audio track. You can just copy and paste your existing audio track several times and use Blender's knife tool in the video editor to cut the strips into the right sized peices.
The other way to do it is to render out your track to a single action and then duplicate the action and delete the keys that you don't need for each character.
Parrot requires you to have actions on your target character that make the different mouth positions. The easiest way to do that is with a rig, at least for the mouth area. But you can use whatever mouth pose actions you want. Just remember that in Blender actions can only apply to a single object which is why rigs are used - you can create actions for the rig and then the rig can manipulate other objects like meshes.
I haven't set up a Discord yet, but you can post here or on the Github page which might be better.
@@GalaxyPedlar so then, can I have 2 parrots open in THE SAME FILE and each one guiding a different mouth without them needing to be rigged? :)
@@soringolub5420 You only need to install the addon once. Just run it once on each audio track that you want to generate a lip sync for. Parrot works by combining mouth pose actions, so however you set up your mouth poses should work, as long as they're all on the same object. I'd recommend using a rig, but one isn't strictly necessary.
@@soringolub5420 You will need to set up the mouth actions separately for each character before running Parrot. Right now there is no easy way to switch between characters in the same file.
@@GalaxyPedlar Ok, I get it, it makes sense. I will try it today. Thank you for your work!
eu instalei tudo,mas deu erro
If you're getting an error, please post it on the github page under Issues.
aaaaaaaaaaahhh cooodes :(
you should really remake this tutorial for us dummies who don't understand the first thing about coding. Or finding an easier installation process would be amazing.
@@isaacavacado7744 If you watch the 1.0.5 video, I've simplified the install a lot. You'll still have to install ffmpeg this way, though, since there is no installer for it.
You also don't need to mess with the phoneme table. It comes with one by default and it works well for most languages.
Woah! this is gonna be cool to try! Do you have a discord?
No I don't. You can post to Github, though, or here.
@@GalaxyPedlar Do u think you'd be able to send me a blend file of one of the animations you made using this addon? i'd like to test it out myself before purchasing the addon
@@elihodges You can go to the Github page and download the project. There are example files in one of the directories.