It took some doing but I got it working with with my webcam, Windows 10 and the current Vincent rig. Thank you for making the simplified solution for Windows 10 which worked for me right out of the box. I took the time to get this version to see if dlib in conjunction with the shape_predictor_68_face_landmarks dataset produced better tracking results which it did. Much smoother animation IMO. Can't wait to see more from you.
I just wonder how did you install the opencv on blender using windows? I installed opencv in anaconda prompt and it's available to import cv2 directly in the window, but not in the blender, even if I have added the cv2 location to the sys.path.append. Really need some help here...
@@pillowurs3909 i installed cv2 per the instructions in the video....verified which version of python my blender installation was pointing at the from that python\bin directory, ran the command > "python -m pip install opencv-python opencv-contrib-python imutils numpy --user" from an CMD (as Admin) window. Had to restart blender after that to verify the installation as described in the video. Hope that helps.
If you're having troubles with this setup, please see my other video for a simplified process: czcams.com/video/9FBMoUo6vhY/video.html Or a Windows 10 step-by-step guide for this: czcams.com/video/RY_eErKlilw/video.html
Thank you so much for this amazing guide. There are some problems to install dlib in bundled python with Blender on Windows 10 but somehow I managed to install it and finally I have got the same result as you have shown.
Thanks! Your input is pulling me out of the frustration I'm finding myself these days. Do you know if simultaneous multiple cameras are optional? Like for dialog between several characters?
Ok, here are 2 things a changed to make it work on my Win 10 64-bit PC, Blender 2.8.2: In the Python script changed two things: 1. bones = bpy.data.objects["vincent_blenrig"].pose.bones # not the correct line bones = bpy.data.objects["RIG-Vincent"].pose.bones # is the correct line "RIG-Vincent", not "vincent_blenrig". This is what it's called in the current free Blender Vincent model. 2. Added one more line in front of cv2.imshow("Output", image): cv2.namedWindow("Output") # extra line to open the "Output" window before generating the output cv2.imshow("Output", image) Also, to make sure it's not my camera, I edited the script a bit to read the video from a file. Changed self._cap = cv2.VideoCapture(0) to self._cap = cv2.VideoCapture("D:\MyVideos\BTT\TalkFace1.mp4") #my sample video file. Now it's finally works (hurray!!!) but it's extremely slow, though I have a very fast state of the art PC. The playback is like one-two frames per sec. Now I need to figure out how to fix that and how to make it work with my camera feed.
Good work - maybe the Vincent model has changed recently... In the Render properties, under Simplify, set: Viewport - Max Subdivision to 0. It really helps performance!
Sorry man, thanks for a quick reply and tip, but no change. I figured that the problem is the blenrig. When I try to move an object, like a cube, using OpenCV data points from the video, the speed is fine. As soon as I try to move a control/bone in the Vincent's blenrig from Python, everything gets super slow. I would guess it's on Windows only since your video works just fine. So, basically, as soon as I comment out the following section, everything works in real time, 24 fps: bones["head_fk"].rotation_euler[0] = self.smooth_value("h_x", 3, (self.rotation_vector[0] - self.first_angle[0])) / 1 # Up/Down bones["head_fk"].rotation_euler[2] = self.smooth_value("h_y", 3, -(self.rotation_vector[1] - self.first_angle[1])) / 1.5 # Rotate bones["head_fk"].rotation_euler[1] = self.smooth_value("h_z", 3, (self.rotation_vector[2] - self.first_angle[2])) / 1.3 # Left/Right bones["mouth_ctrl"].location[2] = self.smooth_value("m_h", 2, -self.get_range("mouth_height", numpy.linalg.norm(shape[62] - shape[66])) * 0.06 ) bones["mouth_ctrl"].location[0] = self.smooth_value("m_w", 2, (self.get_range("mouth_width", numpy.linalg.norm(shape[54] - shape[48])) - 0.5) * -0.04) bones["brow_ctrl_L"].location[2] = self.smooth_value("b_l", 3, (self.get_range("brow_left", numpy.linalg.norm(shape[19] - shape[27])) -0.5) * 0.04) bones["brow_ctrl_R"].location[2] = self.smooth_value("b_r", 3, (self.get_range("brow_right", numpy.linalg.norm(shape[24] - shape[27])) -0.5) * 0.04) For now I can only use a prerecorded video and set the timelines' playback speed to 1-2 fps for capture, then play it back at normal speed. It's something, but not exact science since you never know what the actual video playback is when the rig is slowing down the video. I am thinking about moving a set of empties when capturing the face motion (hopefully that will work real-time) and then link them to the rig for playback and baking. I guess. A question: is there any way to capture eye movements with OpenCV?
Something is strange with letters and naming of the files. Tried "njuska" (face in Croatian), but no matter what, it does NOT work. Later discovered that some letters or combination of letters of the .mp4 file make problems. For example, alone letters as is: n, t, u, v (for example, n.mp4) does not work, while other letters alone works. Also, those letters may work, depend of the location in the name. I suspect that camera name or whatever may be also important.
@@omc7250 check this video at 3:51. You have to install everything from your Blender's local Python path. The beginning of each line should read something like that: ./python3.7m ........ and the rest. That's for Linux. On Windows you just navigate to Blender's local Python path and run everything starting with python -m ..... and so on. Where python is the name of your executable in bin folder.
Import cv2 does'nt work for me, I'm on mac with python 3.7.4 It says (when I type import cv2) Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'cv2' But I have installed it and everything is in. Anyone can help?
Could anyone help? Trying this on a mac, downloaded and installed the pip and the scripts. I see the capture button, but Blender (2.9) keeps crashing, also when I run it from Terminal. Should I install python 3 instead of the basic 2.7, cause I thought it would run from within Blender.
Can't get it to work. I get some errors in the script. Traceback (most recent call last): File "D:\bu\blender_models\vincent.blend\OpenCVAnim.py", line 38, in File "D:\bu\blender_models\vincent.blend\OpenCVAnim.py", line 30, in register File "C:\Program Files\Blender Foundation\Blender 3.0\3.0\scripts\modules\bpy\utils\__init__.py", line 849, in register_tool raise Exception("Tool %r already exists!" % tool_cls.bl_idname) Exception: Tool 'ui_plus.opencv' already exists! Error: Python script failed, check the message in the system console
Installed fine but whenever my camera launches in blender it just crashes after I start moving... Traceback (most recent call last): File "/home/bob/Desktop/Desktop/vincent.blend/OpenCVAnimOperator.py", line 116, in modal KeyError: 'bpy_prop_collection[key]: key "vincent_blenrig" not found' Any ideas? Thanks for the video
Trying this on macOS 10.13.6 High Sierra and running into problems. First off while, Blender 2.8 has a python3.7 folder, my Terminal says it's version 2.7.13. Later, when I tried the "python -m ensure pip" command I got the message: "Could not import runpy module". Where do I go form here?
Also, do you have to use a webcam built in to your computer? My eventual goal is to take video from a helmet mounted IP webcam as part of a mocap suit.
import cv2 in Blender is giving error, i followed whole video as u described. I GET this in script window Traceback (most recent call last): File "", line 1, in File "E:\SOFTWARES NEW\blender software\blender\2.80\python\lib\site-packages\cv2\__init__.py", line 5, in from .cv2 import * ModuleNotFoundError: No module named 'cv2.cv2' Plz Help
error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. HOW TO DEAL WITH THIS ERROR ??
hello, after run it will display ValueError: WorkSpaceTool.setup(): error with keyword argument "options" - : 'REGISTER' not found in ('KEYMAP_FALLBACK') may i know how to fix it!?
Got this working! Really cool stuff man. Found the same issue with Apple's ARKit True Depth sensor. It slips just enough and is fuzzy enough around the lips that it's not really useful for much beyond what Snap is doing with the tech. Weta Digital uses dots on faces and that'll stop the slippage with a good enough camera and enough light so I'm going to have to try this idea with dots on faces and old fashioned point tracking. but really solid work here. I'll be poking through your code tomorrow for sure.
What about fps on the cam? I remember the ps3 eye worked really good with iPiSoft because it recorded at 60 hertz, iPhone cam records 1080 at 120 fps, instead of real time performance a recorded performance might work better. Also improving lighting might work as well.
Lawrence Whiteside, hey thats a great idea. I currently do a lot of facial animation via ipjonex depth cam but it lacks detail. If I could add a dot based tracking on top of iphonex depth cam tracking it would be really great. Would that be possible? If you are a programmer I would love to work with you to make such an app.
Ty for guide. Really helped me a lot. As Windows user I had some problems 1. Could not correctly install dlib - had to change python files from owned. >Had to install CMake and adding System path czcams.com/video/8_X5Iq9niDE/video.html so dlib could be installed >Even after installing had problems with importing so just copied local python files while keeping bin Folder as it is 2. As @ American Minion Broadcasting Company said had to change some code as vincent model had changed names. 3. Had really low frame rates - Render properties, under Simplify, set: Viewport - Max Subdivision to 0. helped me I might have skipped some steps if someone really need I might post how or what I used
I'm getting a Traceback error when I try to capture. I briefly got an output window that showed a few frames of video, but there were no points on it. Python: Traceback (most recent call last): File "D:\OneDrive\Documents\vincent.blend\OpenCVAnimOperator.py", line 196, in modal cv2.error: OpenCV(4.5.5) :-1: error: (-5:Bad argument) in function 'circle' > Overload resolution failed: > - Can't parse 'center'. Sequence item with index 0 has a wrong type > - Can't parse 'center'. Sequence item with index 0 has a wrong type location: :-1 Any idea what this means or how to resolve it? Looks like a data type error?
Finally It works in my Windows 10 after installing manually dlib and changing the code. Now my problem is that the pop-up window doesn't close when I click in the X, it opens again.
When I follow the video come to the "python -m pip install dlib" step, I encounter a installing error , then I try to use cmake-gui with "Blender Python(3.10.2)" to compile Dlib directly, and it still complains the same error... but the "Original Python(3.10.6)" is OK. I don't know why ...😢
How do you get it to accurately Lip Sync? I use a very similar method for Facial Mocap via the Kinect 2 for Windows and through a Blender addon I found. My issue isn't getting my models to animate, it's after I record the keyframes using the auto keyframing option on Blender. My lip sync moves way to fast after rendering it and creating a actual video file to playback. Plus my PC isn't that powerful GPU wise, and so it's kind of tough to judge whether it's a GPU issue that causes everything to either slow down to a crawl or speed up after rendering. Like do you ever need to clean up your Blender Timeline regarding keyframes you've captured/recorded?
This is a Very informative video. I am using Blender 2.82 with python3 on MacBook Pro. but when I tried to Register and run OpenCVAnimOperator. Py I received error messages. Can you please explain the actual reason of error and probable solution for that? Thanks.
Thank you for amazingly useful tutorial! In Blender 2.83 I faced with error: "vincent_blenrig" not found. It can be fixed if replace it with "RIG-Vincent". Hope it will help somebody.
I ran into problem where I got the error during dlib installation. Apparently it requires cmake to be installed I installed the cmake library before dlib using the following command python -m pip install cmake adding cmake in the multi installation 3rd step just before dlib should also do the job i know i made it understand in a very complicated way. just trying to help. maybe more people ran into this problem. so would be helpful
Hi, I connect it to my blender and it works perfectly the only thing that I still looking for is how to export the animation from it I can’t find the key frames that it makes, thank u
Thank you but I have a problem with the lbfmodel.yaml file. If I use the raw version of your version, I get the message "python script failed, check the message console". After I run the OpenCVAnimOperator.py. It is a syntaxerror "unicode 'unicodeescape' error codec can't decode bytes in position 2 - 3: truncated" I added: r, open and double slashes but nothing worked. So I can't get capture under opencv animation. How can I fix this? However in object mode I see "openCV Animation" but not the button "capture" I followed every step
I just wonder how to install the opencv on blender using windows? I installed opencv in anaconda prompt and it's available to import cv2 directly in the window, but not in the blender, even if I have added the cv2 location to the sys.path.append. Really need some help here...
I was not able to make OpenCv work even after it was installed, it is very complex to understand the installation, does anyone know if they have an addon to install OpenCv?
It would be more benefit and precision if you could just set video path to process so the script would go frame by frame and make it better (and you could fix it too per frame basis).
Wondering, how is OpenCV dlib working so well for you. In my own experiments I find it can get very confused with open mouth animations. What's weird - mouth animation works better in low light conditions. In bright light, as soon as I start talking the mouth landmarks sometimes suddenly jump up and down.
@gadgetWorkbench was able to get it setup with all the downloads, but as soon as the capture starts on MAC, the camera light goes on, then off, then crashes blender, any recommended hardware specs you need for this?
Ok, don't try this on windows, it's just somehow not working at all for me dlib won't install because it can't find cmake, even though I have that installed, with several ways been at this for 2 hours now and it won't want to work
Running on Mac OS, using blender 2.83 running the scripts successfully but once i click on the UI button to start the motion capture blender crashes, i see the webcam turn on (the green light) and it turns off, then a error message appears that Blender quit unexpectedly
hello! i have been facing an issue for an ETERNITY, the python that blender makes available is (frustrating to say the least) so i saw that in 3:19 you had your own custom python interpreter, is there any possible way for me to do that? i have looked at so many forums and videos online, nothing mentions the way you change python interpreters in blender, much appreciated if you help!
Yes, it works with Blenrig. It's specifically coded for the vincent rigged model. You would need to change this: github.com/jkirsons/FacialMotionCapture/blob/master/OpenCVAnimOperator.py#L116 and the bone names under it if you're using it with a different armature.
I keep getting this dang error even though ive installed, uninstalled and reinstalled cv2 about half a dozen times. No luck. >>> import cv2 Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'cv2'
There is some clue why it does not work in Windows 10. After installing all that needed and as described on the video, unable to run script 'OpenCVAnim.py'. Ignoring that, tried to click on 'capture' on new tool: "OpenCVAnimation ->Capture" and got error message: "Traceback (most recent call last): File "D:\Blender Projects\Mocap using py\Vincent_plus_scripts.blend\OpenCVAnimOperator.py", line 84, in modal cv2.error: OpenCV(4.1.1) C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor' location: :-1". Now, D:/... stuff is mine, but "C:/projects/opencv-python/opencv/modules/imgproc/src/color.cpp" is not. Folder 'project' does not exist on my disk, nor any related C++ file as is 'color.cpp'. What did author of this video has installed on his disk that he forgot to tell us? Thank you in advantage.
Well, found OpenCV in 'full glory' on github: github.com/opencv/opencv But, there is now color.cpp and at line 182 says: "CV_Assert(!_src.empty());" - exactly what indicate as error. At least found all that one. BUT (!), even when move all files into newly made folder with color.cpp file: "C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:182" - that is not real folder. Tried to change line 182 in color.cpp, but that does NOT do anything. The same error remains, even now this line of code is changed. I think that pre-compiled version of all that folders is broken or something missing. Also, I am not sure how it is possible to find webcam. I am using Mobius camera with removed microSD (only then it works as webcam when on USB port), and tested in OBS studio. Mobius camera is there ONLY if 'Capture' button is NOT pressed in Blender. That means that it tries to reach camera, which is good sign. OBS studio can't share camera, or better said, if camera is 'busy', then it does not show anything. Remaining question is about drivers in OpenCV - does it support Mobius camera as webcam? Currently I have no other webcam. So, maybe it will work with other webcam, or... not sure... maybe?
@@MilanKarakas I had the same problem Mark. I had to comment out lines 142 - 144: #self._cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.width) #self._cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.height) #self._cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) I think they are incomparable with my web cam,
Not that I know of. It's the dataset that the trained models were created on that is not for commercial use. I suppose you could use video of your own face with annotations provided by a commercial product to train a new model. If the commercial license permitted this...
Is there anyway to just get facial capture data without having it run as a blender app. So that a VA can cam their performance send me the dataset and audio?
You could have them record a video, then on this line: github.com/jkirsons/FacialMotionCapture_v2/blob/master/OpenCVAnimOperator.py#L210 pass the filename instead of using a camera, syntax: docs.opencv.org/3.4/d8/dfe/classcv_1_1VideoCapture.html#a949d90b766ba42a6a93fe23a67785951.
May i Ask your for help with exact something in ICLONE 7? Iclone 7 also have PYTHON inside. But im complete newbie in Python this is dark magic for me... . Having thing like this, can save thousands of hours, to creating Facial Animation. Im goona to learn python soon, i installed pycharm,i installed also Python 3.8 , i installed also cmake (in pycharm), - Im not able to instal DLIB - ppl say, that DLIB only works with Python 3.6 smth. So can you help with this in iclone? Thanks for great video in blender. This is a Free Face Tracker ... and ppl Paying 2-3K $$ for payable thing like this...
Great video , i am dying to get a indie friendly budget or free facial mocap system .... ;) some questions tho . - is this a markerless solution ? - can we get better animations by increasing bones resolutions , i mean adding more bones or something like that ? - is there a way to capture eyes blinking and movements ? - what kind of webcam is needed to do this ? thanks a lot !
Yes, this is markerless. For more bones, you would have to link more facial landmarks to bones. Yes, blinking could be done by the distance between the upper and lower eyelid landmarks. Any webcam will do - just needs a video of your face :)
I've been meaning to figure out how to do this for month, this will be immensely helpful for getting this working for other characters.
Is this work with shape keys?
@@pranshu512 You could probably combine shape keys and drivers to get better results.
It took some doing but I got it working with with my webcam, Windows 10 and the current Vincent rig. Thank you for making the simplified solution for Windows 10 which worked for me right out of the box. I took the time to get this version to see if dlib in conjunction with the shape_predictor_68_face_landmarks dataset produced better tracking results which it did. Much smoother animation IMO. Can't wait to see more from you.
I just wonder how did you install the opencv on blender using windows? I installed opencv in anaconda prompt and it's available to import cv2 directly in the window, but not in the blender, even if I have added the cv2 location to the sys.path.append. Really need some help here...
oh p.s. I use the blender2.8version,hope its not too different?
@@pillowurs3909 i installed cv2 per the instructions in the video....verified which version of python my blender installation was pointing at the from that python\bin directory, ran the command > "python -m pip install opencv-python opencv-contrib-python imutils numpy --user" from an CMD (as Admin) window. Had to restart blender after that to verify the installation as described in the video.
Hope that helps.
If you're having troubles with this setup, please see my other video for a simplified process: czcams.com/video/9FBMoUo6vhY/video.html
Or a Windows 10 step-by-step guide for this: czcams.com/video/RY_eErKlilw/video.html
Thank you so much for this amazing guide. There are some problems to install dlib in bundled python with Blender on Windows 10 but somehow I managed to install it and finally I have got the same result as you have shown.
how did you resolve the dlib problem?
@@kmadisha I work on something similar in 2D but this is much better. I'm having problems installing dlib.
4232.cf/animacion-in-alive-with-obs/
@ChameleonIVCR Thanks this worked for me! :D
And here I thought I finished with all my scripting for my pipeline. Guess ill be knee deep in python again when I switch to my new machine!
very handy that the tutorial is on manjaro
Thanks! Your input is pulling me out of the frustration I'm finding myself these days. Do you know if simultaneous multiple cameras are optional? Like for dialog between several characters?
Wow, thats a great tip. please do more of these.
Ok, here are 2 things a changed to make it work on my Win 10 64-bit PC, Blender 2.8.2:
In the Python script changed two things:
1. bones = bpy.data.objects["vincent_blenrig"].pose.bones # not the correct line
bones = bpy.data.objects["RIG-Vincent"].pose.bones # is the correct line
"RIG-Vincent", not "vincent_blenrig". This is what it's called in the current free Blender Vincent model.
2. Added one more line in front of cv2.imshow("Output", image):
cv2.namedWindow("Output") # extra line to open the "Output" window before generating the output
cv2.imshow("Output", image)
Also, to make sure it's not my camera, I edited the script a bit to read the video from a file. Changed
self._cap = cv2.VideoCapture(0)
to
self._cap = cv2.VideoCapture("D:\MyVideos\BTT\TalkFace1.mp4") #my sample video file.
Now it's finally works (hurray!!!) but it's extremely slow, though I have a very fast state of the art PC. The playback is like one-two frames per sec. Now I need to figure out how to fix that and how to make it work with my camera feed.
Good work - maybe the Vincent model has changed recently... In the Render properties, under Simplify, set: Viewport - Max Subdivision to 0. It really helps performance!
Sorry man, thanks for a quick reply and tip, but no change. I figured that the problem is the blenrig. When I try to move an object, like a cube, using OpenCV data points from the video, the speed is fine. As soon as I try to move a control/bone in the Vincent's blenrig from Python, everything gets super slow. I would guess it's on Windows only since your video works just fine.
So, basically, as soon as I comment out the following section, everything works in real time, 24 fps:
bones["head_fk"].rotation_euler[0] = self.smooth_value("h_x", 3, (self.rotation_vector[0] - self.first_angle[0])) / 1 # Up/Down
bones["head_fk"].rotation_euler[2] = self.smooth_value("h_y", 3, -(self.rotation_vector[1] - self.first_angle[1])) / 1.5 # Rotate
bones["head_fk"].rotation_euler[1] = self.smooth_value("h_z", 3, (self.rotation_vector[2] - self.first_angle[2])) / 1.3 # Left/Right
bones["mouth_ctrl"].location[2] = self.smooth_value("m_h", 2, -self.get_range("mouth_height", numpy.linalg.norm(shape[62] - shape[66])) * 0.06 )
bones["mouth_ctrl"].location[0] = self.smooth_value("m_w", 2, (self.get_range("mouth_width", numpy.linalg.norm(shape[54] - shape[48])) - 0.5) * -0.04)
bones["brow_ctrl_L"].location[2] = self.smooth_value("b_l", 3, (self.get_range("brow_left", numpy.linalg.norm(shape[19] - shape[27])) -0.5) * 0.04)
bones["brow_ctrl_R"].location[2] = self.smooth_value("b_r", 3, (self.get_range("brow_right", numpy.linalg.norm(shape[24] - shape[27])) -0.5) * 0.04)
For now I can only use a prerecorded video and set the timelines' playback speed to 1-2 fps for capture, then play it back at normal speed. It's something, but not exact science since you never know what the actual video playback is when the rig is slowing down the video.
I am thinking about moving a set of empties when capturing the face motion (hopefully that will work real-time) and then link them to the rig for playback and baking. I guess. A question: is there any way to capture eye movements with OpenCV?
Thanks!!! That works. Slow, but works.
Something is strange with letters and naming of the files. Tried "njuska" (face in Croatian), but no matter what, it does NOT work. Later discovered that some letters or combination of letters of the .mp4 file make problems. For example, alone letters as is: n, t, u, v (for example, n.mp4) does not work, while other letters alone works. Also, those letters may work, depend of the location in the name. I suspect that camera name or whatever may be also important.
@@omc7250 check this video at 3:51. You have to install everything from your Blender's local Python path. The beginning of each line should read something like that: ./python3.7m ........ and the rest. That's for Linux. On Windows you just navigate to Blender's local Python path and run everything starting with python -m ..... and so on. Where python is the name of your executable in bin folder.
Import cv2 does'nt work for me, I'm on mac with python 3.7.4
It says (when I type import cv2)
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cv2'
But I have installed it and everything is in. Anyone can help?
Is it possible that we use it on our own character model?
Could anyone help? Trying this on a mac, downloaded and installed the pip and the scripts. I see the capture button, but Blender (2.9) keeps crashing, also when I run it from Terminal. Should I install python 3 instead of the basic 2.7, cause I thought it would run from within Blender.
damn i dont see the capture button ( also on a mac), any idea what could be missing ?
I am having trouble with the dlib module. The module doesnt seem to work with python 3.7? Do i have to downgrade or is there an easier way?
took me a while but got it working! Thank you!
is this working on other characters?? i.e own created characters??
In windows python script failed how to fix?
Can't get it to work. I get some errors in the script. Traceback (most recent call last):
File "D:\bu\blender_models\vincent.blend\OpenCVAnim.py", line 38, in
File "D:\bu\blender_models\vincent.blend\OpenCVAnim.py", line 30, in register
File "C:\Program Files\Blender Foundation\Blender 3.0\3.0\scripts\modules\bpy\utils\__init__.py", line 849, in register_tool
raise Exception("Tool %r already exists!" % tool_cls.bl_idname)
Exception: Tool 'ui_plus.opencv' already exists!
Error: Python script failed, check the message in the system console
Excellente video ! But I have a question. This add on works in every face characters ?
dlib is not working for windows 10
U got one more subscriber and one more thumb up and it’s so cool I was looking for something like that
Thats so amazing!!!!!! Please, tell me that it'll become a complete addon!!!! Would bê awessome!!!!
Installed fine but whenever my camera launches in blender it just crashes after I start moving...
Traceback (most recent call last):
File "/home/bob/Desktop/Desktop/vincent.blend/OpenCVAnimOperator.py", line 116, in modal
KeyError: 'bpy_prop_collection[key]: key "vincent_blenrig" not found'
Any ideas? Thanks for the video
If the armature in the project is not called "vincent_blenrig" just modify this line in the script to the right name.
Trying this on macOS 10.13.6 High Sierra and running into problems. First off while, Blender 2.8 has a python3.7 folder, my Terminal says it's version 2.7.13. Later, when I tried the "python -m ensure pip" command I got the message: "Could not import runpy module". Where do I go form here?
Also, do you have to use a webcam built in to your computer? My eventual goal is to take video from a helmet mounted IP webcam as part of a mocap suit.
import cv2 in Blender is giving error, i followed whole video as u described.
I GET this in script window
Traceback (most recent call last):
File "", line 1, in
File "E:\SOFTWARES NEW\blender software\blender\2.80\python\lib\site-packages\cv2\__init__.py", line 5, in
from .cv2 import *
ModuleNotFoundError: No module named 'cv2.cv2'
Plz Help
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
HOW TO DEAL WITH THIS ERROR ??
hello, after run it will display
ValueError: WorkSpaceTool.setup(): error with keyword argument "options" - : 'REGISTER' not found in ('KEYMAP_FALLBACK')
may i know how to fix it!?
Wow, this is amazing!! Could you show how this method can work with the Autorig addon? I think that would be great!!
I'm also looking forward to this feature
Got this working! Really cool stuff man. Found the same issue with Apple's ARKit True Depth sensor. It slips just enough and is fuzzy enough around the lips that it's not really useful for much beyond what Snap is doing with the tech. Weta Digital uses dots on faces and that'll stop the slippage with a good enough camera and enough light so I'm going to have to try this idea with dots on faces and old fashioned point tracking. but really solid work here. I'll be poking through your code tomorrow for sure.
I have not managed to figure out how you keyed everything while recording. auto keying button doesn't do it. hmm.
What about fps on the cam? I remember the ps3 eye worked really good with iPiSoft because it recorded at 60 hertz, iPhone cam records 1080 at 120 fps, instead of real time performance a recorded performance might work better. Also improving lighting might work as well.
Lawrence Whiteside, hey thats a great idea. I currently do a lot of facial animation via ipjonex depth cam but it lacks detail. If I could add a dot based tracking on top of iphonex depth cam tracking it would be really great. Would that be possible? If you are a programmer I would love to work with you to make such an app.
Ty for guide. Really helped me a lot.
As Windows user I had some problems
1. Could not correctly install dlib - had to change python files from owned.
>Had to install CMake and adding System path czcams.com/video/8_X5Iq9niDE/video.html so dlib could be installed
>Even after installing had problems with importing so just copied local python files while keeping bin Folder as it is
2. As @
American Minion Broadcasting Company said had to change some code as vincent model had changed names.
3. Had really low frame rates - Render properties, under Simplify, set: Viewport - Max Subdivision to 0. helped me
I might have skipped some steps if someone really need I might post how or what I used
I'm getting a Traceback error when I try to capture. I briefly got an output window that showed a few frames of video, but there were no points on it.
Python: Traceback (most recent call last):
File "D:\OneDrive\Documents\vincent.blend\OpenCVAnimOperator.py", line 196, in modal
cv2.error: OpenCV(4.5.5) :-1: error: (-5:Bad argument) in function 'circle'
> Overload resolution failed:
> - Can't parse 'center'. Sequence item with index 0 has a wrong type
> - Can't parse 'center'. Sequence item with index 0 has a wrong type
location: :-1
Any idea what this means or how to resolve it? Looks like a data type error?
This is great. Can you update it for 2.83 and show us how to use it with other characters?
Finally It works in my Windows 10 after installing manually dlib and changing the code.
Now my problem is that the pop-up window doesn't close when I click in the X, it opens again.
When I follow the video come to the "python -m pip install dlib" step, I encounter a installing error , then I try to use cmake-gui with "Blender Python(3.10.2)" to compile Dlib directly, and it still complains the same error... but the "Original Python(3.10.6)" is OK. I don't know why ...😢
Thank you soo much!! Great stuff!!
How do you get it to accurately Lip Sync? I use a very similar method for Facial Mocap via the Kinect 2 for Windows and through a Blender addon I found. My issue isn't getting my models to animate, it's after I record the keyframes using the auto keyframing option on Blender. My lip sync moves way to fast after rendering it and creating a actual video file to playback. Plus my PC isn't that powerful GPU wise, and so it's kind of tough to judge whether it's a GPU issue that causes everything to either slow down to a crawl or speed up after rendering.
Like do you ever need to clean up your Blender Timeline regarding keyframes you've captured/recorded?
I have a kinect that I'm trying to get to work in Blender to do similar. What blender addon did you use? thanks
This is incredible!
in the video you dont install dlib only install python3 -m pip install opencv-python opencv-contrib-python imutils numpy --user
This is brilliant
I know it's an old video, but would this OpenCV work with ManyCam?
Python script failed,check the message in the sytem console .how to fix this sir?
i wonder if we could have the plugin already installed in blender?
some way to adapt the script to new skeletons like rigify? would be very helpful
hey man, would you help me install this. i cant get it to work on my. ios mac.
This is a Very informative video. I am using Blender 2.82 with python3 on MacBook Pro. but when I tried to Register and run OpenCVAnimOperator. Py I received error messages. Can you please explain the actual reason of error and probable solution for that? Thanks.
This is great, but does this also work with the blender rigify or auto rig. in blender? or its just the blenrig biped
Thank you for amazingly useful tutorial!
In Blender 2.83 I faced with error: "vincent_blenrig" not found.
It can be fixed if replace it with "RIG-Vincent".
Hope it will help somebody.
thank you... i am done .
I ran into problem where I got the error during dlib installation.
Apparently it requires cmake to be installed
I installed the cmake library before dlib using the following command
python -m pip install cmake
adding cmake in the multi installation 3rd step just before dlib should also do the job
i know i made it understand in a very complicated way. just trying to help. maybe more people ran into this problem. so would be helpful
update... adding cmake in 3rd step does not help. first install cmake. then run the 3rd step
This is amazing! Can I use this to control shape keys for 2D models?
Sir can u please say which algorithm u used here for the open cv
This would be so amazing IF I could get it to work. Got stuck at SyntaxError: invald syntax when I type python -m ensurepip
That's because it seems like in newer version's of blender, pip is enabled by default and python -m ensurepip has been removed.
Hi, I connect it to my blender and it works perfectly the only thing that I still looking for is how to export the animation from it I can’t find the key frames that it makes, thank u
This is awesome!!!
Thank you but I have a problem with the lbfmodel.yaml file. If I use the raw version of your version, I get the message "python script failed, check the message console". After I run the OpenCVAnimOperator.py.
It is a syntaxerror "unicode 'unicodeescape' error codec can't decode bytes in position 2 - 3: truncated"
I added: r, open and double slashes but nothing worked. So I can't get capture under opencv animation.
How can I fix this?
However in object mode I see "openCV Animation" but not the button "capture"
I followed every step
Facial landmarks database link doesn't work. Anyway to get the database?
how can i apply this on hands and use videos instead of cam?
how to find this terminal in Blender? Is this Python terminal separate?
I just wonder how to install the opencv on blender using windows? I installed opencv in anaconda prompt and it's available to import cv2 directly in the window, but not in the blender, even if I have added the cv2 location to the sys.path.append. Really need some help here...
OK I got it working! So is there any tips to get the tracking a bit smoother!? Thanks again
I was not able to make OpenCv work even after it was installed, it is very complex to understand the installation, does anyone know if they have an addon to install OpenCv?
It would be more benefit and precision if you could just set video path to process so the script would go frame by frame and make it better (and you could fix it too per frame basis).
But the mouth only moves up & down is there a way to have thd mouth move properly?
Dear teacher, I use the latest version of blender3.0, according to the operation of your video, but there are always errors, I would like to ask you
uém pode me dizer o que está errado? Python script failed, check the message in the system console
I successfully compiled for windows 10 and work fine with slow
Wondering, how is OpenCV dlib working so well for you. In my own experiments I find it can get very confused with open mouth animations. What's weird - mouth animation works better in low light conditions. In bright light, as soon as I start talking the mouth landmarks sometimes suddenly jump up and down.
If I want to use my own 3D model, what preparations do I have to do in order for it to work? I'm not very advanced, sorry if the question is dumb
@gadgetWorkbench was able to get it setup with all the downloads, but as soon as the capture starts on MAC, the camera light goes on, then off, then crashes blender, any recommended hardware specs you need for this?
Ok, don't try this on windows, it's just somehow not working at all for me
dlib won't install because it can't find cmake, even though I have that installed, with several ways
been at this for 2 hours now and it won't want to work
It just crash for me when I start the script....
Running on Mac OS, using blender 2.83 running the scripts successfully but once i click on the UI button to start the motion capture blender crashes, i see the webcam turn on (the green light) and it turns off, then a error message appears that Blender quit unexpectedly
Ok had to run blender from terminal on mac os. I think maybe the camera privacy permissions were messing something up
Thanks a lot my friend!! New sub here!!
this is cool. but do we have something drag and drop? I'm a coder who codes python for years but i have no knowledge of opencv.
How can I not find the terminal in my windows! Do I have to install Terminal and Python 3.7 to make it the same as your video?
can you copy the keyframes to another model?
hello! i have been facing an issue for an ETERNITY, the python that blender makes available is (frustrating to say the least) so i saw that in 3:19 you had your own custom python interpreter, is there any possible way for me to do that? i have looked at so many forums and videos online, nothing mentions the way you change python interpreters in blender, much appreciated if you help!
Is there a reason the capture is mirrored OR is this a camera widow artifact mirroring your image real-time?
It's kind of like a selfie camera - its easier when it's mirrored. But you can easily flip the image back - just search "opencv mirror"
@Gadget Workbench, Does it only work with Blenrig? I tried with another armature and it doesn't record. Did you test with other armature?
Yes, it works with Blenrig. It's specifically coded for the vincent rigged model. You would need to change this: github.com/jkirsons/FacialMotionCapture/blob/master/OpenCVAnimOperator.py#L116 and the bone names under it if you're using it with a different armature.
@@GadgetWorkbench Thanks!
@@GadgetWorkbench can u please keep a tutorial on how to use for another Armature rigs like Auto Rig
Super! Thanks
Does this work with full body?
Hi everybody, is there a tutorial to show ho to link this facial mocap system into another character? how to fit it in my own character face?
Does this work with kinect SDK 1.8 or 2 for keypoints to blender realtime? That would be great if there is addon
I keep getting this dang error even though ive installed, uninstalled and reinstalled cv2 about half a dozen times. No luck. >>> import cv2
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cv2'
It is because CV2 wasn't installed properly. Try installing again.
U are a beast :)
There is some clue why it does not work in Windows 10. After installing all that needed and as described on the video, unable to run script 'OpenCVAnim.py'. Ignoring that, tried to click on 'capture' on new tool: "OpenCVAnimation ->Capture" and got error message: "Traceback (most recent call last):
File "D:\Blender Projects\Mocap using py\Vincent_plus_scripts.blend\OpenCVAnimOperator.py", line 84, in modal
cv2.error: OpenCV(4.1.1) C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor'
location: :-1".
Now, D:/... stuff is mine, but "C:/projects/opencv-python/opencv/modules/imgproc/src/color.cpp" is not. Folder 'project' does not exist on my disk, nor any related C++ file as is 'color.cpp'. What did author of this video has installed on his disk that he forgot to tell us? Thank you in advantage.
Well, found OpenCV in 'full glory' on github: github.com/opencv/opencv
But, there is now color.cpp and at line 182 says: "CV_Assert(!_src.empty());" - exactly what indicate as error. At least found all that one. BUT (!), even when move all files into newly made folder with color.cpp file: "C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:182" - that is not real folder. Tried to change line 182 in color.cpp, but that does NOT do anything. The same error remains, even now this line of code is changed. I think that pre-compiled version of all that folders is broken or something missing. Also, I am not sure how it is possible to find webcam. I am using Mobius camera with removed microSD (only then it works as webcam when on USB port), and tested in OBS studio. Mobius camera is there ONLY if 'Capture' button is NOT pressed in Blender. That means that it tries to reach camera, which is good sign. OBS studio can't share camera, or better said, if camera is 'busy', then it does not show anything. Remaining question is about drivers in OpenCV - does it support Mobius camera as webcam? Currently I have no other webcam. So, maybe it will work with other webcam, or... not sure... maybe?
@@MilanKarakas I had the same problem Mark. I had to comment out lines 142 - 144:
#self._cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
#self._cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.height)
#self._cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
I think they are incomparable with my web cam,
Ok, so is there a version of this using a lib that *is* for commercial use? Or do we basically need to build our own from scratch?
Not that I know of. It's the dataset that the trained models were created on that is not for commercial use. I suppose you could use video of your own face with annotations provided by a commercial product to train a new model. If the commercial license permitted this...
Will this work on any character or only Vincent?
getting error while running the script
can you use this addon to a custom 3d face?
Is there anyway to just get facial capture data without having it run as a blender app. So that a VA can cam their performance send me the dataset and audio?
You could have them record a video, then on this line: github.com/jkirsons/FacialMotionCapture_v2/blob/master/OpenCVAnimOperator.py#L210 pass the filename instead of using a camera, syntax: docs.opencv.org/3.4/d8/dfe/classcv_1_1VideoCapture.html#a949d90b766ba42a6a93fe23a67785951.
why is this not on reddit
sir there am not getting capture button in my mac any help??
Great video! I'm installing opencv apparently correctly in the spot you said to, but when i try to import it in blender, it is not working. Any tips?
same man, do you have a mac ?
awsome...
Anyone know how to do this with another character you have built? Ie from Fuse ?
Will you please give the link of paper/documentation??
Hi. I have a problem. The "Capture" button is not displayed and the script throws an error at startup. Can you help please?
Same problem 😣
May i Ask your for help with exact something in ICLONE 7? Iclone 7 also have PYTHON inside. But im complete newbie in Python this is dark magic for me... . Having thing like this, can save thousands of hours, to creating Facial Animation. Im goona to learn python soon, i installed pycharm,i installed also Python 3.8 , i installed also cmake (in pycharm), - Im not able to instal DLIB - ppl say, that DLIB only works with Python 3.6 smth. So can you help with this in iclone? Thanks for great video in blender. This is a Free Face Tracker ... and ppl Paying 2-3K $$ for payable thing like this...
So does this work? Using Blender 2.91. Someone please let me know
Great video , i am dying to get a indie friendly budget or free facial mocap system .... ;)
some questions tho .
- is this a markerless solution ?
- can we get better animations by increasing bones resolutions , i mean adding more bones or something like that ?
- is there a way to capture eyes blinking and movements ?
- what kind of webcam is needed to do this ?
thanks a lot !
Yes, this is markerless. For more bones, you would have to link more facial landmarks to bones. Yes, blinking could be done by the distance between the upper and lower eyelid landmarks. Any webcam will do - just needs a video of your face :)
@@GadgetWorkbench Very Cool, i will definitely give it a try when i can .
btw what kind of webcam / resolution is recommended for this ?
It's at 800*600, but you can change this in the code. Good lighting is probably more important than resolution.
@@GadgetWorkbench got it,
thanks again .
How did you mive the workspace to show Scripting on 2:52 ?