My Rokoko MOCAP Guide for Unreal Engine and MetaHumans
Vložit
- čas přidán 22. 07. 2024
- This is my process of doing offline Motion Capture / Performance Capture using the Rokoko Smart Suit, Smart Gloves, and Rokoko Studio. Rendered, retargeted and MOCAP editing in Unreal Engine 4.26 and UE5.
*this video was originally produced for the Citizen Meta One CZcams channel, but I've uploaded it here and will be taking it down from that channel - Krátké a kreslené filmy
thanks for sharing all this info. I just got into mocap with unreal and metahumans a few days ago, and I almost feel like its a crime for me to able to access your knowledge that has accumulated through out years, condensed into a few videos. Your content on sharing your method, examples and entire workflow pipelines is really really valuable and beneficial for someone like me to get intuned with the current landscape. Thanks a whole lot.
For anyone wondering what those batteries used with the gloves are, they are called
Anker PowerCore 5000
Amazing that this can be done
Awesome video dude, appreciate it!
The metahuman looks more natural than before, and not just body (which should be better since you mocap) but the face expressions aswell.
Nice. Great intro.
This is great, Matt! The battery situation is bizarrely the most frustrating part about Rokoko hahaha. It’s because they can’t ship from… Denmark or whatever. 😂 I have my suit and a glove powered with one usb power brick, and my second glove on another usb power brick. And you’re right - if you put both gloves on the same usb power brick… they don’t work! Argh. I wish I could re-write their “help” stuff and also how Rokoko Studio works - connecting everything is a pain. And the 2g/5G lights - don’t get me started. WUARGHHH. 😂
Reading this after getting the anker battery….🤬
Wow super cool! I was wondering if that is a custom metahuman body. Looks amazing!
Excellent thank you for the video - any tips you learned about calibrating the Rokoko suit - I think it could be one of my issues....
Great stuff, thanks!
Super cool!
Hi! Cool videos thank you it’s very inspiring :) Do you know if the Rokoko gloves are compatible with the XSens suit? Xsens can detect gloves within its software MVN Animate and I don’t find any info about Rokoko gloves.
23:18 do you have the link you mentioned for the face?
Hi Matt, your workflow to adjust the phonetic is amazing! I am in need of something similar as i will do some projects in my company. Can you help me or guide me to have something similar? Cleaning up phonetics is a pain with the keyframes, I am really in need of a solution. Cheers!
hello i have a question , if i record a basketball game and give the rokoko suite to one of the players , is it possible to replace this player with a real avatar ? can you use the motion capture system outside between real peoples ?
This is so cool 🤩🤩. Your channel is awesome! How would you go about making the motion look more natural? Changing the shutter angles, frame blending etc, motion blur? 🤔 it would probably disguise many of the small details that are too time consuming to program in the mouth movements and some of the harsh arm movements?
With Rokoko, you would basically do a better job of cleaning up the MOCAP than I did. I was a bit heavy handed with this clean up because it was my first time really. I would probably filter a bit less on the first pass and try to use more of the raw performance. MOCAP clean up is it's own art.
@@CinematographyDatabase oh ok 🤩 I was thinking if you cut out every second frame , and applied motion blur it might also work🤔. It's almost slow motion but still so awesome 🤩 have you thought of doing music videos too? Virtual production is the entire future of the media industry🤩🤩
I am developing a wearable of my own. I am curious as to what made you choose Rokoko vs. other mocap systems for tracking. It would be useful to know what ASUS router you used as well.
How did it went? Did you manage to finish your wearable? I am also developing a full body system of my own. I know it is 1y later, but regarding the router, any dedicated router should suffice. Also the internet connection is not required (should actually be avoided) as only the wlan is necessary to connect to the sensors.
@@dimeland2 Hi, I'm having some issues with my router is there somewhere we could talk I'd like to ask you some questions if you don't mind
what the hell is a z8 loll amazing vid man! thanks for the inspiration
Fantastic video. Which ASUS router did you go with?
Hi Matt, have you been able to try a Perception Neuron 3? How would you advise in getting a Rokoko Smartsuit pro II or PN3?
I haven’t used the Perception Neuron system myself.
would love to see this character with the new Facial motion capture
Hello brother, can you kindly share a link from we can get the helmet (or more info)?
And thanks for this great video!
wdu think of the new chest mount from rokoko for iphone?
its great but hopefully Rokoko will bring out a plug in to make this process easier! this is quite a chore just to get it working in UE. i mean with the retargeting etc. the blender rokoko plug in retargeting is awesome! they should bring out something similar for UE.
I’ve heard the C4D retargeting is pretty easy too with the Retarget tag. UE is more complicated in general compared to DCCs because it’s a game engine / real time. It isn’t simply baking data to curves/key frames. It took me a while to get comfortable with this workflow but most MOCAP to UE look exactly the same.
@@CinematographyDatabase thanks for replying. Your meta human combined with the Rokoko mo cap looks amazing though.
Hey, nice overview) can you tell, why did you choose Rokoko? I ask, because I plan to buy a suit, and right now on choosing stage between rokoko and perception neuron.
Can you share your decision?
Thanks
A year or two ago Rokoko sent me a suit to test out with Unreal Engine. Then I recently got back into MOCAP and they sent me gloves to test with MetaHumans. I see good stuff from the PN team but I’ve never personally used it.
Shown in this video, I really like that the face, body, hand data is recorded all together into Rokoko Studio. That is such a simple process compared to recording each separately.
Rokoko is also very active in the community and Sam does some great live streams to show how to set it up and shares helpful Blueprints for MetaHumans. Without his initial work I wouldn’t have been able to produce this video.
@@CinematographyDatabase thanks
what camera do you use? and what format?
Hey Matt, after watching this video again I figured what the hell I want to watch the "very long" cleanup live-stream. Where can I find it? The link is not in the description.
Those streams are offline now. If you look up MOCAP clean up YT videos they are pretty much all the same process.
Please can you tell me how to retarget the bones of the hands from rokoko to metahuman, I have only been able to make the body, and I have not been able to find a tutorial on this, thank you very much
@CinematographyDatabase
Was the Metahuman in T-pose to get it to work with Rokoko?
Yeah the MH needs to be in a Tpose before the Rokoko LiveLink node. Rokoko should have a nice Control Rig for putting the MetaHuman into a proper Tpose. I made mine from scratch.
does the headset comes with the suite ? or do you have to buy it seperately ?
The face systems are separate. Most consumers use an iPhone and a Standard Deviation HMC. High end is Faceware or Facegood.
@@CinematographyDatabase so you put the iphone on the helmet ? does the helmet also have sensors ? where can you buy this head set helmet
why i export nake body metahuman will got green spot on the skin around the neck and hand ?but u didnot?
Do I have to pay an annual program fee to use this program?
I hope someone can replay to this. Can this be used Visa Ipad face tracking like it can be used for an App capped animaze. I would like to know if I can make a MetaHuman rigged for V-tubing? I use Ipad 2021 with the A1 chip and Leap Motion tracking. These are the tools I have on hand. can these be used with a rigged MetaHiman?
Anybody know where to find the livestream to the lip sync tutorial he mentions?
Hi! Can you use the face cap mount without the Rokoko suit?
Yeah, but then it won’t capture head movement. If the camera is on a desk/stand it can also capture head rotation.
hello plz, Can i shoot with a Gopro camera?
Nice guide, thank you very much! Now all we need is for Rokoko to send their smart suit as promised and stop delaying again and again :(
yeah I've heard there is a lot of interest in the suits and gloves, so they are a bit back ordered
They are worth the wait, and it’s a small company so I think they are overwhelmed right now (with Covid etc). But the suit and gloves are awesome
I feel you. Waiting for so long already and it keeps getting delayed.
@@CinematographyDatabase They say they're sending like hundreds of suits in a per week basis. I think that's an over exaggeration otherwise it would mean the entire world is now getting rokoko suits xD
@@virtualfilmer I really need them to see how awesome they are ^^'. Right now only got the gloves, but can't do anything serious with just those.
how much does the suit and helmet cost and where can i get one, whats the link friend
Thank You😊
i want to do this so bad 😲
Hi, I'm just studying Mata Human. I envy your equipment so much of your equipment.
Is there any fix for the accuracy of foot capture yet?
Not that I've heard. You have to calibrate correctly, have a reliable way to keep the foot sensor on your foot, use the locomotion 2.0 filters, and then do manual foot contact clean up in Maya/UE/etc.
friend where can I buy a helmet like yours for facial captures
I recommend Standard Deviation if you are using and iPhone.
Do you have your dedicated router connected to Internet, or it's just connected to power?
I have a dedicated ASUS router that only allows the suit, gloves, iPhone, and Rokoko Studio workstation on it. It does also provide internet, but it's not needed.
@@CinematographyDatabase I have also ASUS and Rokoko, and it wasn’t allowing me to connect suit and gloves until I connected cable from ISP router to ASUS.. now its working as it should..
@@m1sterv1sual ahh thats good to know, I didn't try it without the ISP plugged in.
When wearing this hat, does it detect head movements, for example turning left and right?
The iPhone can do the head rotation tracking but in this case the Rokoko head sensor does the head tracking.
Could you put the "lip sync tools" website address here?Thanks a lot!!!
@CinematographyDatabase Have you ever used character creator 3 with your current workflow? Just wondering since they require T-pose but export with A-pose as there bind pose. Been difficult getting the character to link to Rokoko.
Character Creator you can choose what pose they export in and or just export the CC3 Tpose and use that in the AnimBP in UE. Retargeting is one of the harder parts of all of this.
Hi! I am using Rokoko Smart Suit in UE5. Animation Successfully recorded in ROKOKO Studio. Live link is also done properly. But after recording animation with Take Recorder, the Metahuman Skeleton gets distorted in Animation Sequence. Can you please help me on that?
One question is how are you retargeting or are you just directly Bone Mapping? Either way, you need to make sure your animation that you are applying is using the correct source aka the correct skeleton type like Female Medium Normalweight for example or the animation will scale the MetaHuman into odd proportions.
@@CinematographyDatabase Yeah, I am using Character_BoneMap.uasset file provided by Rokoko. But that is for UnrealEngine 4.6 and I am working in UE5. So is there any other way for retargeting or Bone Mapping for UE5?
@@CinematographyDatabase Thanks a lot 👍
genuinely thought it was real for the first 10 seconds
Where did you get the helmet?
this one is from Technoprops, I recommend the Standard Deviation one for iPhones. Technoprops isn’t publicly available.
@@CinematographyDatabase $1350 inc counterweight - oooof that’s a lot of cash, considering we end up with AR kit data at the end of it 😂
What's the name of the mocap helmet headset
This is a Technoprops, but I recommend the Standard Deviation for iPhones
hi brother i need Rokoko motion capture tools please locate by help me
Can you do stuff like this with more then one person ?
Yes. If you use an inertial suit like this though, interactions between the characters won’t really like up.
@@CinematographyDatabase so it's a yes and no with doing a project with two or three people and make a video like that ?
can you make an update on rokoko mo cap
Yeah I’m kind of interested in their coil tech, I haven’t no idea how that works. But I don’t have that system atm.
Looks great. The mouth still looks weird
can you link the 'very long livestream' about cleaning up mocap? Thanks
those streams are on my VTuber channel - czcams.com/video/wa8SB03O_cQ/video.html
Why is it private? 🫠
🤘
这个不错啊。不知道要多少钱。
Where can I buy these technoprop bad boys?
they aren't publicly available. I recommend the Standard Deviation HMC if you are using an iPhone.
@@CinematographyDatabase Awesome tip! Thank you they look exactly like what I was looking for.
@@CinematographyDatabase only the price is confusing. It looks like 100 but costs 1150. Is there anything that has a better value for money that you could also share with us?
Seems like AI and the work Nvidia is doing to automate a lot of the phoneme work will make this workflow seem like a thing of the past somewhat soon.
I think that real time MOCAP like this gets us around 80% towards realism and then GAN/DeepFake will provide tools to get the rest of the way there from the tests that I’ve seen.
Haha the HANDS CROSSING - yes. It’s a big issue with the Rokoko suit/gloves. I wish we could fix it inside Rokoko Studio. I used to think it was because my Actor (in Rokoko Studio) was set to be the wrong size. But that didn’t fix it. I then thought maybe It’s because I’m an unusual size (6ft5) so maybe the math is screwy with large bodies. But yeah. I can’t get the hands NOT to cross - it’s like the “mannequin” inside Rokoko is just… screwy. So we have to clean it up after. Booo! 👎 :)
Rokoko Studio doesn't have a ROM with prayer pose to try to estimate the correct bone lengths for hand position. It will definitely always require clean up later to hand positions where you want them.
@@CinematographyDatabase exactly. V annoying!
why is your avatar moving so weird?
I thought it was a real person OMG
this is sick but maybe put a shirt on dude
Now imagine this performance capture with ML machine learning deformer
Why I love envido elements?? i don't give a fu*k
Hand movements are way over the top.