The Face & Eye Tracking Rabbit Hole [VRChat]
Vložit
- čas přidán 1. 06. 2024
- I dove into the world of Face and Eye Tracking in VRChat! What seems like only a niche technology, hides a fascinating peek at the future of online communication. I interviewed leaders in the community, who spend their time shaping open-source hardware and software to bring this extra bit of realism to our interactions.
Checkout SideQuest’s physics based social VR app BANTER for a chance to win a FREE Quest 3: sidequestvr.com/app/10831/ban...
Join my Discord!:
/ discord
VRCFT: / discord
EchoTheNeko: / echotheneko
EyeTrackVR: / discord
PHIA's Socials
Twitter: / phia_bunny
Twitch: / phiabunny
Instagram: / phiabunny
TikTok: / phiabunny
Patreon: / phia
Voicemod Code "PHIABUNNY" 5% off:
www.voicemod.net/voicemod-par...
=TIMESTAMPS=
0:00 - Intro
1:35 - The Face Track Conversation
4:45 - Sponsor: SideQuest/Banter
5:31 - VRCFT
7:58 - Avatars
9:45 - EyeTrackVR
12:08 - PRIVACY!
Join the TVRS VRChat Group:
vrc.group/TVRS.0429
TVRS Official Twitter: / tvrs_official
TVRS SHORTS!
/ @tvrsshorts
PHIA Second Channel:
/ @phia_bunny
TVRS Studio Channel:
/ @tvrsstudio
Editing help by Moonwatcher & Thrill
/ moonwatcher68
Business Contact: phia@thevirtualreality.show - Zábava
![The Most ADVANCED VR Game in the WORLD! [Resonite]](http://i.ytimg.com/vi/eLoVnZZWwEM/mqdefault.jpg)
![The Most ADVANCED VR Game in the WORLD! [Resonite]](/img/tr.png)
Make sure to checkout SideQuest's Discord for Banter to get info on that Quest 3 giveaway :)
discord.gg/bantaverse
I’ve been a Banter user (ARK1701 in VRChat and Banter) for about four months. Banter is event driven and a very friendly community.
discrod lol
Hi Phia I contacted you a while back via Email about a potential research into VR etc. I have not heard back, maybe went to your spam folder. Let me know - you may find it very interesting.
Yeah I don’t want those headsets that fail and overheat.
Can i have some trackers so i can make a video on them
f a c e t r a c k i n g 👁️👄👁️
Cidil
trilseker
facetracking, but your eyes are covered by a magic brick that sends to you to cyberland
Psst, check out project babble to do diy lip tracking. Goes great with eyetrackVR
BABBLE BABBLE BABBLE
@@summerbreeze2167Where my headset mr programmer socks?
Thank you so much for featuring EyeTrackVR! The future of facial tracking deserves to be open, and I'm incredibly honoured to be a part of such a talented community working towards it!
Ello!
@@RidgeXR_2hey ridge, lol
@@prohurtz *rigde
"If they don't do it, we will. That's the beauty of the open source community." Truer words have never been spoken.
Thrill + PHIA is the ultimate VR power couple. They both cover very different but very significant sides of the industry and work off of each other really well.
EEE THANK YOU SO MUCH FOR INVITING ME TO BE A PART OF THIS!! Im really passionate about face tracking and I’m so happy a whole new audience is being exposed to it!
imagine you were still mute
yooo Echo :D
Are you currently taking any new commissions?
Wishing you well with surgery, take as much time as need to recover ❤
We really need more headsets to be released with face and eye tracking, or at least eye tracking. It's annoying that headsets being released still don't all have eye tracking.
Sure, we can do it open source by ourselves, at least for eye tracking, but we shouldn't *need* to do it ourselves.
Looking at you, Meta...
Being able to look someone in the eye while having a conversation is why I always thought VR is the first true social media application. Everything else feels para or anti social in comparison. Great vid phia !
Seeing someone with eye tracking look at you in VRChat hits different, especially if they are flirting with you.
As long as my QPro doesn't make my eyebrows start doing jumping jacks for an hour straight just because I raised one slightly
Disabled VRChatter here! I'm blind in an eye and one day I'll get my hands on face tracking and folks will be able to TELL because of how it just-- kinda follows my good eye but doesn't move on its own. It's hilarious.
ALSO Echo's face tracking is FANTASTIC. She's worked so hard on all the tracking and has spent YEARS in Blender [someone please bring my daughter back from the Blender wars]. I tell anyone who wants face tracking to hit her up because of how EXPRESSIVE it is.
So does that mean you have similar eye movement to actor Tiny Lister aka Zeus in No Holds Barred/WWF aka Deebo in the Friday movies? He was also blind in one eye, so he had what you described in his eye movements as well.
Wait could I hit her up? If you don't mind giving out that info. I'd love to learn from her!
Honestly, it is just mind boggling how much VR is being advanced these years, makes you wonder how far we will advance in just 1 or 2 years from now.
I love the face track conversation having the virtual and actual perspectives, you've truly outdone yourself with this video!
HOW DID I JUST FIND OUT U AND THRILL ARE TOGETHER?!?!
That's not Thrill?
@@NixonAxi it is..
Yeah is that really him?? Like i recognize the voice but i find it so wild???
Maybe you guys should do a VR chat podcast
This is wild. That EyeTrackVR tech really opens doors to majority of us without native tracking headsets. I might explore this.
hard to go without it once you got Facetracking! Just the subtle amount of movements you do in a conversation is crazy.
Thank you for making this video. the VRCFT community has inspired me so much and is so special to me. Huge thanks to Ben, Jerry, Azmidi and all the module and avatar developers who made it happen. Happy to see my friends Hash and Kombi featured here, they are awesome!
I'd say you guy's relationship are goals for any VR community member! I like these videos that feature the both of you!
EDIT - You just made countless thousands of people want face and eye tracking!
I love seeing you both in a video together! My wife and I met in second life almost 20 years ago and we love being together all the time in real and VR! ❤
I love face tracking, hope to get it eventually. it's so neat to see how completely transforming interactions can be when seeing peoples expressions.
really glad to have been able to provide some footage. wasn't expecting my memes avis to be in there (I skibidied)
Epic video. I've been loving Face Tracking on the Quest Pro. For my purposes, I only wish the microphone was far better for Content Creation and Streaming on that headset, to be honest.
It's also been fun learning about how it all works via OSC + VRCFT Software and more.
Using it with my own Karin and Rusk avatars that I've spent 100s of hours on in Unity, to perfect to my likings, has been really neat too.
Also I hope your surgery goes well and that you make a swift recovery. And once you're back to being able to make videos, I wish for you to be able to easily keep up the good work on the entertaining and informative content you produce. Take care. :)
Most informative. I'm Trying to decide between quest pro and quest 3. I want to game but I also want to be very expressive. I'm still trying to find something somewhere in between or choose one or the other.
I have struggled with applying facetracking on my VTuber Avatar myself, but in the end i am really happy, that i have accomplished it :3
i ordered a quest pro a few days ago and have been so excited looking for new videos on it and HERE YOU GO 8 HRS AGO!
wooooo!
1:26 "...pulled my partner Thrill[seeker] into a..." so it's official? 😉
Exactly what I was thinking! Always had my suspicions but didn't want to jump to any conclusions or anything
They’ve been out in the open about their relationship for a while now
yes, they even mention the fact they live together and see each other every day
The face tracking is SO impressive, and so was the way this video was edited. I was hooked 🙌
Fact that this channel didn't go viral yet, is for me simply mind-blowing
as a kid who got into vtubing first, then VRC, imagine the goldmine this was lol
I was surprised to learn that hey, the rigging knowledge I might have in 2D rigs might be useful in 3D rigs!
and the way that people make a living off it too!! I didn’t know it was like that in the 3D community- I thought it was limited to 2D models haha. I’m excited to see where all of this goes in a few years and maybe even get into it myself :0
Wishing you well in your health and please take the time you need for yourself!
Love your videos.
I really want a new headset to come out that has the same speakers + microphone quality as the index, But with the option to go wireless, better resolution, FOV and face + eye tracking. And of course it still needs the option for base station tracking as well, so I can use base-station tracked full body trackers.
XR and Social VR is the next phase in the evolution of telecommunications. Face tracking is another important part of that evolution.
We’re the early adopters, but in another ten years, I believe Social VR (VRChat, Banter, RecRoom, the Zuckerverse, etc) will be main stream.
I do think it’s super weird as much as an evolutionary step forward as the Apple Personas are, they don’t have their own Social VR platform, just FaceTime. Why take a step forward with one “foot”, only to make a diagonal step backwards with the other “foot”? 🤷🏻♂️
i cant afford to use any vr stuff since im poor. but i like watching your vids and keeping up with the tech evolving and what people are doing with it. i hope by the time i can afford to get into vr, things like eye tracking will already be standard and accessible!
as someone that motions a bit with their hands while talking the hand gestures in VRchat make it really hard to convey the proper emotions, being on the fence about jumping into the VR enthusiast rabbit hole I hope a gaming focused standalone headset comes along with native face/eye tracking.
Haven't been on your channel in three or four years...I'm so glad I found it again! Amazing how much improvement in technology has come in the past few years!
13:17 good to see vyde from New Eden making an appearance
My favourite thing to do when I music mirror dwell, is to gesture sync to the song, basically acting along and syncing my face gestures to the song, acting asif my avatar is singing it in a play or something. It's amazing and fun to see others that KNOW their gestures and actorly naturally use them when talking or acting; I love doing it so people can really see how I'm reacting, and feel better immersed when talking to me ^-^
My main avatars are not picked by how they look and toggles, but the most thing I look for a diverse gestures with a wide range of emotions. ones with diffrent gestures on each hand, or ones that have perfect mixed gestures.
I always do different gestures per hand on my avatars, but am soon going to be adding a 3rd set for double gestures! :D I cant wait to use it ^-^ (and of course adding a toggle so people can turn off 2 of the gesture sets for when they just wanna chill or play games hehe.
As someone who has never had enough time to use vrchat often enough to memorize which hand gestures made which faces, face and eye tracking is a godsend. I can finally express myself in vrchat with literally no effort, as opposed to having to memorize hand gestures. Also, having started with vive wands never helped, since the circle trackpad was split into 5 sections for hand gestures. Because of the split like that, it's always been very un-intuitive and hard to memorize. It wasn't until 2 years ago when i upgraded to index controllers that i started to be able to more easily memorize hand gestures finally.
It's strange how many times I caught myself responding empathetically to you due to the micromovements you're eye-tracking was picking up.
I’m glad you mentioned the need for open-source technologies. It’s very important to not let companies rule this realm too.
I think you've had that surgery by now... Wishing you the best on your recovery! Hoping everything goes as smooth as possible. Health stuff getting in the way is definitely frustrating.
Thanks for the video. Very informative and fun, as always!
I'm pretty excited on where VR is heading with tech like this, though I share the very based concerns about corporate abuse (IE Excessive advertising and data collection).
That said, I'm glad i'm recognizing some of the folks featured, makes me feel connected ^_^
it's great to see the community grow like this, face/eye tracking allows so much
Another big opensource project in this field is "Project Babble"
It's being worked on by a small but passionate development team, working on making it so that if you can get a camera onto your headset, you can use an opensource AI model made by the community to get face tracking up and running. Right now, it's somewhat picky, but they're working on getting more and more data, and using different techniques to train models. They've also been getting more and more people in to help render out new faces, and they recently announced that they'll even need to get help from the community just to train the next model, not even just making the images
Eye tracking is a fascinating technology. When i get a new headset I will certainly try to get eye tracking with it. Also i hope your surgery went well
Ever since I started using face tracking in vrchat with my quest pro, I can't live without it. I have a quest 3 that sits on my shelf because face tracking is just that important to me. People are always blown away. And people comment that they can really tell that I'm listening to them when they talk to me. It's very important.
It also keeps people from talking over me with the voice latency because they see my face expression forecast that I'm about to talk. It's great
so this is cool 1 minute ago it got posted thats cool that i got here this quick :)
As someone with the jankiest headset ever, the pimax crystal, setting up eye tracking and face tracking took me so deep down every reddit thread and discord server. After five third party programs and a lot of lost brain cells, I finally figured it out tho and its so cool :D
I love my face and eye tracking no way could I ever go back to a headset without it everyone finds it so cool when they see it.
This is a pretty interesting aspect, I recently came across your channel recently and now I'm going to subscribe because I enjoy the editing and the ways your manage to get your content across. While I'm not too involved with this as all my characters are... knights/mass effect aliens/sci fi soldiers. This is good to stay in the know.
Wow!! All the guest stars made this possibly the best video yet. I love the virtual reality show! Get feeling better, surgery is the worst 😭
I have gotten so good at puppeting my models with hand gestures and locking them on and off, people have thought I had face tracking multiple times now.
I hope you get well soon... 2 months belated
I love the long form content! This was so fascinating and I am hellbent on trying face and eye tracking out now!
Hope you have a good recovery from your surgery! ❤
To fellow commenters, where do I get a really solid option for a custom avatar?! I have been to so many avatar worlds for about two years of searching and I am struggling to find my fit. I think I need to have one made just for me.
Face tracking is daunting, but once the technology is more available, I think it'll become commonplace. In his review of the AVP, Zuck said he's planning on bringing eye tracking back for future Meta headsets, and I expect that to influence the industry. I can't wait to get that extra layer of immersion myself!
Hope you have a successful & speedy recovery on your surgery 💖
Take as long as you need to recover. Love your vids
Social anxiety in vr was already a problem for me but now it’s going to be words with eye contact 😭
We're getting there! Can't wait for a future where face and eye tracking is available in most mainstream headsets
when me and my friend have a full face tracking convo we start wheezing its so funny 🤣🤣🤣
What a great video! We always learn so much from you PHIA! Thank you.
finaly a new video. :)))
Hope a speedy recovery
I recall seeing a listing for plug and play HTC Vive eye trackers... was 1400 euros back in 2020. 1400 for two plastic clips that hold a single camera pointed at the eye, and patched into the HTC Vive Camera... the price for convince will remain high so long as most people want the simple solution that doesn't require soldering or setup, and if it isn't natively supported in the application, they likely will not blame the +$1k addon. Didn't know they had a solder solution, and will be looking forward to that, as last I checked there were no good solutions :D
the skibidi toilet at the beginning caught me off guard 😭 i can't escape it
future president of vrchat, you do us proud
I think it would be interesting to use the eye tracking to see if your eyes are open, so it doesn't have to spend a bunch of computing power rendering the world while you're sleeping, but you can still see it when you wake up and open your eyes.
I personally bought a quest pro purely for the added built in Face and Eye (and now tongue yippee) tracking. It just adds an extra level of expression that feels way more natural to me than using controllers to determine how my avatar will look
i have been playing vrchat with face and eye tracking with my vive headset since 2021 and that has made it really difficult to adapt and move to another headset if I'm playing VRChat. it makes a hell of a difference to my experience in the game.
6:47 That second of rage 🤣
Hope your surgery went well, keep being awesome~
I've always wanted more tracking available in VRChat. Initially, I thought VRC eye/facetracking seemed off. When anime characters are making these intricate and almost non-exaggerated expressions, it seemed to fall into an uncanny valley of sorts. It seems like it's a matter of how the original avatar was styled as, since IMO it looks a lot cleaner on you than Thrill.
I think that's why a lot of the furry avatars seem better for it.
Thanks for this great and nuanced video! I find the last part about larger corporations exploiting this information to be so interesting and important.
ALSO hope the surgery went well and wishing you a speedy recovery Phia!!!
I love the expressiveness Face and eye tracking give me with the Quest Pro. It's basically the only reason to visit Horizon. I had it set up in VRChat a few times but I need to relearn it and find or buy a good avatar.
Wishing you a speedy recovery
.WingZero.
I'm still using a rift. Best headset ever made period! They were built likes tanks, never broke as often as modern headsets and are hella cheap now. I just got my 5th one back in January for $60 and it had only been used twice in it's life.
Thank you so much for the video. Im super interested in what VR headset tracking is recommended by you and others or if DIY is still the way to go. Hope your surgery went well, best wishes
3:18 exactly just got facetracking 4 days ago it changed everything never going back its just too good!
nyaw, hope u get well soon phia
Jerry's face tracking is amazing! Please check them out on gum road. Love my quest pro face and eye tracking. Only had it a week, and I do not think I could go back.
I am pretty surprised that this didn't cover babble, and a little sad. Hopefully it will in the future, we have a pretty great community working on it
Sidenote but I absolutely love the thumbnail! Really great composition and colours.
For a while I had EyeTrackVR working on my valve index. To find out it was annoying. I use the portable mirror a lot for phantom touch/sense. And every time I peek at the mirror, the person(s) near me got confused where I am lookin at, even looking over their own shoulder to see if maybe someone else sneaked up next to them. But I was just checking the mirror for touch feelings, spacemove positioning (clipping in chairs) and if eyetrackvr wasn't glitching out and needed recalibration. In the end, I only found it usefull for blinking detection, and not for eyemovements itself. I just prefer to have phantom touch over eye movements, without other people getting confused.
YOUR EYES. So expressive!! 🤩
I have FBT and face tracking and alot of my friends do also. It absolutely gives a since of presence. I exclusively look for face tracked avatars now. Being able to smile and make eye contact is huge. I have a quest pro and with a few mods is an amazing and comfortable head set. Pitty they droped it.
I bought a Quest pro (used for cheap) last christmas, and one thing that most people don't talk about is the availability of face + eye tracked avatars. I found 2 worlds with public face + eye tracked avatars, and about 80% of the avatars were furries or just memes.
I ended up buying a rigged one instead.
This is a great video. I had a Quest Pro for a bit for the eye tracking, but other factors had me go back to my Index, and Quest 3, but I really do miss the eye and face tracking. I have been interested in the DIY eye tracking, but I keep putting it off
Goddamn! He started eyetracking at 14? Here I am at 41. I've got a lot of catching up to do! That's cool though.
YES, making the gestures is so hard to forget. After getting face tracking I still try to make the face gestures XDDD
I am fairly sure the Quest 4 will have facetracking because of quest pro's appeal for facetracking and the apple vision pro's face tracking, so I'm waiting for the Quest 4 to see if they have face tracking with a nice headset at an affordable price!
hope the surgery went well!!
I wish you best of success with your surgery!
FacetrackVR is so cool! I made slimes this winter! Now I can build eye tracking too!
I'm really excited to see where this goes; while I really want facial expression tracking I wouldn't really be able to afford it currently - and hoping that headsets eventually all upgrade to track expression. I can only imagine that soon there will be programs run to block out tracking/streaming data to the companies themselves, but I am hopeful.
I hope your surgery went well!
I would love to see full face tracking natively added to newer and modern headsets, especially in the context of AR and MR, where if they really want XR to be able to substitute in things like office meetings and social experiences, the fidelity is absolutely required. this is especially important for things like eyebrow movement which not all eye tracking systems support, foveated rendering for high fidelity applications, and look-based control systems.
I’ve been holding on before building any DIY systems simply because they are not capable of supporting all the features I would like, and some methods involve purchasing hardware which has since been discontinued.
amazing video, absolutely loving it, I just wish that foveated rendering would have been talked about more. I think it is an amazing way to enhance vr performance with literally no downgrade in visual fidelity
Glad facetracking is getting more awareness, PHIA your tracking looks so good. Can I ask your set up?
As Randy Newman puts it, "It's a jungle out there!"
I love it when a silly person does silly face gestured at me (It gives me nightmares in reality)