I always wondered why none of the face tracking stuff was being developed for android, and now I know! I'm honestly not even that super interested in face tracking, but at least now I know what's up if I ever do decide to go that route! You make an excellent teacher btw! Thank you for getting down to the nitty-gritty for why stuff does and doesn't work XD
Thanks again for letting us no, theres a free option and an expensive one, and even if you don't know how to use some of the free options, theres still literally always a work around. Might barrow a friends old iphone to try face tracking but I honestly just use VSeeFace's synthetic gaze and put my body movement up high so that when I move s,ight his head turns and so do his eyes, and expression hotkeys for expressions. ily
Thanks for the other options. A quick dig suggests kinect would be the best option if i wanted to go with it. Originally designed from the same technology base, better suited to my platforms of choice. For me it's not even really about the money, it's about having to deal with that ecosystem.
@@TenshiKisai Agreed. That's what I was implying with "designed from the same technology base". I haven't dug enough into the technology to know how close together the Kinect and FaceID really was but it seemed obvious that they both came from Primesense (Kinect via a partnership agreement, FaceID via acquisition).
I know this is fairly old and might be outdated, but iv been digging for so long of if I need to look into an Iphone, what the options are and if it has to be iphone WHY, and after dozens of videos this is by far the best explanation both in information, and delivery.
In theory, since google develops Android, they could develop the framework for it and then all other android phones use it regardless of the hardware manufacturer, but like you said there's a hardware side as well.
Exactly, and the fools who make Android phones didn't think that hardware was worth the extra $50 despite the fact that supposedly face ideas objectively more secure than a fingerprint. This is really sad, Google can't force their hardware partners to do anything because they would get hit with an antitrust lawsuit. Apples literally only able to get away with what they do legally because they're the minority, the minute they cross over that threshold is going to be an absolute legal nightmare for them.
Its a hardware issue. The iphone X and up use a front facing depth sensor. Some androids may now have depth sensors but apple also bought faceshift several years before the iphone X. Its what makes all that stuff work. With modern ai tech some android phone maker could definitely add similar functionality with just RGB video if they wanted.
I'm glad I came across this video. I knew that the facial scanning was more or less "baked in" when it comes to the iPhone X and up. What I didn't know about was the other forms of imaging, particularly in the IR spectrum; I just thought it was all baked into the silicon (I imagine there's plenty of other stuff in the silicon, but I'm getting way ahead of myself here). However, when it comes to getting a used device as opposed to a new one, I still ended up getting a brand new iPhone 13 Mini. I have several reasons why I went this route rather than getting a used device: 1. I don't have the money to buy something used up-front. Instead, I have three phones (yes, I have three phones) through my carrier, of which I have updated two of them. The third, I went ahead and "upgraded" to the iPhone. Now, I get to pay like $40/month for two years. Yes, I had to pay a bit more up-front, but nowhere close to the entire cost of a used unit up-front. For VTubing alone, I will leave it as an exercise to the reader to determine whether it was worth the time and money to acquire a new unit, but I had additional justification, which I will touch on in a little bit. 2. I'm getting a product at nearly the very beginning of its lifecycle. Yes, I'm an early adopter, but since it's new, I get to get AppleCare if I so choose, as well as any warranty through my carrier. So I'm not all that worried. 3. While I'm primarily going to be using it for my VTubing and not in place of the Galaxy A20 it "replaced" (which is my business phone I use for IRL work), I have other justification for getting it: I'm a freelance IT consultant as well as a small hosting provider for some of one of my old job's clients, where I'm hosting DNS, web and mail. It's especially that last thing that I need to be able to know how to handle if a client uses an iPhone. It's a need that went unserved for the nearly decade and a half I've been doing the hosting gig. Plus I can help those who may be just getting into their iPhones or iPads, which will likely net me more up-front money than VTubing could. I could go on about how I use general-purpose gear that can be easily repurposed in case the streaming gig has to be eighty-sixed, but this should be enough for now. Anyway, thanks for that bit of info! I'm glad that the capabilities are there, despite the fact I seriously don't like Apple and its ways.
I'm gonna get a iPhone when I get my note 10+ paid off in December. I want face tracking. If only picking out a channel name and designing channel art was that easy. Now my die hard iphone buddy can collect on a bet we made. I'll have to eat the hottest pepper on the planet. THANKS A F'N LOT ANDROID
What about using some of those phones with a mount and basically using them as a webcam? using the rear cameras instead of the crappy single front camera. I tried it with my Xiaomi Redmi Note 4X ( a middle-low spec phone from 2017) and it doesnt work that bad.
me who never use my phone just spend 4 hours doing work on my avatar for having a ARKit tracking and realize that you are obligate to have a phone that cost +300$ I hate my life
I'm curious, has anyone tried using the Galaxy S10+ for Vtube Studio? It has dual front facing cameras, with depth sensing. You can get refurb on amazon for a hair over $200, so if it can also face track well that's another option. It's more affordable, but also is a smartphone as well, and if you're an Android user it would make for a good option if it can track face well.
unfortunately, no income, trying to just make it only as a streamer, doesn't really give you a way to make 500+ dollars for a used iphone x. the root of a lot of problems, is the fact that phones are way too expensive, and should have their pricetags cut in-half, there is no adequate reason why a phone needs to be that expensive other than purely greed.
Is there any reason why this would have to be done with a front facing camera? Most of these android phones do have IR for image stabilization and flash detection in the rear camera displays but those rear facing camera's are usually much better quality options. Do these apps only allow front facing camera options?
and as for vseeface, i've used it for a year, and I have seen no actual face tracking, only things you can mimic by setting up gesture keybinds, and simulated mouth opening and closing dependent on your speech itself, not entirely on your lip/facial movement.
I'mma just wait until either a software on windows or android shows up and fixes this problem (the backside of the phone has those, why must it be the front?). getting a stupid expensive iPhone just for this isn't worth it. It's cool as heck, but not worth fueling that evil corporation.
You'd want to modelise glasses of your liking and then select the AR_T_head_1, then select your LR_T_glasses_1, then depending on the program you'd essentially want set a parent constrain in XYZ translations and rotations lock. If you were to merge them to the mesh of your character it very much could fuck with the blendscapes and visemes since the data is stored into the vertex numbers. And if you add more polygons, a.k.a more vertex, those number changes, so a few verticles of a blendshape can offset themselves and just break the whole blendshape set :p
Your channel, knowledge and tutorials are the best!! I just followed your vsee face tutorial with waidayo, it worked for me but I don’t have any head or body movements just mouth tracking for some reason. Maybe because I don’t have a working webcam in my laptop, so it only translates from the iPhone to vsee face. 🤷♂️. I want to know if you know any easy way to translate talking to unity? I can get vroid or even face rig characters into unity no problem and they come with blendshapes amazing… but I haven’t found a productive way to animate the blendshapes yet but it seems it would be so easy in unity because all these vtube software do it and many are made with unity…go fig. Of course you can spend hundreds and do it, but I’m broke lol. I did find a tutorial to rig a ready player me character to get face tracking but the setup slows down productivity and seems there might be a better way.. Do you have any thoughts?
Sorry when I say facerig I mean ready player me/Animaze characters the new version of facerig. I even found an interesting way to bring green screen footage in unity and key it out.. but if I recorded the face animations in Animaze with green screen and key it out in unity..just seems under productive
Fofamit i understand that youre trying to show all alternatives but its hard to take an argument against face tracking from someone who is using it during their argument
5:45 Am I seeing this right? 362.93 US Dollar/month?! (translation from Euro to US) WTF Also 4:55 I heard face rig doesn't use vrm models, so vroid users rip
Android phones will start to create an open standard in the nest year copying the apple arkit system there's actually a lot of money in it, the movie industry, vr, arch viz. But until they do apple is the only way to go for this. I can get an iphone X for $200 and plug it into unreal in less than 2 hours and have full face tracking of a metahuman. I would never buy an iphone for any other purpose, but it really works. Use the tool that works until android manufacturers catch up. They will eventually. Everyone wants avatars that emote properly. good video.
I want face tracking cause Im an expressive person and Im like I want to show my expression but then its like well shoot why everything hard for my small brain idk where this comment went
I think you have a severe lack of understanding on how programming with android works. Developing face tracking for android would relatively be simple for developers, there's only a couple different OS variations for androids and the only people who use custom ones are those that know how to do it. Developers would only need to develop a face tracking program that works for the more common OS variations. Not hard.
my phone have only 1 camera lmao Webcam can do the thing better, due to wider camera lens, better motion, better framerate (I have tried it with my Realme c21y, one camera and that small camera, if i moved bit more left or right, motion has lost track :D)
So how do I get a free Macintosh so I can actually use that iPhone? Apple refuses to allow any development on Linux or Windows. Also, free development license, since Apple charges for even developing for yourself. Apple is *evil*. I will take less precise face tracking over buying a pocket computer that artificially can't be used to compute.
Your comment isn't really related to the purpose of the video. Mainly people were and still are complaining that everyone is recommending using an iPhoneX or higher to do real time facial capture. There are RGB Solutions that are free that use a webcam (now) like XRAnimator which is not as good as the quality of tracking you get from an iPhone but is an option. There is also the NVidia face tracking but requires an RTX Card. Honestly, I still stand by my recommendation. You can get an iPhoneX used good condition for like $100 USD now in 2024. And you can get the facial data out from free applications like LiveLinkFace and Waidayo.
@@Fofamit plus the cost of a Macintosh computer plus $100 a year to enable basic features like running your own code. You're massively underestimating the costs.
@@Fofamit ... I must be. You seem to be saying to just buy an iPhone if you want face tracking because it's better at it. You seem to be saying the costs include only the cost of the pocket computer, neglecting the fact that to use it as a computer you need to also buy a Mac and a yearly developer license. What am I missing?
@@keiyakins The purpose of the video is to specifically address face tracking with VTubing. For example, sending facial tracking data from something (like a webcam or iPhone) to an application like VSeeFace a free Windows VTuber application. I am not talking about development at all in this. I never said to use the iPhone as a computer, or do development for the iPhone that is a completely different usecase which this video is not talking about.
My Twitter: twitter.com/fofamit
i had no idea u had a twitter or a twitch! thanks for sharing! lol "ty for coming to my TED talk" oh fofa :P im glad i found you
the best thing being brazilian and watch your videos/streams is: fofa is cute in portuguese lmao
I always wondered why none of the face tracking stuff was being developed for android, and now I know! I'm honestly not even that super interested in face tracking, but at least now I know what's up if I ever do decide to go that route!
You make an excellent teacher btw! Thank you for getting down to the nitty-gritty for why stuff does and doesn't work XD
I'm patiently holding out hope there'll be a $100 version someday in a stick or webcam form with just the essential cameras for pc face tracking
like a LeapMotion for face tracking :>
@@Rhodair You might be able to get the Vive Face Tracker to work without an attached headset for this.
Intel’s RealSense looked like a possibility, but last month they announced they are “winding down” its RealSense computer vision division.
Slinging nux out at the end just killed me. I totally forgot he got that new vtuber avatar
This is actually a good idea. Making presentation and using model avatar at the same time. Very creative. I love it.
but fofa why iphone no android >:(
ty for the video it's very good for people who're confused or don't understand
Thanks again for letting us no, theres a free option and an expensive one, and even if you don't know how to use some of the free options, theres still literally always a work around. Might barrow a friends old iphone to try face tracking but I honestly just use VSeeFace's synthetic gaze and put my body movement up high so that when I move s,ight his head turns and so do his eyes, and expression hotkeys for expressions. ily
i just realised I'm watching a very good PowerPoint presentation
thunbs up idk how to do the emoji
Thanks for the other options. A quick dig suggests kinect would be the best option if i wanted to go with it. Originally designed from the same technology base, better suited to my platforms of choice. For me it's not even really about the money, it's about having to deal with that ecosystem.
Apple owns the technology (Primesense) from the original Kinect, which is where FaceID came from.
@@TenshiKisai Agreed. That's what I was implying with "designed from the same technology base". I haven't dug enough into the technology to know how close together the Kinect and FaceID really was but it seemed obvious that they both came from Primesense (Kinect via a partnership agreement, FaceID via acquisition).
I mean cant they use.... the other side of the phone? Like - it has many cameras on the backside.
I know this is fairly old and might be outdated, but iv been digging for so long of if I need to look into an Iphone, what the options are and if it has to be iphone WHY, and after dozens of videos this is by far the best explanation both in information, and delivery.
In theory, since google develops Android, they could develop the framework for it and then all other android phones use it regardless of the hardware manufacturer, but like you said there's a hardware side as well.
Exactly, and the fools who make Android phones didn't think that hardware was worth the extra $50 despite the fact that supposedly face ideas objectively more secure than a fingerprint. This is really sad, Google can't force their hardware partners to do anything because they would get hit with an antitrust lawsuit. Apples literally only able to get away with what they do legally because they're the minority, the minute they cross over that threshold is going to be an absolute legal nightmare for them.
I say it still can be done in Android cuz Snapchat android can capture face and lips movement and Samsung AR avatar can also.
Of course it can.
i'm an android user :(
i just don't have an iphone because they are stupidly expensive.
Its a hardware issue. The iphone X and up use a front facing depth sensor. Some androids may now have depth sensors but apple also bought faceshift several years before the iphone X. Its what makes all that stuff work. With modern ai tech some android phone maker could definitely add similar functionality with just RGB video if they wanted.
I'm glad I came across this video. I knew that the facial scanning was more or less "baked in" when it comes to the iPhone X and up. What I didn't know about was the other forms of imaging, particularly in the IR spectrum; I just thought it was all baked into the silicon (I imagine there's plenty of other stuff in the silicon, but I'm getting way ahead of myself here).
However, when it comes to getting a used device as opposed to a new one, I still ended up getting a brand new iPhone 13 Mini.
I have several reasons why I went this route rather than getting a used device:
1. I don't have the money to buy something used up-front. Instead, I have three phones (yes, I have three phones) through my carrier, of which I have updated two of them. The third, I went ahead and "upgraded" to the iPhone. Now, I get to pay like $40/month for two years. Yes, I had to pay a bit more up-front, but nowhere close to the entire cost of a used unit up-front. For VTubing alone, I will leave it as an exercise to the reader to determine whether it was worth the time and money to acquire a new unit, but I had additional justification, which I will touch on in a little bit.
2. I'm getting a product at nearly the very beginning of its lifecycle. Yes, I'm an early adopter, but since it's new, I get to get AppleCare if I so choose, as well as any warranty through my carrier. So I'm not all that worried.
3. While I'm primarily going to be using it for my VTubing and not in place of the Galaxy A20 it "replaced" (which is my business phone I use for IRL work), I have other justification for getting it: I'm a freelance IT consultant as well as a small hosting provider for some of one of my old job's clients, where I'm hosting DNS, web and mail. It's especially that last thing that I need to be able to know how to handle if a client uses an iPhone. It's a need that went unserved for the nearly decade and a half I've been doing the hosting gig. Plus I can help those who may be just getting into their iPhones or iPads, which will likely net me more up-front money than VTubing could.
I could go on about how I use general-purpose gear that can be easily repurposed in case the streaming gig has to be eighty-sixed, but this should be enough for now.
Anyway, thanks for that bit of info! I'm glad that the capabilities are there, despite the fact I seriously don't like Apple and its ways.
This was very educational
I'm gonna get a iPhone when I get my note 10+ paid off in December. I want face tracking.
If only picking out a channel name and designing channel art was that easy.
Now my die hard iphone buddy can collect on a bet we made. I'll have to eat the hottest pepper on the planet.
THANKS A F'N LOT ANDROID
There is an android or pc alternative for hand tracking to vseeface ?
Elite motion i think it's called
Some expensive hardware for handtracking
I think VUP in steam also reads facetracking with webcam and is free. Its not great like iphone but still has some cool features.
What about using some of those phones with a mount and basically using them as a webcam? using the rear cameras instead of the crappy single front camera. I tried it with my Xiaomi Redmi Note 4X ( a middle-low spec phone from 2017) and it doesnt work that bad.
Thank you for sharing this!! I love all of your vids and I can be a vtuber because of you! Thank you!!
me who never use my phone just spend 4 hours doing work on my avatar for having a ARKit tracking and realize that you are obligate to have a phone that cost +300$
I hate my life
I'm curious, has anyone tried using the Galaxy S10+ for Vtube Studio? It has dual front facing cameras, with depth sensing. You can get refurb on amazon for a hair over $200, so if it can also face track well that's another option. It's more affordable, but also is a smartphone as well, and if you're an Android user it would make for a good option if it can track face well.
unfortunately, no income, trying to just make it only as a streamer, doesn't really give you a way to make 500+ dollars for a used iphone x. the root of a lot of problems, is the fact that phones are way too expensive, and should have their pricetags cut in-half, there is no adequate reason why a phone needs to be that expensive other than purely greed.
Is there any reason why this would have to be done with a front facing camera? Most of these android phones do have IR for image stabilization and flash detection in the rear camera displays but those rear facing camera's are usually much better quality options. Do these apps only allow front facing camera options?
Just commenting for whenever some explains why. I'm also really curious about this
Can you make a detailed video on your gear?
This was a really smart and informative video, easily difestable great job 👏
would I need an actual plan for the iphone (like data plan & monthly fees) or can the program work as a plug and play kinda thing?
and as for vseeface, i've used it for a year, and I have seen no actual face tracking, only things you can mimic by setting up gesture keybinds, and simulated mouth opening and closing dependent on your speech itself, not entirely on your lip/facial movement.
I'mma just wait until either a software on windows or android shows up and fixes this problem (the backside of the phone has those, why must it be the front?). getting a stupid expensive iPhone just for this isn't worth it. It's cool as heck, but not worth fueling that evil corporation.
SOLID Advice. thank you
I appreciate this video!
Ill look into getting an iPhone when I upgrade
Very cool Very informative
thanks
is there a way for you to teach us how to add glasses to your model because I'm learning this for a project and saw that your model uses it.....
You'd want to modelise glasses of your liking and then select the AR_T_head_1, then select your LR_T_glasses_1, then depending on the program you'd essentially want set a parent constrain in XYZ translations and rotations lock. If you were to merge them to the mesh of your character it very much could fuck with the blendscapes and visemes since the data is stored into the vertex numbers. And if you add more polygons, a.k.a more vertex, those number changes, so a few verticles of a blendshape can offset themselves and just break the whole blendshape set :p
Your channel, knowledge and tutorials are the best!! I just followed your vsee face tutorial with waidayo, it worked for me but I don’t have any head or body movements just mouth tracking for some reason. Maybe because I don’t have a working webcam in my laptop, so it only translates from the iPhone to vsee face. 🤷♂️. I want to know if you know any easy way to translate talking to unity? I can get vroid or even face rig characters into unity no problem and they come with blendshapes amazing… but I haven’t found a productive way to animate the blendshapes yet but it seems it would be so easy in unity because all these vtube software do it and many are made with unity…go fig. Of course you can spend hundreds and do it, but I’m broke lol. I did find a tutorial to rig a ready player me character to get face tracking but the setup slows down productivity and seems there might be a better way.. Do you have any thoughts?
Sorry when I say facerig I mean ready player me/Animaze characters the new version of facerig. I even found an interesting way to bring green screen footage in unity and key it out.. but if I recorded the face animations in Animaze with green screen and key it out in unity..just seems under productive
I still have iPhone XS Max. Will that work or is it single front facing camera??
I'm looking forward for when I upgrade my iPhone from a 8 to a 13
Any good resources for learning the apps for these?
I’m not looking to be a vtuber, just do proper rigging
Iphone series?
Fofamit i understand that youre trying to show all alternatives but its hard to take an argument against face tracking from someone who is using it during their argument
good discussion :3
Great debunking!
Hey fofa... Pls make a video About LIVE2d
5:45
Am I seeing this right? 362.93 US Dollar/month?! (translation from Euro to US) WTF
Also 4:55
I heard face rig doesn't use vrm models, so vroid users rip
I just found out Metro no longer offers a free phone upgrade every year/6 months ; w ;
I'll get iPhone with my tax return
Android phones will start to create an open standard in the nest year copying the apple arkit system
there's actually a lot of money in it, the movie industry, vr, arch viz.
But until they do apple is the only way to go for this.
I can get an iphone X for $200 and plug it into unreal in less than 2 hours and have full face tracking of a metahuman.
I would never buy an iphone for any other purpose, but it really works.
Use the tool that works until android manufacturers catch up.
They will eventually. Everyone wants avatars that emote properly.
good video.
hey fo im having trouble unfavoriting a banned avatar in vrchat , do you know how to?
is the iphone 8 as good as the x for vseeface?
Can we use coustom cast app
Whts ur suggestion
which type of mobile we want use for a facial motion capture pls comment mobiles name too pls bro
I want face tracking cause Im an expressive person and Im like I want to show my expression but then its like well shoot why everything hard for my small brain
idk where this comment went
Relatable
What about webcam with TrueDepth? I really want to compare this with iphone
The Azure Kinect I think had that, honestly there just isn't enough market for devs to care.
@@Fofamit Never heard about it! Sadly $399.00 isn't very cheap
@@WanderyenErin You can buy an iphone x for like $150 USD
everything is expensive TT^TT
But can you do this using a Kinect Webcam? It has the same technology
I talk about this in the video
Cries in has iphone 5 and 4s T-T
Man, it's annoying android will copy removing the headphone jack and not including charger cables ect. But they won't do this
It really do be money
I think you have a severe lack of understanding on how programming with android works. Developing face tracking for android would relatively be simple for developers, there's only a couple different OS variations for androids and the only people who use custom ones are those that know how to do it. Developers would only need to develop a face tracking program that works for the more common OS variations. Not hard.
Please let me know when you're Android face tracking application comes out :).
Who else have iPhone here? iPhone The best
my phone have only 1 camera lmao Webcam can do the thing better, due to wider camera lens, better motion, better framerate (I have tried it with my Realme c21y, one camera and that small camera, if i moved bit more left or right, motion has lost track :D)
*Sad in poor* yep
298 euros per month??? It must be cheaper to pay for surgery to become a 3d model
the problem is i'm not as entertaining as nux😅
even virtual avatar have to thot her way up.
android bc like OOOOOOOOOOOOOF
So how do I get a free Macintosh so I can actually use that iPhone? Apple refuses to allow any development on Linux or Windows. Also, free development license, since Apple charges for even developing for yourself.
Apple is *evil*. I will take less precise face tracking over buying a pocket computer that artificially can't be used to compute.
Your comment isn't really related to the purpose of the video. Mainly people were and still are complaining that everyone is recommending using an iPhoneX or higher to do real time facial capture. There are RGB Solutions that are free that use a webcam (now) like XRAnimator which is not as good as the quality of tracking you get from an iPhone but is an option. There is also the NVidia face tracking but requires an RTX Card.
Honestly, I still stand by my recommendation. You can get an iPhoneX used good condition for like $100 USD now in 2024. And you can get the facial data out from free applications like LiveLinkFace and Waidayo.
@@Fofamit plus the cost of a Macintosh computer plus $100 a year to enable basic features like running your own code. You're massively underestimating the costs.
@@keiyakins I think you're having a misunderstanding of what I am saying and the video.
@@Fofamit ... I must be. You seem to be saying to just buy an iPhone if you want face tracking because it's better at it. You seem to be saying the costs include only the cost of the pocket computer, neglecting the fact that to use it as a computer you need to also buy a Mac and a yearly developer license. What am I missing?
@@keiyakins The purpose of the video is to specifically address face tracking with VTubing. For example, sending facial tracking data from something (like a webcam or iPhone) to an application like VSeeFace a free Windows VTuber application.
I am not talking about development at all in this. I never said to use the iPhone as a computer, or do development for the iPhone that is a completely different usecase which this video is not talking about.