Head Tracking for Desktop VR Displays using the WiiRemote
Vložit
- čas přidán 20. 12. 2007
- Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space. By Johnny Chung Lee, Carnegie Mellon University. For more information and software visit johnnylee.net
- Jak na to + styl
I'm honestly mad that your concept never blew up. It is basically drowned out now with VR being a thing. This was mind-blowing for 15+ years ago. It's bonkers no bigger tech company snatched or approached you about this.
I unfortunately can't use VR for long before getting motion sick, and I still use this method at times just because the immersion (except for seeing depth) is still there.
It is being picked up by a program that makes a joystick out of the input which is picked up in a different program for faking TrackIR functionality.
When I don't want to keep using the batteries in the Wiimote and the LED's I wear an active 6-axis gyroscope that is connected to an Arduino Leonardo and is picked up the same like in the first example.
It's interesting to see how this indeed didn't blow up as much as it could've...
Look up something called TrackIR, fairly sure this was inspired by that (TrackIR started selling their products a few years before this video was made) and while that system used to cost a pretty penny they've actually sorta 'finished development' for quite some time now (their 'latest device' is actually 14 years old) and price keeps getting lower because they can manufacture them cheaper now, so these days its 'only' around 200 bucks (depending on where you shop ofc) but even today its still a really cool system (essentially a better version of this, with official support in a lot of sim racers etc)
He was on the Xbox Kinect team.
.... do you know who Johnny Lee is...? He most definitely got snatched because of this
Dude wasn't snatched by Nintendo, Microsoft got him and he helped develop the Kinect. There are some kinect side-projects that were VERY OBVIOUSLY Johnny Lee directed that made use of light-guides to replicate his monitor-as-window trick without any external hardware at all.
14 years ahead of its time. it's even more impressive that it doesn't need stereo vision to trick the mind. Only parallax was needed.
It doesn't need stereo vision because the camera can't record stereo. With your own eyes or with a stereoscopic camera, this trick wouldn't work so simply.
@@Mr_Yeah Incorrect. The 3D effect is created by moving and scaling the objects on screen relative to the position of the viewer's head. When the head tracking is not enabled, the objects are static.
If you were looking at this in person, with the infrared glasses on, it would look exactly the same as this. If it didn't work, there wouldn't be any point to this demonstration.
It’s still ahead of its time 14 years later. Nothing has been made of this very cool idea.
Buuuut, if you had a display capable of 120fps you could have 2 players wearing active shutter glasses each getting 60fps that could have their own perspective. Player one gets frames 1, 3, 5…. Player two gets frames 2, 4, 6….
@@Mr_Yeah I tried it at home with an even cruder setup using asus xtion depth camera for headtracking. Believe me when I tell you the effect is absolutely amazing. Much better immersion than plain stereo. Comes really close to VR headsets, but without - well - the clunky headsets.
14 years later and still the effect blows my mind. Like seriously how is this not a thing
It kind of is, now, the core concept is how lighthouse tracking systems work in modern VR headsets so when you move your head the game mirrors that, except they have the added benefit of inherent stereoscopic 3D and full picture immersion that the monitor couldn't provide. Sure, with this, when you move you get that impression of 3D, but your eyes are still basically seeing that non-stereoscopic image on a regular TV, if you stop moving your head you stop really perceiving the 3D effect. Whereas with a headset the different screens for each eye always give you a stereoscopic image for 3D immersion even when at rest.
It really surprises me how important stereoscopic vision is to people.
It matters, it helps you thread needles and other fine motor vision problems. But it's not a hard requirement for 3D vistas for example.
@@BDNeon Acer just released a glassesless 3D display with eye tracking (called Spatial Labs, you can buy it right now). All they have to do is implement head tracking correlating to camera motion then boom, HT+Stereo and NO glasses since it just tracks your irises (which is essentially what the wii sensor bar did with 2 points of IR light 25 years ago)
Dear Johnny,
You inspired me more than decade ago to leave my dead-end job and start a tech career. I hope you're still having fun building cool things, wherever you are nowadays.
Please post more.
Sincerely,
A, lifelong, recently subscribed fan.
Man it made me happy to read your post. I show this video to all my engineering friends to be like... here is a thing you could mess with... in your house!
How have your tech carrier turned up?
Great 3D technique. No glasses, no 120Hz display, no polarisation, viewable by people with one eye or vision problems, no '3D sickness'... and the effect is flawless.
So crazy that he just inverted the intended uses and it made the experience that much more incredible. Brilliant stuff.
7 years ahead of your time eh? :)
10 years now
I was surprised similar product is releasing only this year on kickstarter...
11 years now
Which one?
12 years now kkkk
this is the coolest thing Ive seen on youtube all year
+Bobus Touchdown you should have seen it in 2007!
+Bobus Touchdown It is pretty damn awesome, what's cool about it is that you can see the illusion through the camera, and in VR you can't so i get what ur sayin'.
But today, you can buy a VR Headset for 9 usd! it's wireless and you can take it anywhere you want, and use it with any PC game you want.
+TMM Azrox What VR headset are you referring to Sir?
The quality options like the Oculus Rift are $600 USD (I paid close to $600 Canadian for each of my DevKits).
And the cheaper "headsets" (which aren't really headsets at all), like Google Cardboard, ViewMaster VR and the rest, start at around 20 dollars and get considerably more expensive than that. Albeit you *could* build one for cheaper and I'm sure you could find one of the cheap ones for less.
Regardless, none of them can be used with "any PC game". That just isn't the way these things work.
There are 3rd party drivers like VorpX and Vireio (used to be Tridef but they dropped their support when Oculus revealed DK2) that will let you play **some** PC games, but many more do not work than do, and those that do don't always work well or even close to it.
VR is awesome, there is no doubt about that; especially when it is done right (as with the Rift and the Vive), but games need to be built with VR in mind in order to be truly compelling in VR. And perhaps even more importantly, comfortable.
:)
It is like google cardboard but it works perfectly fine with full HD phones.
With Android you can use it for PC gaming so it's just awesome.
Yeah but it only looks good on the video. I tried it in real life and you can't fool your eyes. It looks two-dimensional unless you move around all the time.
This video has aged quite well, especially considering how similar this is to the outside "lighthouse" style tracking done in modern vr headsets.
This guy was ahead of his time. I remember setting up this home it really was amazing for its time! Now I'm so fortunate that i can play around with Vive which is the best VR out there now, This market will explode!
Not at that price - going to be a slow burn.
@@ManiLink well how about now hmmmmmmmmmm
Mani Patel yeah how bout now
@@connorzinn9033 Valve index buddy
NightcoreVoid I use a quest actually
Saw you in the Project Tango video, congrats on your success now 6.5 years down the road.
Watching this over my Quest 2 VR Headset. How far we have came, man.
Why didnt this become a thing!? I remember I saw your video when it came out 13 years ago and I was amazed! Now your video on homemade ventilators got recommended in my feed and I immediately noticed you as that genius ninentdo VR guy, haha. It's good to see you still innovate and bring fantastic contraptions to the world!
I'm wondering the same thing today.
Probably because it requires you to mount a thing to your head to work properly. Would be a neat thing for people who already wear glasses though.
VR actually became a thing and VR glasses became relatively affordable
@@TheLingo56 With the growth of VR and AR, I can see this making a huge comeback.
@@TheLingo56 It could also work via face tracking
BREAK THE TARGETS.
NEW RECORD!
+David Davidson *ahhh...*
TOR BYEAAAAAAHT
Still in shock there is a zelda fps on wii that is offical
16 years old and Im still talking about how great this video was! You started something incredible!
Wow......Nine years before....
Amazing!!!! even on a embeded youtube window it managed to give the impression of real 3D
I watched this 14 years ago and I keep waiting for head tracking on 2-D displays. How is this still not a thing? This would be so cool with the right developers.
I watched it 14 years ago to and built one and used his software was so much fun
@@gosimons it is, its called track ir
I am baffled to learn that what i am searching in 2020 has been beautifully explained by you sir 13 or more years earlier ..
It is extremely fantastic to work on this project.
WAY AHEAD OF HIS TIME!!! i saw this video years ago, but forgot it through the passage of time. i had a flash of memory that brought me back here, and im glad i chose to follow through. this is awesome
this guy is very smart. I wish Nintendo would hire him!!!
+marpa he works for Google he runs the Google tango project
+marpa actually this is Nintendo NX technology.
+HISTERIMON well the acquired a patent for it. still no word if they will implement it. let's hope
I hope not! When companies hire Geniuses, they become dumb because Management is about money not innovation!
Listen to some Guy Kawasaki stories, he used to work at Apple.
+Jiv V I think each console/corp brought something good to the scene, MS/XBOX brought online gaming experience, Sony/PS supported third party developers eased the production of great titles.
of course he's working at google now
He is genius!
7 years ago! Johnny Lee you are a bad man!!
love your work
Of all the projects involving the Wii Remote on the internet, this one probably takes the cake. And I even learned something about how the infrared sensor works on the Wii. Good job explaining and showing it to us as well. Thinking of the monitor or TV bezel as a window frame through which you are peering into a scene makes the 3D perception that much more believable, because rather than become an artifice, the frame actually participates to the scene being viewed.
still can't believe games weren't made like this. i wanna do it
This is amazing, it's such a simple thing if you think about it but no one has never implemented it in a mainstream application, instead we created vr headset. damn this video is from 2007 i wish i had seen it 10 years ago(mmmm i was 8 years old at the time so i don't know what i could have done with it) i now study IT and this blows my mind, 1st because it's from 2007 and i seen this just now randomly trying to find a way to gather 3d coords of a quadcopter flying in a room and second because is such a simple thing that has a lot of potentiality but no one has ever used it.
I hope you've gone on to incredible things! It's insane to think that I was still a kid when you created this. The world has changed so much, and you were so far ahead of your time!
Bro. I watching this in 2021. You are just too ahead of the time. Love you man
8 years is past. Where is my games for this technology ? NVidia 3D Vision 2 is perfect base for such kind of games. Why nobody used it ?
+Gigagamer74
You mean the Nintendo DS2?
Chris Katko I mean PC platform, with big 3D monitor.
+Gigagamer74 I think one issue is that you get a real popup effect only when you watch the screen with ONE eye.
If you open both eyes, then the screen looks flat without depth, because binocular view makes the brain realise that he is looking a flat surface.
I know that because I actually tried the system in the video.
So yes you are right, only a binocular system like 3D vision would benefit from such a system.
Or VR like Oculus Rift and HTC Vive, they use that already of course.
+Mirak M Oculus Rift and HTC Vive is my dream devices. But headtreacking+3DVision got a unique advantages. First - a mouse+keyboard control configuration, fully functional in game mouselook, time-tested FPS 3D shooter gameplay and light glasses for 3D in game picture. And second - zero lag time, which for FPS genre with a mouselook gameplay is extremely important. So the headtreacking+3DVision is not a competitor for Oculus Rift and HTC Vive. Headtreacking+3DVision got it's own application area.
Nobody uses it because it does not work. It only looks like 3D on a 2D video. You can try this at home, just need a wiimote and a PC. Prepare to be disappointed.
You actually could do this with more than one person if you had glasses and a TV with a polarization filter.
This video was made in 2007, and consumer-oriented 3D televisions didn't become possible till about 2011. Though we could do this now.
Leland Batey any CRT would be able to do what 3D tv's do. It's about the Hertz not the '3D'
ethicks Some use polarised light though.
Or possibly (though this would be incredibly finnicky,) do microadjustments to make it stereoscopic.
PROPS TO YOU SIR! This isnt an oculus rift or anything, but a whole different type of VR. It's amazing to see something like thos and realize how simple it really was the whole time haha. Thanks for your work
That was an OUTSTANDING demo. Here I am 15 years later seeing this for the first time. Carnegie Mellon...not surprised.
It's the end of 2019. Still an amazing stuff!!👍👍👍
I saw your video when you first released it. It has stuck in my mind ever since because there are still no games with this specific rendering technique. I was inspired to replicate this in Nintendo Labo VR for the Switch, and to my surprise it worked. Though without rending the camera across a moving and rotating plane, since Labo is limited in its rendering options. I posted a video with step by step instructions to use the Labo Snorkel to position and rotate the camera in the game.
Yeah, crazy it's been now 13 years and this still isn't really a common thing. I too saw this when it came out. I was only a kid, and I homebrewed my wii and installed his demo immediately. Obviously VR has come since then, but if 2D games used this, it would be a great in between for people who don't want to do full VR but still want games to feel like they are looking into a real area instead of watching a 3d image on a 2d surface.
@@theboxofdemons I use this for a simulator that I "play" (I don't really consider it a game :P) and it works absolutely wonderful, even with a small screen it feels like you are immersed much more (without actually using a headset of some sort).
I am curious to try the function out on a videoprojector, but I am afraid the WiiMote might get blinded by the projector and will not see the infra-redleds any longer.
(And with the idea of being able to play closer to the projector screen hopefully being able to make more use out of rotational movement of your head when looking for oncoming traffic in the simulator).
@@theboxofdemons it is now :) czcams.com/video/kscIp_xuXuo/video.html&ab_channel=Roomality
Hi. I am trying to make something like that. But in other area. If you are interested in cooperation. Please replay :)
Johnny Chung Lee is now a legend, I have taken a screen shot and I will be printing & framing it and putting it on my wall in my Asian section next to Kim Jong il. Hopefully one day I will get it autographed. We love you Johnny!
I remember watching this when I bought my Wii 10 years ago. Here I am 10 years later gaming on the HTC Vive.
This is amazing! This should be the NX.
Wow i remember watching this a looong time ago. This innovation was way ahead of it's time.
I'm glad I'm not the only one who sometimes still remembers this. So ahead of its time.
Okay, it has been SEVEN years since this came out.
when are we going to see this tech in real life?
Another 7 years gone.
Realistic virtual windows? So you can see through it like it was a real window? That would be a good application
This is 7 years old?!? Why isn't this in products.
sam183796 now 11
@@JJRicks now 12
@@mateusrodrigues69 now 13
It actually was in production at the time you wrote this comment. This technique was used in the Oculus Rift DK1 in 2012 about 4 years after this video and is still used for VR headset such as the Steam Index.
Obviously they use super high quality cameras and many more infrared lights than the sensor bar's two but it's the same concept on steroids. The big difference is that instead of a TV, the screens are mounted to your face
@@Luigi64real Ehh sorta. I want to try this out
Lol, I remember finding this video when it was first uploaded. Glad to see it’s still up!!
I remember when this video came out. I'm still saying WoW! Fair play buddy. Pure genius! 🤘👍
I remember watching this back in 2007 and it's still just as mind blowing 14 years on. Where the hell is this technology?!!
6dof headsets like the Quest/Quest2 and PSVR work using similar concepts and it works pretty much like it does in the video. This guy was WAY ahead of his time!
@@Steelburgh Actually, at the core Steam VR is very similar to this technology, because it uses IR in almost same way.
Amen! What the heck???
Duuuuuuude, remember watching this back then! Great idea you had!
I remember watching this video back when it was uploaded and it blowing my mind!
The WiiU is an amazing system. I just hope that Nintendo and other developers don't overlook its obvious potential like they did with the Wii. This technology should have been adopted, and Johnny Lee should be running Nintendo's R&D Department.
HOW THE HELL IS THIS NOT A THING??
Genius! This looked so extensively more effective than any other current tech out there! Just another example of how we overcomplicate things, using gas instead of electricity, using engines instead of magnetic force, using glasses that screw with picture quality and overcomplicated systems to create a 3d effect.
Dam, Amazing video!
Found it 15 years later, but wow so advanced at the time
someone finally found a use for the wii
Xenoblade Chronicles, the best JRPG of last generation. And Super Mario Galaxy 1/2 the best games of last generation.
J1V4M35 and fragile dreams farewell ruins of the moon
+J1V4M35 I've had xenoblade for a long time but never finished it. Is it worth it?
Epic look at 3:50
is 2020 now. what a great time back in the day!
Excellent video! Appreciated your articulation and how well you explained the process and logic.
This video is so old now! Why have no game developers take this to heart! This is why film is considered a higher are then video games. We actually take movies seriously. Sad.
+David Humphreys Occulus Rift and Nintendo patented something with VR.
Well this isn't VR though. More like VRs little brother.But even if that's True that it wasnt utilized for that reason, id say it only shifts the blame from one group to another. this is still something we had the tech to enjoy 10 years ago. But nobody saw it's potential. Just kinda sad is all.
I remembered seeing this years ago. Came back after seeing the behind the scenes of The Mandalorian and the Oblivion movie.
They used a same technique with a video game engine and a gigantic LED screen, instead of using a green screen.
Much better than a 3d tv. No headaches. Only drawback is that only one person can
use it, but still very cool.
I remember that in 2008, 11 years from now I bought a WiiRemote just for testing this, and this was an amazing feeling. I don't play much games, but I can't believe that it's there no games using this (AFAIK). Or just head tracking with a any webcam, specially on smartphone where a front facing camera is standard.
who is here after MKBHD drop that Google Project Starline in 2023 !! 😂😂
OMG IM NOT THE ONLY ONE WHO THOUGHT OF THIS
This is *exactly* what I thought of.
izzy é nois! kkk
Tamo Junto! 99Vidas cade vcs?!
mano a parada é muito futurista
This is the more than intriguing. Realest holographic display
I have ever seen
thank you Johnny Lee iv been looking for a cheaper method for a vr headset.... i really like this concept i think this will work a lot better than most Diy head tracking available...
Funny thing, Nintendo just patented something like this. =P Hype.
Fireshrine oop too late...
Fireshrine The latest update for Bit! Boy! Arcade on the Nintendo E-Shop uses face tracking similar to this. Just on the New 3DS, though.
SilverWingedOne
cool.
and he never saw his games
I remember watching this as a kid, it's so sad this never became a thing. I just heard William Osman talk about it on his safety third podcast (from a week ago). We need more of this today
Just remembered this video after over a decade since last watching. so cool.
But this only works, if you watch it with just one eye. Two eyes would still see a moving 2D image on the screen. You need to combine this with a 3D TV to get the real imagination of looking through a window.
That's the problem with todays 3D cinemas: If you move your head, the image stays the same.
Eye tracking with deep learning seems to deliver better result today!
In the description it says Category: How to & Style because my boy Johnny is the most stylish genius you've ever seen
Absolutely legendary. I think my headphones got the head tracking update last year. So ahead of your time
Great video! Very informative and I really enjoyed the presentation. Hope with all the VR stuff now-a-days this becomes the next big thing in games and in entertainment in general.
This was incredible to see then, it is still incredible today...
this is actually the most amazing thing i've ever seen. Wow!
15 years after I saw this video for the first time, it's still blowing my mind.
hey, this is Hiro of HEROES 's, he travelled through the times to take back this technology! good job !
omg - just read about this in wired - way cooler than i could have imagined
I really liked the concept of the 3D viewing in this video for the Wii console. When the 3DS was first coming out, I expected the same type of technology so it would actually look like a fully rendered environment, rather than a one-sided perspective. If they were to do the same form of tech with the 3DS and Wii U, they should provide a form of eye detection so two images are only linked to either eye, depending on how far/close you are to the screen.
Well done Johnny you have come a long way and its so amazing you have stuck to your vision. I think project Tango will bring things we could not have dreamed of when you made this video thank you.
so you could basically play a show and feel like you're watching it happen outside your window. trippy stuff dude
Dr Lee, you're simply brilliant!!!
Man, it's been so long since I last saw this video. I was still playing on the Wii when I watched this.
I haven't really seen much of this technology being used since then, though...
Johnny Chung Lee you are a genius.
DUDE UR A GENIOUS!THIS IS 3D WITHOUT GLASSES AND WITHOUT 3D TV! guys we gotta do something!let's share this video and put it in our websites so a Nintendo developer can watch this!
12 years later, more relivent than ever
Damn. So much ahead of time
Some years ago i was thinking at this same concept.. its mindblowing seeing this working
Watching this in my Valve Index, how far the tech has come
I'm seeing a lady that's a physical therapist for soldiers/veterans (specialising in people who've had multiple concussions, trouble with balance and pain) and she described one of her tools as basically a laser pointer on the patients head to navigate a map on the wall. I instantly thought back to this video and how technology like this could be used to diagnose or potentially treat patients.
This demonstration is most impressive, this is 3D i can actually see :D
My brother was impressed with this as well. He has a problem with one of his eyes that makes him unable to see 3D like in headsets or with active glasses, and really liked this possibility, mainly because (to quote him) "This doesn't give me a headache."
Awesome work displayed here!!!
2023 Johnny Lee you were a genius I'm still using my Wii remote for vr
How ur using bro wiimote vr application is not opening on my pc
Amazing use of todays tech to put together tomorrows experience. Why is this not mass market/coming to market? Truly impressive work.....
How am I only just learning about this? Quasi-gimmicky though it may be, this is amazing!
You deserve some big credit, man!
GREAT work!
Johnny! awesome job and great showmanship -you've hit the bigtime and congratulations, a marvelous video and even more wonderful invention and also pretty hip (in the 40's they called it 'hep')
I'm so glad I was able to find this again. I remember being blown away by this. I'm still confused as to why this didn't catch on.
99% sure this was inspired by a system that started selling a few years before this video was made called TrackIR, they are still around, their latest device is 14 years old now (lols) but still super awesome, its essentially the same thing but with nice bespoke hardware instead of hacked together wii parts & that system has official support in a lot of sim racers etc. Funnily enough there's people discussing using this Wii hack with the TrackIR software in these comments :)
Great example and explanation. Thanks.
give this man a medal!!!!
Dude, you rock. After 3 years, I assume you got a really good job somewhere.