Does Tesla FSD Recognize Hand Signals? We Tested It!
Vložit
- čas přidán 6. 04. 2024
- This is a HW4 Model Y on FSD (Supervised) V12.3.3
Use discount code DirtyTesla to save 5% off premium Accessories from Jowua! www.jowua-life.com/dirtytesla
☀️Ordering Tesla Solar or vehicle? Use a referral code!
ts.la/chris41676
My favorite Tesla and CZcams Accessories: www.amazon.com/shop/dirtytesla (Amazon Affiliate Link)
10% off S3XY buttons with code DirtyTesla: enhauto.com/?...
Patreon: / dirtytesla
Use the totally FREE tools from DroneQuote to see potential EV and Solar Panel savings: dronequote.net/dirtytesla
⚡TESMANIAN: Get 10% off accessories with Tesmanian discount code DirtyTesla www.tesmanian.com/discount/Di...
Merch: dirtytesla.creator-spring.com/?
Use code dirtytesla for 10% off Tesla Accessories at scentwedge.com/dirtytesla
Twitter: / dirtytesla
Instagram: @DirtyyTesla
Music: / sammorykwas
Filming Equipment
Sony a6100: amzn.to/2BuqYwc
Gimbal for a6100: amzn.to/30xelNg
Rode videomicro: amzn.to/30bbZ4m
Rode Wireless Go II: amzn.to/35xV2mw
GoPro (films screen/road): amzn.to/3aiMsKt or www.insta360.com/sal/one_r?in...
GoPro media mod: amzn.to/2Ze2q6G
360 Camera: www.insta360.com/sal/one_r?in...
Camera mount for GoPro: amzn.to/46WThgg with amzn.to/3BXBtmb
Mic: amzn.to/383CuJO or amzn.to/2NjYSXu
Face/vlog style camera: amzn.to/3lWvFDD
Mount for face camera: amzn.to/34lEfjP
Tripod: amzn.to/335vV7D and amzn.to/2N6CiC1
As an Amazon Associate I earn from qualifying purchases. - Věda a technologie
Now having watched the full video it is pretty much confirmed that no hand signals are taken into account, but that it's instead a result of excellent object / trajectory tracking and understanding of intention / human behavior.
It’s not confirmed, just as been said at the beginning of the video, fsd is trained on recordings of people driving, I bet there aren’t many recordings that match this particular scenario, but there might be many where someone enters a parked car and waves on or traffic officer is using hand gestures to control traffic. In these situations Tesla might be able to determine what it’s supposed to do, Tesla doesn’t really understand what hand gestures are but it can detect patterns it learned.
@@marcin8112 entering a parked car is trajectory detection and intention understanding rather than hand signals, and if they wave you further while doing so, then that's the same motion / action as detected here.
I truthfully doubt it currently can recognise an officer from a regular person, but testing it with official handsignals rather than common ones would be interresting.
@@HTX-enjoyer I’d assume Tesla fsd ai looks at an image as a whole rather than its individual elements. It doesn’t detect trajectory and waving as seperate things, it just sees a series of images that include someone moving next to a parked car and waving their hand, which based on training data resulted in driver car moving forward.
At least that’s my assumption but I don’t actually know how their ann models work
@@marcin8112 This conclusion is also my assumption. However, I think they run a first pass on the image and detect the person which is it's own training model. That is, the person is interpreted from the images, then identified as a single critical object with a trajectory to avoid. Regardless of gesturing, running, being backwards, sideways, bending over, etc, the car's inference engine is first making that one object a set of variables with position, direction and speed that it will pass into the driving inference engine, a separate neural-network processing path.
To get there, they'd need to first train the image processor to handle gestures and turn that into an additional gesture variable they pass to the driving training to relearn. It's likely that's not a high value objective at this juncture because slowing is still something they'd want to do even if gesturing to drive-through .. thus more a subtle gain in efficiency from what it may already need to do (but a good one to be V5 ready).
Great test, btw, Dirty Tesla.
Just what I was thinking. FSD visuals show people and the direction they are facing/moving. Would be cool to repeat with the pedestrian turning, facing, and walking different directions.
Two other things you should (have) tested:
1. Stephanie stopping on the sidewalk, instead of in the street, and doing the same gestures / movements.
2. Same things at crosswalks, with and without entering the street.
1. is really good point, human on street means for a car obstacle that need to be avoided no matter of what.
Thank you for the video. Really great that you approached this in a "scientific" manner. You did confirm hand signals aren't being taken into account but you also confirmed how safe FSD is with pedestrians.
Why would Tesla say it does that if it doesn't. Not good.
@@rui569 Maybe they are recognized, but the behaviour is just the same?
edit: I mean that there might be some training data that includes waving etc. but not every possible situation. Actually that means: we do not know if it is really reacting to the hands. The camery definitely sees it; but only a very small amount of gestures might affect the behaviour at the current time.
@@darktemp_de at this point the system seems to be realy good in understanding intention but that's not the same has recognizing hand signals.
Yeah, great stuff. Had a hypothesis. Eventually tested it "scientifically" and proved it wrong. Now need to loop back and post results on X to the original tweet as well as link tweet to the testing.
Nobody seen the child size mannequins FSD runs over repeatedly
If it was taking pedestrian gestures into consideration people would start just trolling teslas in the road 😂
you could do the same to normal drivers
As long as the Tesla continues on like a normal driver and it doesn't pull over for a stranger 😂
people will do that anyway knowing the car will definetely break lmao
@@The1QwertySky I think the point is that EV and FSD haters will specifically mess with the car out of spite. Just like they try to stop and block Cruise and Waymo vehicles and they pull the plugs out of cars at Superchargers, and sometimes they even leave notes saying how they are afraid the car will overheat and catch fire. I'm not trying to say that this sort of thing is widespread, but these people do exist. Anti-EV and Anti-FSD sentiment is widespread. I think EV adoption in the U.S. will be slow for a few more years yet. I also believe that companies like Charge Point and Electrify America are not helping the situational all with many stations not working or poorly working which is giving the EV industry a bad name.
@@mikemccormick6128 People can not unplug a Tesla at a Supercharger unless you leave your car unlocked. If a Tesla is locked the charge cable is locked into the car. My car automatically locks when I walk away...
A very good experiment. Thanks to you and your wife for doing this!
So pleased it errs toward safety. I love it.
😂😂
I had this experience with mine. I'm on a 30-day free trial with the FSD 123. This is my first time I never tried the beta. My second trip was on a two-lane country road construction was one lane closed one lane open. I actually had to go into the oncoming lane from hand signals and a handheld stop/slow sign. The car stopped for the stop sign. When he turned the sign around to slow and pointed towards the open lane which was actually the oncoming lane. The car proceeded through the construction around the equipment and then went back into the proper lane at the end where the other flagger was pointing towards the open lane. I was really nervous but I left the car do its thing and it was amazing. It totally figured it out by itself and it wasn't acting weird It drove through it like a normal person. Thank you 😊
This is the type of quality content that I subscribe for
Your voice for the person commenting about the robbing scenario was priceless!!!! It sounds so much different from your regular voice
Well actually 🤓 ☝️
😂
Thx for doing this. I have a crossing guard near school and it basically ignores her hand gestures.
It definitely needs to learn how to respond to crossing guard or law enforcement officer or construction worker hand signals for those times when there is temporary human traffic control (school/event traffic flow management or work zone). As long as there is a human driver, that person can override in those scenarios… but if Tesla / Elon wants to achieve full robo taxis, those scenarios are common enough that the car needs to know how to interpret the guidance.
Don’t know why anyone would think hand gestures were included. Surprised Tesla even insinuated it did by not denying it. Even from the first video, it’s clearly a zero velocity situation.
Oooft, would love to see you come up with heaps more pedestrian/driver/traffic management people hand signal examples. One edge case that springs to mind: When the traffic sign people at construction sites forget to turn their sign around from "stop" to "slow" but are using hand signals to move you onward. Or the other way around.
👏👏Round of applause for the lovely assistant Stephanie. Putting her life on the line for SCIENCE😄
Great set of tests that show the hand signals have essentially no effect on the behavior of the car ... how about having Stephanie just standing near the road and then whips out a *real* stop sign, simulating what a crossing guard would do when they want to have children cross the street? Except don't do it at the intersection, but in the road where there is NO crosswalk - that would be a rather interesting kind of test ... but thanks for doing the video - great stuff! No matter what, FSD is super impressive - since we got v12.3.3, we've been really amazed at how good it is - with a few minor exceptions that I know will be addressed before the end of the year - like, learning how to enter a gated community, where you have to decide if you go through the guest gate on the left near the guard station or the resident's gate on the right side ... and of course, slowing down until the gate actually opens up! Even when I trick it into going through the resident's gate on the right, it just doesn't notice that the gate is down and needs a few seconds to register the sticker on your car so it can raise up and let you through ... but I imagine that is not a super-high priority item ... but for the day we enter robotaxi land, that will clearly need to be solved - thanks again for the video!
Cheers for the tests
Not surprised hand signals are not taken into account. Avoids this headline: "Pedestrian run over because he was waving away a fly."
it's a little more nuanced than that lmao, you can safely assume the pedestrian continuing to walk in front of the car/being in the way will supercede hand gestures and the car will stop
Thanks to you and your wife for doing this!
Thanks for the testing
I believe it is the gait/leg movement speed/direction/length that the car is sensing for pedestrian behavior.
this was a great experiment!
Tesla will be very thankful for the training material you both just provided 😀
Very interesting test! Thanks for doing this.
Never before have I been so early
I'd like to see a similar test with the pedestrian wearing safety clothing and using a "slow" and "stop" sign as are typically used in construction zones.
Great job on testing this!
Hello from germany, fsd is in coming!
Is that German for coming?
No that would be "kommt" 🙂@@josesammut9396
The EU wants to loosen some regulations. Whatever that means!
At the end of 2024 👏👏👏
Nice ❤
Excellent tests.
Excellent set of test scenarios.
I appreciate this video
Appreciate the methodical testing.
Your wife is VERY patient. She is a keeper.
Exceptional testing criteria 👍 This was a good way to go about proving / disproving the hand signal theory.
Hey Chris, Great content! Thanks! Greetings from Hungary!
Glad you liked it!
Great video. Thanks!
Knowing that this fsd has gotten so much better is so dope. Can’t wait to get a MY with fsd.
How would it handle flagmen and their typical directives??
Huge fan of you bro keep it up!
Just quick note she was on her cellphone. Depending on sensitivity of the AI it might have and influence on the car behaviour. Interesting and fun experiment. Kewl new stuff to test on FSD
Great video! Great way to learn about FSD for us non-Tesla-owners.
Good test
This is one of your “bestest” videos. I remember box man videos a few years ago. So glad that FSD supervised is now doing what we thought it should do 5 years ago. And we’ve got to thank the Tesla team for making progress. We just wish the progress was much faster.
What if she held a 'crossing guard' or roadworker stop sign and stood at the side of the road? and then try it in the road? And then spin the sign to say 'slow' and see if it reacts.
I like these tests, you should do one that shows how obstructed signs can be with beta still recognizing them
My family and I did this experiment tonight. I jumped out in front of the car, it stopped and I gestured for it to continue going and it did.
Thanks for being honest. But honestly FSD right now is incredible.
It is
Yep, this is the best FSD channel on CZcams.
Some thieves will put a woman in the road to make you stop and then they jump you and steal your car
Conclusion: FSD should be trained on dash cam footage from South Africa
😂
Do you know the recovery rate for stolen teslas?
It is either 100% or damn near it.
@@michaelwilkens6865 They could rob or kidnap the passenger instead.
Yeah kinda ruins the robotaxi idea for me. Nope nope nope
Great demo thanks to your wife too!
This was great!
I love her patience to help you with all the tests
Great experiment
Great video. It’s crazy that Tesla would outright lie about this.
Thanks for that. Really smart tests. While it looks like hand gestures aren't taken into account now, I predict they will be eventually once enough hand gestures are seen in training. Probably just not enough of it happening during training runs to affect the model weights yet but it should learn from hand gestures no differently than learning from stop signs, gates, etc.
I, too, have experienced the machine responding to human gestures many times. At the very least, it's trying to interpret human intent. Perhaps it's not 100% consistent, but it's there.
Far more likely that the teaching material has people stopping and waiving, so the interaction timing matches with how people react to FSD cars. So like someone steps out, then back in and waves. The reaction for this is timed, because of the process, in a way that it looks like the waiving does something.
You think you have experienced it. How do you know it was responding to the hand gesture and not the body motion? How do you know it didn't just happen to decide it had waited long enough? That is what this test was for.
@catbert7 I have over 1000 miles in V12. My wife and I have both seen it. A pedestrian starts to cross and then stops waving is on and the car goes. Was it the hand gesture or body movement? I don't know. I do know twice a driver with the right-of-way has motioned for my machine to go ahead, and it takes off. I'll leave it to you to judge.
@testi2025 Perhaps, however if this is truly a NN, who can say? What human knows what another human is thinking? How can any human, even the developers, know what is going on now in the mind of the machine? I say it is beyond human understanding at this point.
I agree that I feel like it's more than just the hand but the complete body motion interpretation of intent that the car has learned. I think the hand gesture is a secondary one that we recognize but the AI training has brought an understanding of the whole overall movement that determines intent.
What about those stop signs carried by construction workers and road workers? Would be cool to test that.
There are many videos showing FSD responding correctly to hand held Stop Signs. However where I live hand held Stop Signs aren't used. Its all hand gestures by police and road crews.
@@edhill8568 Construction companies should be allowed to innovate with a portable traffic signal, like they do in Europe. It would eliminate the need to pay a guy to stand there all day with a handheld stop sign, or a police detailer.
@@carultch You are right of course. The problem in Massachusetts where I live is the police have to approve any non police details for construction sites and they won't. It's ridiculous of course but they are protecting the income the police make.
@@edhill8568 I live in MA as well, and I'm well aware of that. I was shocked to not see construction company flagmen like I see everywhere else in the country. Only time I've seen construction company flagmen, is West of I-190 on 2-lane back-country roads, where police unions probably have less of a stronghold.
It is union rules that stop construction companies from innovating, and just using portable traffic signals. MA requires police detailers, and most of the rest of the country still requires human flagmen.
@@edhill8568 I have seen portable traffic lights used in NH, where a part of a road is closed and not currently under construction. Closures due to natural disasters, like flooding or landslides, that they still haven't assigned a crew to repair.
This is a really good unbiased investigation!
Thank you Ted!
Not sure about the US, but we have official hand gestures in the highway code in the UK. That said, I've had cause to use them recently. Maybe a third of people followed them correctly. Another third were really hesitant and wouldn't move for ages when being waved on. And the last third just ignored me all together, even with a stop gesture. So even if FSD reached the point where it followed hand signals SOME of the time, it would still be better than the majority of humans...
Awesome tests. At the intersection, I was thinking it should be away from the intersection. Great change up. 😊
lol at the stranded ev due to cold temperatures. 😆
You tried so many different ways that we wanted to see thankss
Hand wave is part of traffic communication. Should include it but only when its safe.
Nice test! It's plausible that the neural nets takes it into consideration a bit.
Great video. I’m sure they are thinking of this but we’re just not there yet.
Very satisfying
I've seen it yield to other cars twice, only to proceed immediately when the other drivers waved us forward.The timing was so prompt, it really seemed like it understood.
Yep, I've seen the same...
Yeah, I don't know why the Tesla account suggested that it's reacting to hand signals when it obviously does not. That's undermining trust and potentially dangerous. Really bad move.
Thanks for confirming this again.
It's not a binary thing, it's not beyond the capability of an end-to-end model to learn this, but I wouldn't think they've gathered any specific training data for it. But it's still possible it does take it into account in some situations. It's not like V11 where they'd have to train a "wave"-detector that's fed into the planner, not even Tesla can know for 100% sure whether this behavior is supported, unless they've meticulously documented which training cases has hand signals in them (which is tough when you have something like 30M video clips).
Hi Chris, big thanks for doing these tests and thank you to your wife for helping.
This helps show one of two things (or a combo of both), with both being just as bad IMHO. Either Tesla is oblivious to what does and does not work with their full NN version and actually believed your "evidence" and thus made their passive post which would lead one to believe it did work or they knew it didn't but made that indirect passive post implying it was accurate nonetheless. So you can pick between incompetence or lies. I personally believe it is a mix of both. They aren't sure what it does or not do (inherent in the nature of NNs), saw that people thought it did it and didn't care or check wether it was accurate or not and made their post. Since they didnt directly say it did it, they can pretend they didn't lie about it but knew very well how people would interpret their post.
Love it!
Thanks for doing this testing! It does seem that the car doesn't care about hand gestures and it just behaves around pedestrians differently when at an intersection since there are more variables to consider. One note - Stephanie being on her phone adds a variable that could be affecting the results. I'd be curious if the results would be different if she was just using ear buds to talk to you instead of holding the phone. My guess is no, it would be the same. But tough to say for sure with that added variable.
It would be really interesting to test similar things when there is a crosswalk. Bc one idea could be that the car was much more careful around the intersection because that's usually where crosswalks are located, even though there was none at this intersection.
Great test. We have the free FSD trial on our 3 but have never engaged it. Your tests look pretty good so maybe it is time to try it out. Thanks.
Definitely give it a try. It's very easy to take over if you don't like what the car is doing
Yeah FSD is 100% assessing traffic-impacting behavior only, not hand gestures, and that makes perfect sense. Imagine the endless phantom braking that would result from FSD misinterpreting all the gestures and movements pedestrians make while talking on the phone. It would be a nightmare. But that raises the question of why the Tesla account "confirmed" hand gestures. Maybe it was an intern LOL.
You have a lovely and supportive wife. Congrats on your success and best wishes to your family.❤ keep up the good work. You are great at testing Tesla FSD 🍀
Wow. Just wow. Amazing hey.
I don't think the waving does anything, it is just the presence of a person being there.
Cool test video though. Take care.
Very cool experiment! The results make sense to me. The training data they would be feeding based on human drivers would be that pedestrians are much more likely to be crossing the road at an intersection than not. If it's just on a straight road and the pedestrian stops, a human would do exactly what FSD did, just go around them (carefully of course) and keep going.
Try the hand signaling with the right hand - it will have a better profile. The left was in line with the body and bot which hides disguises the gesture. Alternatively, continue to use the left hand but cross from the other side of the road.
I agree. Also most of the gestures were done in the shadow of the trees. It is much more difficult to see any gestures without adequate lighting. She should be standing in the sunlight.
I suspect there isn't enough training on gestures to have significant impact on the NN's behavior. Most of the reaction appears to be whether the pedestrian is advancing or stationary.
I would imagine it's mainly context more than the hand gesture, for example if someone is stood beside a car and they stand still, obviously they're looking to get in that car, this is a common situation the car would have seen in training data. Good testing though!
Great video ,. Nice to know, but who knows if this will be added in. 👍
Good afternoon
Great experiment, thank you! It appeared the attempts to wave the car to stop had her hand right in line with her body, no way for the car to 'see' hand motions within a silhouette. Would be interesting to see if it would make a difference if the hand motions were outside the silhouette… (prolly not, but you never know)
Also noticed that the camera showed her while you were in reverse, but the fsd/drive screen (left side) did not recognize a person.
We had a school patrol in our left side yesterday (mid road), directing traffic and the car would have continued moving forward had I not intervened (her hand signals were for us to stop)
This is very interesting - A point that I'd like to make - If whatever version of s/w that you tested is not recognising pedestrian gestures then all this means is that this has not been a sufficiently significant proportion of its training data. Now that you have surfaced this issue and it turns out that this would be a useful thing for the car to understand as well as being a very impressive demo that puts Tesla clearly ahead , if I were Tesla I would add pedestrian gestures to my training data. I think this will be an interesting space to watch over the next few months.
It's not as simple as that, there are different ais for various things and each has to be trained as part of the whole. Each one has a specific role, it's not just one big AI. If you want it to understand hand signals it would need a new AI for that. Tesla has other more important things to work on like school zones which there is no control for yet.
Nice followup to previous speculation.
great video of the FSD taking into consideration objects in the road like pedestrians and noticing them moving, the system uses cameras, radar and LIDAR systems to detect objects.
the next test would be in a scenario where there is a parked car on the side of the road (either side) and a pedestrian was not visible to the car until the pedestrian walks out from behind the view. i am sure several situations like that have come up, that i am certain the driver had to intervene to prevent a pedestrian getting struck in that situation.
It was interesting though at the end, it was slow to react by to Stephanie taking a step backwards but as soon as she turned around to walk off it went. Suggests it takes the way the pedestrian is having into account
… Great, Great test, thank you! I wonder if the car can recognize hand gestures through the windshield of another car, at an intersection, where the other driver is gesturing to tell you to go ahead and proceed first to take the right of way.
how about from the driver of another car at an intersection or parking lot exit
Nice
Great tests and kudos to your wife for her willingness to participate (no way my wife wife would’ve done this :-))
Does it work while backing up? Just wondering because when in reverse i didnt see her on the gray screen camera. But in the real one i could. After 9:42. And what about motorcycles and bicycles?
Reading hand gestures is probably important for a full autonomy. You think tesla will implement this soon?
It still doesn't respond to emergency vehicle sirens, I'd think that's more important
@@DirtyTeslaIt responded to emergency vehicles correctly for me every time. I drive Uber, so I see a lot of different scenarios. It's not "hearing" them, but it sees them and pulls over. But we're also surrounded by other cars doing the same thing. I've not had one show up when I was the only car yet.
Ive seen it too!
I'm curious as to how the FSD will act with people who are asking for money at a traffic light. For example if the light turns green while someone is close to or walking around the car, will it move or stay? Also what about objects that randomly/suddenly come across the road. Can it tell the difference between an animal, person, or piece of trash blowing across the road and how will it react. Will it slow down, try to avoid, or completely stop?
Thee wave is not recognize the MOVEMENT OF THE LEGS is what the car tracks using gait analysis. They track the direction and speed of the legs and use the resultant velocity numbers of the person to project the possibility of path intersection.
Perhaps what would be most effective progress with FSD at this point in its development would be for the car to recognize a hand signal or hand held sign or oncoming emergency vehicle and simply alert you and turn the vehicle temporarily over to the driver’s control
Great video!! Thank you very much to you and your wife, and best regards from Spain 😁😁
BTW: it seems that European autorities will allow more FAD capabilities, beginning at the end of 2024, at least! 😉😉😉
Reminds me of a Star Trek:TNG episode where they had these machines called Exocomps. They did a test to see if they were self-aware, and they appeared to "fail" the test, proving they weren't self-aware. However, it turned out that the Exocomps realised that the test was fake so knew they weren't in danger. In the words of Mr Data, "I believe the Exocomps didn't fail the test - They saw right through it".
THe key to this which wasn't recognized is staying off the road, and gesturing. If bed comes on the street it destroys the gesturing concept. Gesture while off the road, or on the side of the road.
The hand sign to stop and the car not caring, will it not be issue if an officer is directing traffic?
Holding the phone could affect how the car reacts too. Someone distracted by being on the phone could result in different situations in the training data versus someone not holding a phone to their head. Some wireless earbuds would handle this, but either way it was a very good idea to have a phone call going during the experiment, for safety.
Traffic management outside the set stop signs and stop lights is pretty iffy. This is a pretty important factor to work on since hand gestures are used by first responders as well as construction/road workers.