Magnus shocked by world’s most advanced climbing AI
Vložit
- čas přidán 18. 02. 2024
- Please like and subscribe if you want more videos like this!
Project's website: belay.ai
Regine's Instagram: regine_storaa
Shot by Markus Skaane, edited by Sam Tuck.
Magnus Midtbø tries Australia's national comp boulders (crazy hard)
I made an AI that emulates Magnus as your coach. But something is not working... basically says "More chalk, less shirt" all the time.
seems to work perfectly fine then
Magnus deep down would really prefer to climb ass naked and covered in chalk
smart AI
When you're on the wall it should say "come on" every 5 seconds
@@MR-oy5yt good improvement! "Come on! Eaaaasy"
This guy should team up with Moonboard, with set routes and an option to film yourself on the route, it could end up being a personalised coach in each gym
Although this won't be setting or giving real advice on advanced climbs, it certainly could do very well on something static and predictable like a Moonboard, and with many hours of video to train the AI with. Other than that it will take years and tons of compute to even attempt to understand bespoke sets.
Ah that's a good idea. Exactly the type of application that this could work with. I wonder if you could even add pressure sensors to the holds to get any other useful data. Maybe, maybe not.
This is kinda crazy either way spray
@@Cyrribrae With how good the AI's ability to understand the visual data, combined with a basic understanding of gravity, angles, and folcrum, I'd say it could reasonably accurately predict how much pressure is being applied.
This second channel is quickly filling up with the kind of quality content I've been missing. Good job!
The dev missed a great opportunity to call it BelAI.
What about GlaDOS
@@El_Fling Probably copyrighted by the Valve Co.
beta sprAI
Yeah but it’s called BelayAI so it is similar ig 😭
@@neaituppi7306 Good point. GlaDnus with magnus voice
Hello, computer vision engineer here! Looks like this program combines an object detection model, pose detection model, and LLM together into a pseudo-coach. The model is likely trained in a fully supervised manner meaning the dev manually curated an annotated dataset of bounding boxes and poses from likely many CZcams videos. As a coaching tool, unfortunately this will be hard to extend. The application does not appear to provide any added value than superimposing two videos and the LLM providing advice does not know about physics, hold quality, or body types which a real coach would factor into their feedback. Best to rely on human coaches for now.
Well, but this tool can give you extremely accurate movement distributions for every limb. Things like that is something no human can give with such precision. I think a tool like this can be expanded in many ways to give great insights into climbs. For one the ability to compare numerous results and automatically compile the best techniques for each move, like creating a compilation from the best moves out of many runs. This would get you closer to the perfect technique for your body specificaly with every attempt. I think the body type is factored in since the length of every limb is visible to the programm. I also think physics can be applied into such a tool, since it has movement data over time. There's tons of physical information in video, the software can in theory identify how fast things move and how limbs bend depending on speed on impact. There's a lot to this and the developers don't have to stick to the current models they are using.
Hello, also CV engineer here, think there is a lot of value to this system not directly as a coach but as a tool to objectively observe climbing and compare different approaches. I often have it while climbing that a friend struggles with one problem that is easy for me or the other way around and having an objective observer to tell you that you have a higher success chance if you move your hip differently or you need to shift your weight differently at this spot on the boulder would be great to understand what makes the difference between a successful and a failed attempt.
How often is the only advice you can give while climbing to "just do it" because it looks exactly the same between climbers, this system can see the difference and give feedback.
I won't have a coach while bouldering as it is a casual hobby, but just setting up my phone next the the boulder and getting some feedback would be great from time to time.
The LLM part of the demonstration is the flashy bit, but in my opinion the least useful, as you said, the context of the actual boulder are missing.
Fair point. But that's the way to use AI in most cases for now. I work in as an IT Engineer in the healthcare world. We have a lot of ongoing AI projects when it comes to research. It's very hard to have an AI that can come up with conclusion from all factors. AI for us is mostly researched as a tool for doctors to use and ease workload. I think there will be a time when AI can fully replace humans in some cases, but i believe it's gonna be a while until then. I believe this tool on combination with a coach could provide great success.
I think it can be an interesting tool to study the kinematics of the movement and output some statistics, nothing groundbreaking however. Also, there's plenty of literature on human detection and skeleton tracking, one can just use a pre-trained model or other approaches not based on supervised learning. The LLM part can be useful to generate some considerations starting from the data, but nothing compared to a real coach, that, as you said, can consider a lot of other factors such as the dynamics of the movement. These are just speculations however since we do not know in details the functionalities of the software.
@@generic13372 Like he said tho, you can do that by filming the two climbs and superimposing the videos. You'd be able to see the difference on your own that way.
I can't wait for an AI to tell me how I can improve my sit-up technique when getting out of bed for the morning, so that I maximize the energy I have for the day.
Call her Sendie
I threw my back out on that technique once
people sweating every aspect of life instead of live it
Sit up is the wrong beta bro, just roll
i sleep on the floor so that laying down is a negative resistance and getting up is a full body weight @@MeskDaKrull
Too bad its missing data, but if it records many climbers on known route difficulties it could also potentionally set new routes and also predict the correct difficulty for a newly set route, you could also track progression in your climbing with this so measure how good you are becoming, also would be fun to compare performance, so if you have pro climbers then you can see how you stack up against pros and what you need to improve in your climbing. I think this guy is onto something, but there needs to be real app development and LOTS of data filmed from different gyms and routes etc for this to be interesting. Its very early stage, but the potential to build fun stuff ontop of it is endless, I really hope he has the drive or at least gets a team together that work on a climbing app with one or some of these features.
Agreed, I'd be really interested to see it set up in front of a moonboard or similar. Lots of movement options compared against known grades.
The website has career postings for multiple positions and a less than 2 min video that showcases many more functionalites compared to Magnus' video. Once the app is released it could possibly make grading a lot more streamlined with users flocking in and feeding data. Interesting how it'll play out.
@@hugoburgess3699 This would be perfect for the moonboard. It could get hundreds if not thousands of examples for each boulder
No, it can't set new routes because it is only dealing with the 2D physics of body position, not the angles of the wall; the size, shape and texture of the holds, or all of the other necessary factors for being able to set a route that matters.
This is a parlor trick good for generalized positional advice...IF it is trained on that specific set with lots of video of people doing it.
To have it set a new route that actually works it would have to understand the 3D shape of holds, wall, and body. Then it would have to understand friction factors of a particular hold, etc., etc. In other words, a long way to go without any data other than 2D body positions to work with.
@@FibberFinn Disagree completely. The only thing this AI will be good for is a cool app that helps beginners on known climbs, and generalized basic advice at best on climbs that aren't, so that leaves gym climbing out where sets are changed frequently.
Not saying there isn't a use-case for some app to use AI to help with basics, that will be pretty cool in itself, but don't expect an AI like this to be able to understand much more than that. Self-driving vehicles are only now just starting to get enough data to be useful and they have way less sensor input than the human body has and uses on a v7 climb, say.
The AI is only as good as the data it has to work with, and even though recent AI's can figure out basic physics on it's own, unprompted, there simply isn't any way of getting this to be more than generally accurate on a climb that doesn't have data of someone already climbing it.
@Magnus, call her Beta 😊
Bertha
GlaDOS
Betty
Beta sprAI works well too!
BaiTA
Magnus, as a photographer/videomaker myself, I really apreciate the quality of your videos (filming, editing, sound... ). It would be cool to see a behind the scenes here in this channel!
Exactly, would love to see it too! Videomaker here too🙋♂
Insane quality for being a second channel lol. Doing gods work for climbing fans Magnus+Team!
I'm liking the side channel more and more!
love the simple climbing vids. looking forward to this channel more than the 1st
Enjoyed the video a lot. Good to see someone trying the boulders with you too.
Enjoyed the video and also really liked Regine's laugh when things went wrong. Fun collab
i just got back from the climbing gym, so this was perfect timing to watch. great video and really interesting too
I think the concept of filming many attempts, gathering data from them and then using AI to analyze that data to point out the differences and similarities is cool and probably quite useful, but I'd be really careful when it comes to letting the AI draw conclusions from that and tell you what to do differently. It can sound really convincing and smart when it tells you to "move your right shoulder a smidge to the left" or whatever, like a coach would, but you have to remember that literally everything it "knows" is just the data you've given it through the video footage, and then the generative AI just fills in everything else based on probabilities.
It will tell you what you are doing wrong if you give it enough examples of moves done right and wrong... but as a climber you'll know better at that point anyways.
One key of the AI part is in fact pose analysis/computer Vision. I guess the rest is comparing movements of each limbs.
AI can analyze you as a climber, know your limits, strenghts and weakneses.
Taken that in consideration witht the fact that it will self-develop more and more by watching more climbing videos,
He can really know what would be the best possible move for you personally.
Now even gym trainers will loose their jobs 😅
just like with anything else, it is more of a guidance rather than an absolute "truth" that must be followed. At all times, exercise good judgment and common sense.
@@Smixi35 Which should tell you why this isn't very useful for anything but Moonboard or beginner climbing. AI's are only as good as the data fed to it and this happens to be through 2D video right now which doesn't even account for the fingers, or friction, or a host of other necessary factors to be useful on anything bespoke and not set permanently, and trained watching climbers on that specific climb.
The programmer did not do all the AI himself, a significant part of is from openpose, hence involving a team working of years.
I feel like that’s sorta just semantics. It’s like saying someone who made graphic art in Blender didn’t do it themselves because there’s a whole team who worked for years on Blender.
@@jamessmith4172 The right analogy would to take a finished blender picture and to add a few filters and claim "I made this picture". All the training and architecture is finished in open pose.
Your gym looks ideal to me, Magnus. Huge features with dynamic, gymnastic, and technical climbing without a ton of small crimps that make fingers feel like they're going to explode. Sometimes, it seems like setters only increase difficulty by decreasing size of the holds. Hope your gym's setting style becomes more popular! Also, cool use of narrow AI.
Do more hard climbing videos. I love this and honestly learn so much. Keep these super awesome climbing focused videos coming. Love your channel. You are the reason I started climbing.
Do more of this! So enjoyable
Awesome! As a person that is interested in motor learning researsch, I can say that the "angle and body position" data is not really valuable information to progress despite being interesting.
On the other hand, the conclusions/summary that the AI takes can be really really useful if appropriate language is used! (not focusing too much on body parts but rather on movement's result)
love this 😎
Feels like verbal explanations are hardly ever useful in climbing regardless of language. It works way better when someone demonstrates it.
@@777Mikos This might be true for the basic beta of a route/boulder (hand/feet placement, sequence) but once you get to more technical stuff, demonstration tends to be useless because it's about adjustment or body tension you just can't "see"
For example when technical moves require to shift your weight at precise locations/moments or when you need to engage your ankles to create body tension, verbal explanations is key
However I dont see how this can transfer to AI because I'm not sure it can infer this kind of technical advices from visual data only
Magnus has always come off as so wholesome and genuine. I can sit back and watch these videos and feel like im sharing this time with a good friend.
Sick video dude, would love to see a some outdoor climbing vids on this channel! A France trip where you try Rainbow Rocket would be cool, proper Magnus style boulder
It’s a cool piece of software, but calling it “AI” is just laughable; it is a data analysis tool that pipes its results into an LLM (sounds like ChatGPT) and feeds that into a TTS program.
The bare minimum for something like this to be an “AI” would be having it predict the route or work off a single climb attempt.
Exactly, it is getting really tiresome how people are miscategorizing everything on a computer as AI nowaways...
It seems more like a trackman (in golf) than anything
Does computer vision pose estimation not count as AI anymore? Or text-to-speech?
Its literally just a frontend for pose estimation.
I had the privilege of not only visiting Norway last year, but also this gym. There are multiple floors you can Boulder on and a vast amount of top rope and lead climbs. The cafe had beautiful desserts and all the staff were super friendly. A wonderful gym that I know the community loves to have as a beacon for all climbers.
You just stated what was in the tour video...
@@Dr.Spatula Well maybe because I went to the exact place being toured… 🤣 I don’t need to lie in comments for clout homie.
12:54 -- What a mind-bender of perspective. Looked at first like there's a hole in the wall.
i've had a similar idea but i am not a climber so i never really pursued it. cool to see someone is!
my idea was to use 3d scans of the wall and a 3d actor to attempt moves and calculate the forces someone might endure to complete a move so root setters could have more insights into exactly what type of climber will struggle where
when magnus called the project an app i cried a little for the dev.
This could be big for speedclimbing I imagine!
That, and as a beginner climbing app advice is about all it's going to be able to do since the holds and setting is predictable.
It was mabye built/based on speedclimbing fotage, because its the same route and very easily comparable with other people
There should be an excited Adam Ondra voice option as well!
Congratulations on such an amazing video, can't wait to see your channel grow even bigger.
Having her there makes Magnus look even more superhuman. She's SO talented, but still can't send it like Magnus. Just puts into perspective how useless we'd all be in those same situations lmao
I wanted to see more attempts from the lady
I love Regine's energy. Would love to see more videos with her.
she wants himmmm 15:37
a bit of a waste , putting her on strenght things she does not have. didnt get to start or finish anything . her gigles started to anoy me.. i actualy wonder why she was in the video .. she must be a girlfriend of some guy who was there..
For someone whose in the national team I was expecting more of her..
Wished you would have shown more of Regine's attempts
I can't wait for AI route setting, that is going to be some sick stuff!
keep it up with these videos!!
Really impressed by people who can fall on their backs from > 2 m and get up like nothing happened. Regine must have great neck muscles!
Well they aren't landing on concrete
Still, I've fallen from that height onto a similar mat onto my back, and it was scary. I thought I'd get whiplash but somehow I was fine.
i think this might be almost more useful for other sports, as a huge part of climbing is in the fingers and hands while the ai seemed to focus much more on arm body and leg movements.
So thinking about olympic sports this might be incredible for certain disciplines
But every body is different in architecture (proportions, weight distribution, strengts and weaknesses) . You can't just compare limb positions and joint angles directly 💟🌌☮️
Indeed. It's a shame this wasn't addressed, because unless proportions and relative strengths are taken into account the numerical comparisons are quite limited in usefulness. That said, it's early days for people applying pose detection to climbing, and the relative numbers for one person over multiple attempts can still be valuable as a starting point (e.g. "this attempt your heel was lower and your left hand was higher on the piece, which worked better" kind of thing).
Over time this will hopefully allow scanning in physical geometries (or at least providing a set of limb lengths), and ideally will eventually make use of stereo / lidar-augmented footage to treat things as properly 3D (including the wall and hold geometries and poses), at which point it can get much more powerful, and provide more meaningful individual and comparative insights.
Forgive my earlier dumb comment. I deleted it, honestly didn’t think you would notice, and I’m touched that you responded. The second channel is great so far and I, among many others I’m sure, have been enjoying all of it so far. Please keep uploading (I didn’t actually sell my kidney btw)
Loving the 'side' channel!
Reminds me of a program we use in weightlifting where I train... just WAY more complex and advanced. Truly fascinating.
I think it would be super cool to see it as a analytic tool for championship
It would be interesting to see the ai applied to setting routes as well. Like giving to the ai the holds, set the grade and style like overhang or slab and the tool will return the route… I guess even easier for the moonboard and alike
Great idea to use a language model to make the results easily accessible (the speech synthesis is the cherry on the cake)
Great video, but too short. I miss old Magnus, when an hour of entertainment was not uncommon. Don’t listen to those marketing fools, Magnus. Follow your heart, make longer videos. Peace ✌️
More Magnus channel is so chill that Magnus doesn’t even iron his t shirts 🤣
You iron your t shirts??? 😂
who does???
When lots of different people do speed climbing with this software, the data is going to be VERY interesting to see
As cool as the ai stuff was honestly would love to see the 2 of you just having a session trying hard without the distraction
we absolutely need more videos like this, bouldering, maybe with some additions to them, but just your climbing sessions help to progress on my level by spectating)
I love your new channel
Just look at Magnus' face and neck color from 9:00 to 9:45. It does the whole pale -> furious red -> back to normal gradient. Really interesting.
have you considered integrating lidar data so that you can scan outdoor climbing environments as well. the algorithm would have better access to the three dimensional space that climbers are working with, and really taking it to the next level, understanding the physics the climbers are overcoming and integrating to send their climbs :3
Imagine having all magnus several hours training session filmed for data set. Would be insane to have it analyze all of it :)
At the end it would have been interesting to have your thoughts about the last input of the AI and also some final thoughts about the whole system.
Would be cool to pair this with something like Kilter Board with repeatable boulders more people can access. Could even do a “climb like Magnus” type marketing.
Yeah it would work so much better with known repeatable problems to start filling in datasets.
Gotta be careful with the difference in climber sizes. Trying to climb like Magnus could be a good idea if you have a similar height and wingspan (granted, that's a lot of people), but even then, you won't have his strength. Maybe you need to do a figure 4 in a move where Magnus would "simply" do a one-arm pull-up.
I guess it would be interesting to find a strong climber who matches your own size and style.
@@andeolevain that's a good point. I'm just starting out, and I'm an older guy (46), so if I even tried to do something like him, I'd probably rip a tendon right off the bone. This video was interesting though because my career has been boring indoors programming and high performance compute cluster architecture. So blending in computing analytics with climbing is super interesting!
Especially so if we can learn more about safety and risk management.
A lot of effort went into the side channel. It will pay off 💪🏻
This was fascinating, a really interesting use of AI. The potential efficiency gains alone would be valuable for competition climbers. That voice would annoy the hell out of me though, I think it should be John Cleese!
The smart guy's(didn't catch his name) T-shirt is sick!!!
if i saw correctly you are using the drago lv in this video and often you use the regular drago. what makes you choose one over the other? @magus
Magnus, you should talk to the company that tested peoples deadlifts last year at the Shaw Classic strongman competition. They had me do a relatively light deadlift on camera and predicted my max, the hardest point in the lift, etc.
Would have liked to see climbing her more. Like a 50/50 split would have been nice - still a fun video though
He should/could release the AI as an app.
This will really shine on system boards
This is very exciting to see in action! As a data scientist and an avid climber, this is a great example of how AI can help! Will definitely check out the website to learn more on how it works etc. Can see future use cases to help train people, almost like a pocket coach! Which would be really cool! Nice work
Do you think that the given example of instruction from the AI in the video would help you on your problems?
Possibly. I think it would all come down to how it’s trained. Is it solely looking at my performance on one boulder, is it trained from other attempts on the same boulder from other climbers or does the algorithm find similar climbs and give tips based off those moves? I would certainly love to try it and see how it works in real time!
This is so impressive! Exactly the type of implementation of AI that I think is beneficial for society in general.
I've been wanting to do something like this for a long time, the only thing is convincing a gym to invest in and buy cameras for speculative tech. Best of luck to the dev
her slab climbing was crazy!
incredible
Interesting AI project. Cool video, too!
Another great day!!! Love your videos Magnus. Keep up the good work 😊.
The AI can't help Magnus that much when he simply flashes the boulder 😂
More Magnus is now by far the best climbing channel on YT.
The dark side of the moon shirt ❤❤❤
Cool stuff!
probably very helpful for speed climbing
It was good with Regine there too. Maybe some videos of climbing with her and your sister, for the different styles and the interactions?
this paired with the moonboard app would be so good!
this guy could apply this to all sports. This is amazing honestly what a great tool
Loving the climbing content but could the editor put the boulder grades on the screen please? Like Magnus used to. I like to know what I'm looking at but I'm even worse at 'guess the grade' than Branden
Never mind. I rewatched it & noticed the gym has changed to colour grades. What a shame
this is awesome, good climbers can help feed the AI with inputs and helping beginners that way all over the world ! great use of AI
I want this :D - with Magnus-Coach!
I'm writing something similar for bowling, this is very well done.
Can't tell you how thankful I am for the climbing content, Magnus ❤ feels like the old days that I'm super nostalgic for 😂
From the last gen that began in etriers and goldline hemp ropes? Gotta admit?
a little scary with the whole AI embrace
at 12:55 there's a weird sort of optical illusion. When I first saw it I thought the camera was panning to the left making the left side of the wall come closer which made the undercling look inverted and my brain felt like it broke. It took me like five replays just to figure out what was happening.
Thank you for the new channel! Your climbing related videos are 🤌 excellent'e.
Magnus: I took an arrow to the knee.
Very cool video! :)
This is sick.... This would be super handy Asa newcomer to climbing.
Cute project!
One of the instances I aprove (not that anyone asked my opinion) and can see the benefits of AI
All praise Basilisk!
Honest question, does it make sense to talk about angles when comparing climbers with different heights? Aren't angles changing based on length of arms and legs?
Time to start building a dataset!
Interesting tech. Like most current AI, the optimal use case still seems to be quick analysis of large amounts of data that can produce findings that human experts can then use to draw conclusions and develop strategies from.
Not sure I saw a beta calculator demonstrated here and certainly not as good as the humans, but that doesn't mean it's not possible and certainly doesn't mean it has no uses.
One thing I'm curious about is how good the input setup has to be to collect data. Do you need special IR trackers or can you get by with normal phone camera in mediocre lighting? That would change the potential use cases a lot. Ie: is this a tool that every beginner can use to create a personal climbing regimen on a moon board? Or is it one reserved for elite athletes that can justify the expense of understanding their tendency to lift their elbow too high?
I'd watch all 3 hours
I thought it was hysterical when the Ai told Magnus to slow his role.
I would think that given each climber has a unique center of gravity, length of limbs, weight etc. that it would seem possibly inaccurate for the AI to give blanket advice.
It seems like since the body dynamics of every climber is unique then the style in which they climb would likely be unique.
Especially when you consider differences in flexibility and potential old injuries that make some movements less effective.
12:25 that is an Ondra move. I have seen him do it in some of your videos.