Trackmania AI Learns To Drift and Beat Pros ? | Hockolicious
Vložit
- čas přidán 21. 07. 2024
- We trained an AI to play Trackmania on Hockolicious, one of the game's most prestigious tracks. The AI set a strong time, beating the 2012 World Record previously set by CarlJr.
Link to map: tmnf.exchange/trackshow/414041
Link to replay: drive.google.com/file/d/1b0p2...
Game: Trackmania Nations Forever
Project Github: github.com/Linesight-RL/lines...
This project would not have been possible without the awesome TMInterface tool, provided by @Donadigo.
00:00 Intro
03:00 Learning to finish the map
05:06 Gold Medal
05:48 Author Medal
06:48 Best Run
07:50 Comparison to human runs
#AI #reinforcementlearning #trackmania #worldrecord - Věda a technologie
WTF??? an AI drifts cleanly on hockolicious!??!??!
The most frightening thing about AI is that it's absolutely terrible in everything until it's not, and when it's not, it absolutely trashes every record ever. It's everything or nothing.
It's not really frightening when you consider that it has no chance on levels which it didn't see before.
@@bbbndddl until it does. And when it does, it'll probably beat every human driver ever no matter the track
Does it ? Have no chance ?
What would be a good benchmark for tracks it has never seen before ?
Let's say it is able to finish 90 percent of maps that it has never seen, is that any good ? Does it need to do bronze, silver, gold or author medal ?
Curious to have your expectations, you know, for a future video 😁
@linesight-rl what I meant was that it doesn't have much chance at performing as well as top players without any further training. This map is pretty uneventful. I'd say that AI having a top 100 time with a few hours of training would prove that Trackmania can be solved in the near future. Right now, I think it's highly overfitted to this particular track, and that's why it's so good.
Please try a trial map, even if only a small one. I would love to see what the AI can achieve on tricks that take hours for humans to perform. Maybe oachkatzlschwoaf on ice? Or only a segment?
it'll take months but i kinda wanna see that lmao, or some easy kacky xdd
I second this for sure, the reward function would be kinda hard to make and would likely prevent shortcuts and such but it would be extremely cool to see regardless.
this
problem with kacky is that its black and wight often, you either get it or you dont, so setting good rewards for getting close is SUPER hard, if they could do that i'd be super impressed.@@thomaslomba5181
this is a great idea, though an rpg seems easier for it.
I think the next step should be meta-learning, so that agent can start learning on the new map without weights resetting. RL^2 is the simplest meta-rl algorithm, give it a try!
Absolutely this. It might be really tough to integrate because it seems like a lot of the reward system that you use is based on learning the track, but if you can implement some form of meta learning it will cut down on your work per map by half at least.
This would then free up more computing power to attempt higher frame rates while keeping the same amount of time to get results.
Great work!
Yes maybe I didn't understand, if they tried to make the AI trained on the previous map play this map it wouldn't work ? Would the AI just try to follow the path of the previous map ? I'm confused
@@gregzerThe results from the inputs the AI is getting are tailored more specifically to this track. If you change the track, the AI would probably do "ok-ish" depending on what type of learning they used here, but it definitely wouldn't be optimal at all like we see in this video.
Think of it loosely like learning to comprehend English, and then being told to read and comprehend Spanish. You've trained for so long to read English and comprehend that, but now you have a different problem you're trying to solve with (mostly) all the same types of inputs. Sure, you can "read" the Spanish, but you won't really know what to do with it to get any value out of it.
It's not exactly the same type of thing, but the general idea is there.
@@gregzer The AI doesn't learn by doing random inputs until it finds ones that work. It instead looks a an image of the screen and tries to predict what it needs to do next, the longer it does this for, the better it gets at predicting
@@jjjoker5766 Sure but with a reward function running on 7 second, it's likely that the AI's strategy toward the end of the training is to find the closest picture from current one on the better previous runs and deviate very slightly on the 140 next inputs (20 fps x 7 second).
Which means that if you give it a new map to learn, it will get confused cause it will try to replicate parts of the 1st map it got trained on, and the more maps you try to add the more confusing it will be, which means it will perform worse on the maps where it previously got close to WR while never reaching such times on the new maps.
This is where meta-learning can provide good performance improvement, the IA would be less specialized and a bit more flexible, while (with a fair bit of luck) saving some data so they could improve fps or picture quality.
Well failing 53.xx on past streams I did not expect the AI to do so well to literally equal my time! Such a smart way of explaining and running this experiment too, mindblowing work!!!
Man you have to find a way to get yourself in the spotlight because this is amazing content and you deserve more than a couple 100 subs
Just send the video to wirtual on stream, i dont use twitch so im not gonna do it
Let time do its job, like the old days, it's good content after all...
This is so cool! I always though ai driving in tm would never be this optimized. Well done! 🏁
Thats awsome! Well done. But my bet for the best further improvment is not the framerate, but the resolution. With higher resolution the AI should be able to take the corners more tightly
If performance is an issue with higher resolution, it would maby be an option to ignore parts of the frame? Like ignoring the top of the screen and what is located left and right of the car, to not bother about unnesesary pixels.
@@nanoder7te maybe even the shape of an eye is enough data
Higher resolution is already gained in a way by higher frame rate. In the same way you can see through very small vertical slots by walking past them, the AI can use pixel data from one frame to the next to determine very precisely where the walls are and drive close to them. Increasing resolution would be more costly as far as performance and likely would show less results than framerate increases.
frame rate is also very important, in the video the AI can only use a combination of 20 different key setups per second. With higher framerates, this number will increase so there is a lot more space for improvement
@@vitorrodriguez4278 makes me think of riolu... that's basically what a higher framerate would mean for the AI.
I love how your videos related to AI with trackmania is one of the main reason why I'm passionnate about AI now that I know how silly I could be if I could hold this much power into making my own little cars go vroom. Thank you for this video, I loved it !
This is incredible to watch. I also love how you intuitively explained how the learning system works and all, that's very cool
Thank you very much!
It's brilliant! Well done, you're super good ! Good luck for the rest
That’s genuinely so amazingly cool that we can watch a computer program do things like this, and also things like TASes.
This is incredible, i love this. I am studying AI in uni and i hope, with more experience, to be able to make fun projects like this soon. I can't wait
I would love to see a more technical video about this AI, like what/how you got game information, which infomation you have used to feed the ai, and drawing GUI into the game window, awesome video btw!!!!
Very interesting to see the improvements. Well done and very well explained. Congratulations on this achievement!
Never expected AI to get this far this fast. Awesome job! Makes me wonder what is to come. Loved your explanation too
The caveat is that it's making its improvements over people via being able to perfectly speedslide in a way a human physically can't. It's fundamentals are a lot weaker.
Not only a super impressive time by the AI, but also a sick video
Absolutely phenomenal ! I will eat my hat if an AI ever finishes Deep Fear respawnless (or at least a couple checkpoints) with that same learning method, prove me wrong !
Noted
Noted
Note to self: return to this comment section with a tasty hat once AI finishes "Deep Fear" respawn-less.
Should be easy for an AI if he can watch the records 🤣
AI wtf man just copy the wr..
This is insane. Amazing work! I'm interested to see it learn to do other tricks on its own like wall bangs, air brakes, speed drifts, ramm hit, etc, or maybe even start finding cuts that no one sees.
I'm impressed how well you got it working with visual input! It's interesting to see how people either overestimate or underestimate how well AI will perform. A balanced take is quite rare. As someone who's trained racing AI before, I'd love to see this applied to other prestigious tracks :D.
I expect it would train a lot faster if you give it data like speed, the dot product of velocity with the car's forward vector, and one with the sideways vector. The latter really helps it learn to control the drifts in my experience.
Thank you for your kind comment.
I'm curious which kind of racing game you tried to apply RL to?
As for the additional inputs you're suggesting, I did not take much time to explain this in the video but we are using this type of inputs. They are necessary along with the image to judge the car's dynamics, orientation, etc...
Very nice video 👏👏
Your next challenge is to create a ‘general intelligent’ driver that could finish an arbitrary map. Maybe by changing the map every few training steps the model could build more general understanding of trackmania driving.
this channel will explode if you keep doing ai racing games
It is important to note that this model wasn't trained to play Trackmania, but was trained to play this one map. It has most likely just remembered what each turn looks like and the optimal way to take it, rather than figuring out how to take each turn as it sees them.
A good way to train a neural network for this sort of role is to give it lots of pro and intermediate level replays of different maps and have it learn to predict the player's inputs. Then after a while of training for this, you take this model and make it do its own runs on lots of different maps and reward it based on speed for further training beyond what humans have achieved.
This is how AlphaStar and AlphaGo were trained. Alphastar is a Starcraft II AI that is the best in the world and AlphaGo is a Go (a board game more complicated than chess in strategy) AI that is also the best in the world. They were both made by DeepMind, one of the biggest AI research companies in the world, owned by Google.
One last interesting bit of information, it was DeepMind who made AlphaFold, the AI that solved the protein folding problem and will likely win a Nobel Prize.
Wow! this is incredible!❤❤❤
I would love to see you teach the AI glitches, such as bugslides, or nosebugs. Maybe it could even learn to noseboost!
this is amazing! i'm relieved to be part of the 23 who outclassed the AI x)
This is a solid success. Well done mate!
very impressive indeed. I love Hockolicious and every insight is welcome
this is so cool, definite new sub here :O
Kindheitstrauma would be incredible to see how the AI handles extremely high speeds, wall rides, and loops.
Additionally a kacky map would be absolutely insane, but honestly, from a layman's pov, Idk if it'd even be possible 😅
Anything is possible, you would just need to define the reward function differently to take into account the unique approach kacky requires.
I love that one AI training's most primordial arenas is its use in driving. Automating travel is, ironically, one of those things that splits people down the middle in terms of modern issues. Some people see driving as something that is dangerous to automate because of potential injury or fatality, others would rather sleep on their hour-long commute to work if they're far from home.
Then, on the opposite axis of the grid, you have Trackmania players; automating the same exact thing, but for COMPLETELY different reasons. Trackmania having AI drive its tracks could be useful for any number of reasons: it could eke out milliseconds and show players tricks or strategies they may not see, they can be used to make more definitive gold/silver/bronze times with a more reasonable baseline of difficulty by tuning its abilities based on what medals you want it to achieve.
I don't know where I'm going with this, I just found it funny that even in the virtual world, some people still have arguments against having AI drive cars despite demonstrably less stakes at play.
This was a fascinating video and gave me a lot of food for thought. Thank you for all your hard work.
Amazing work 👏 🙌
I love how it went from „Struggles to get over a bump“ to „Hard for a veteran player to achieve“ real quick
really impressive work! it is often i come across ai playing video games and then being disappointed because the performance wasn’t great. This is a different case. Absolutely amazing job at getting the ai to work so well. I think it might lie in the rewards system being so well made(i am not an expert so please tell me if there’s other reasons for the ai’s performance) Props to all the people who worked on this, amazing job.
thats amazing work i cant believe it achieved those times. next up map suggestions ofc kacky
Incredible Work! How good does this agent perform on other maps? Did it learn to drive good in trackmania overall or did it overfit on Hockolicious? How important was the graphic input compared to the engine input?
God I love ml so much. Fantastic video. Curious on the optimization of the relationship between framerate and resolution. Read some other comments noting that an increase in framerate would essentially be an increase in both, although I would love to see a simple experiment using a simple drift track with framerate and resolution control values for testing. Can’t believe tmnf community is still going so strong after 2 decades, makes me so nostalgic
Wow, very impressive progress!
This is really impressive. Great work !
I think you should experiment with a depth map input layer. Maybe by pluging in the DirectX's API. It might help the AI's better perceive it's position on the racing line and act accordingly.
Can you train an AI on the random map challenge? That way you'd get a more general AI. It might not perform as well on individual tracks, but overall it might do pretty good.
agree on that, or just take way more maps. Else you end up overfitting super hard
the wirtual ai
I don't think that's how it works, the AI learns using a trial and error method, meaning the AI can only learn a map at a time. If you put it into a different map, it will be just as bad as it was in the start, and will have to learn everything again.
At least that's what I understood from the video
@@vitorrodriguez3799 It could work. It would probably require an extra layer and more neurons per layer, so it would take longer to train. Also there's a chance that it won't learn the maps well enough because of the complexity, but you don't know until you try.
@@vitorrodriguez3799because it's trained on a single map. But if it's made to be trained on random maps, it will learn probably slower but it will eventually be good on all sort of map (as long as it doesn't requires tricks the AI doesn't know)
Well done Linesight, 400 subs overnight!
Really good video !!
Incredible, it really is the perfect run, every drift endes exactly where it should, and I think the car accelerater a bit faster when “sliding forward”
This is great. I'd love to see it on a fullspeed map :)
6:01 I like this diagram. It shows the meaning of training. Not only virtually, but also reality.
This is incredibly impressive!
I would LOVE to see this AI become quick enough at learning new maps to compete in Cup of the Day. See if it can get division 1.
this is super impressive!
I would love to see a write up with more detail about the training. I know it wouldn't make a good CZcams video but I'd be absolutely fascinated. Obviously the next step is generalization. You could use Cup of the Day Maps as a potential training pool, though perhaps it would be best to ask the mappers.
This is very super impressive. Since you asked whats next: no idea how feasibly this is but would be cool to get gold medal on every track in the campaign/career in TMNF, driving on different surface types with the same model might be interesting
So, the objective would be to have one neural network that is trained on all maps at once, and able to reach gold medal on all these maps?
It's definitely something we can try, I'm curious to see how it'll go.
@@linesight-rl man you gotta have a held-out dataset to show real performance
Wow congratulations!
Keep up the good work!
it would be nice to make an ai that is made to be good at generalization, eg, capable of doing new maps it hasn't yet seen fairly well/learn new maps quickly as it starts with good baseline "instincts", rather than it just potentially overfitting. But i understand that that would be much harder to do so its totally fair that it is the way it is. Incredibly impressive work! keep it up.
Very cool, would like to see your improvements to the AI
this is amazing well done
how explorative did you make the AI? if you have more compute time on your hands it would be interesting to see how the AI would do with higher exploration (and FPS, as you mentioned). loved the video+presentation btw :)
We are doomed. That's legit scary af and also impressive af. Damnnnn. WTF??? an AI drifts cleanly on hockolicious!??!??!.
Good Job :)
Further improvements could be using a pretraind vision transformer (e.g. fastvit) or cnn (e.g. mobileone) from the pytorch-image-models
git and maybe (pre) train the model with rl from observation (since you can extract a lot of wr-runs and their inputs from the exchange)
this is so crazy, ggs
As a guess, I think it could improve the result further if the ai can also steer values between 0 and full, or increase the inputs/time it can have
This is incredible. I hope to see it beat all human players soon!
I think it would be awesome to see it try to learn a track containing every different surface!
If we're going in this direction, the AI has also improved its time :o)
I think our last run on map5 went down to 2:02:08, but we didn't update the video.
I have to wonder if a "two part" AI might be possible: A general "how to drive in Trackmania," portion and a "what do I need for this track?" portion. Training might involve cycling randomly through a couple dozen tracks, with the track specific portion being swapped in and out.
This is so cool !
great vid and yeah 20 fps is definitely holding it back alot id imagine just a higher fps would allow you to beat the current world record
Finding a simple cut would be an amazing AI achievement, because it has to divert from usual driving behavior using some complex information.
a few things that could improve the ai:
- increasing fps like you said in the video (i think 30 fps is good)
- let it have analog steering input (if that’s too complicated limit it to 0, 20, 40, 60, 80, 100% values)
- maybe give color screenshots?
amazing video!
nice editing
I think this is great, somewhere between a TAS and a human run. Feels like it could be used another benchmark.
It basically is a TAS because he's performing on the same data that's being used for training. There's no generalization
Super cool project!
I have a question tough:
Wouldn‘t the current speed in x, y and z be a useful input for the network to know? Especially for turning into corners I would assume that the static scene information is not enough. After all, when playing ourselves we get this information alongside the scene.
I guess apart from feeding it as input, you could use a recurrent network that maybe is able to infer the speed from subsequent screenshots. I feel like using a recurrent network makes intuitive sense anyway.
I would also guess that adding this input information would greatly help with generalizability. I worry that the model strongly overfitted to achieve this result. But to be fair, there‘s also a high degree of overfitting being done when we humans practice these runs. 😅 I‘d still be very curious how this model does on a different track.
I wonder if it would become more capable if you rewarded it for its position throughout the run, instead of its own estimation of the final time (if I understand how it works anyway).
Very interesting project, this deserves way more views
THE tastyy ?!!!!! Your time on this map is really impressive, I hope my computer will beat it some day because I personnally struggle to just get the gold medal!
I would love to see ice next
I'd be really interested to see how much of the learning transfers from map to map. What if you took your flat trained model and used it on Hocko? Would it converge faster?
Also, what is the "additional information" from the game engine going into the AI?
AI coming for our trackmania streamer jobs!!!
What I'd like to see (or try) is to have a selection of a N (e.g. 20) different, popular maps, and run learning in parallel and in batches, with random sequence of maps for different learning periods, and then test those against different maps and see if they can e.g. beat human record on 1st try :D
Great vid, doing everything i can to make youtube help you :)
This really makes me want to attempt this with melee, much harder though since thats a player vs player game
Hey @Linesight Don't know if this is possible, but what about editing the camera so it only sees the most useful parts of the track (wide angle in from front tyres onwards.
Also, I expect higher FPS to be less effective than higher resolution. I might even say 15FPS on twice the resolution could make the AI way more precise.
Have you tried to change the camera settings? With cam 1 it sees more what lies ahead, but with cam 2 or 3 it can possibly resolve corners and curves better and thus cutting it more tightly
You might find an improvement in time if you add a reward for least amount of time per checkpoint or overall time. I suspect that your max distance in 7 seconds is making the algorithm optimize for a slightly less than ideal line as the shortest route to the finish is likely to be the fastest. So if the current AI would be given an additional target of quickest time that weighs more than the distance traveled it might find more optimizations.
I think that more FPS is going to help the AI mainly in busy sections where there is a lot of changes on the screen (repeated highspeed curves and so on) but I doubt it is going to do much in the way of increasing the overall time.
One more reward that you might want to think of even without the time based reward is a reward for less direction changes, this should result in a smoother ride and less direction changes which in turn will result in a higher speed as cars are always fastest in a straight line and will slow down when changing direction.
Besides that I'd say keep up the great work I love seeing how quickly the AI progresses from knowing nothing to reaching world record pace in just 70 hours is pretty impressive.
Very impressive.
At first I was sceptical of this approach, I thought having the AI deciding what to do next in each moment would make it gittery but it was very smooth in the end.
I'm thinking tho, as humans we obviously don't decide what to do next for every frame, perhaps the AI would be more smooth out of the box if it had to commit to turns (like left 70% or full right and break) and then have a separate agent that considers the risks and if the risks breach a certain threshold, the AI aborts its turn and does something new.
Another idea: Ik this is way easier said than done but would it be possible to instead of button presses, you give it certain actions, like turn left, turn right, bugslide, and so on, but obviously way more options, all the actions could be induvidually trained beforehand with the system you're using now using many different maps so that it's a pro at all the separate actions.
Time to train some ZRT Trackmania Cup maps
They way you train AI reminds me a lot of my elementary days
I would love to see if your AI could match the winner of the midori map.
Especially if it can learn to use the unique turning needed on it to get that extra speed.
Can you PLEASE upload a tutorial video, maybe even on a different tutorial channel? Your voice is extremely understandable and your explanations are amazing and make sense. Instead of watching 10 hours of people repeating themselves over and over again I would like to actually learn. Pleaseeeeee consider. thank u
can you use assisted learning like this? for example, directly adding the wr's inputs so it starts from there instead?
Some great work! Your code for a CZcams video is cleaner than code submitted with conference papers… Interesting you’re using IQN 👀. What do you do for the reward shaping?
I expect this video to do great, but just as a tip from one CZcamsr to another, it would probably even do better with a different thumbnail.
😍 Love your videos, I've been subscribed for some time to your channel on my personal account !
Thanks for the compliments on the public code, our latest (private) version is both better and cleaner.
I thought I had done a good job for the thumbnail, can I get in touch with you to discuss what could be better ?
I'll use the email in the "about" section of your channel.
Happy to hear 😊
And sorry, I didn’t mean to be too blunt about the thumbnail comment 😅 it’s a decent thumbnail, I just think the video is top tier, and if you can get enough people to click on the video I’m guessing it will blow up
And yeah that email works 👍
Nice. What are the inputs? Just time or a distance array to walls infront of it? Any sensory inputs?
Man i cant even beat the AT wtf us this AI !
Really nice job this is amazing.
i wonder if you decrease the penalty for hitting the wall in increase the reward for going further ( along the track) if it would start to abuse certain mechanics
Just wow!
Lol beat Wirtual. SUBBED❤
Please explain the reward function in more detail, like how does it measure track distance, dit you pre set a track path?
i have a small question, what youre working on now? are you training the ai on a map or smt? i wanted to know, because i really like how the ai are getting soo good to the point that they can already do simple maps in trackmania, i would love to see more complicated maps tho, anyway, gl on future projects
Thanks for checking up on us :)
We're still hard at work on this AI, we have enough results to make the next video.
It's a lot of work, but I think you'll like the next video. Just need to be a bit more patient :)
Very impressive. How did you choose the hyperparmeters for your network?
Intuition, trial, error.
Well done!
Now see if it can get AT on Campaign 2023 map 21. 🤣
Stunt maps would be fun
i wonder if you could filter out some blocks to give them as an input. after all a map is nothing more than a stream of blocks with which the player(car) is interacting. so if you could just pass that stream to the model it would help with generalizing the model no?
quite curious to see what it could do in that case.
Putting the AI on an RPG track would be really interesting. Seeing it try to find the right track, and when it finds it balancing on beams or driving on blue walls would be interesting. Some sort of easy Mini RPG would be interesting to see.
I've seen many people suggest maps that are not available on TMNF... Do you have a specific map in mind? Let's start with an easy one if we're already changing map style though :)
@@linesight-rl I don’t really have a map ready, but it should be pretty easy to build. A couple blue walls that you’re supposed to drive on, maybe downhill ones to add a bit of challenge and then those double beams that you can barely drive on interspersed with checkpoints on platforms should be enough challenge. Maybe like the end of oach, so that it’s supposed to go fullspeed and jump into the finish.
How long until it can take on a 5+ min RPG map? Or maybe a sub 2-min miniRPG map first?
Or more complex, a map that has multiple paths with similar distance, or one that is straight up symmetrical. I wonder how it'd deal with something like that, but that's probably just adding too much confusion to the AI for now.