Francois Chollet recommends this method to solve ARC-AGI
Vložit
- čas přidán 10. 06. 2024
- ARC Prize is a $1,000,000+ public competition to beat and open source a solution to the ARC-AGI benchmark.
Hosted by Mike Knoop (Co-founder, Zapier) and François Chollet (Creator of ARC-AGI, Keras).
--
Website: arcprize.org/
Twitter/X: / arcprize
Newsletter: Signup @ arcprize.org/
Discord: / discord
Try your first ARC-AGI tasks: arcprize.org/play
chollet has been my favorite voice on AI for 5+ years and I don't see that changing!
Thanks Chivo!
So he is saying that LLMs guiding hyperparameter space will get us to AGI? Seems a little simplistic to me ... and there are better ways to search that space.
Love this
The question is if someone manages to get high score in ARC style problems will they really share the code for such little amount of money rather than raising money from venture capital farms based on that for their startup.
We encourage an open source solution to ARC-AGI!
These recent code diffusion models sound simmilar to this idea 🤔
This can be achieved fairly easily by employing a diffusion based LLM reasoning module that has accurate image captioning capabilities that ignore irrelevant details when feeding the llm during inference or test time?
Stop overusing the term LLM. GPTs are not LLMs. LLMs are GPTs.
@@Hohohohoho-vo1pq please refer to the presentation here. Author of the research calls it LLM. czcams.com/video/kYtvqbgCxFA/video.htmlsi=nrFEIED7mmZAE_Zf
@@Hohohohoho-vo1pq
LLMs can be GPTs.
GPTs can be LLMs.
GPT is the architecture. LLM is the size.
They neither imply nor contradict each other.
@@RecursiveTriforce reducing GPTs to as "mere LLMs" is very misleading. People don't even understand what that means when they say that.
But we, as humans, don’t do tree search when solving ARC. We solve it in one shot, almost immediately without trying / searching through different solutions, because the spacial patterns look very obvious, no?
Large multimodal models with ability to input AND output images natively will be able to solve this in one shot. Case in point: gpt5 with native image output.
I’m not a researcher, but the notion that ‘this is obviously not what humans do’ seems moot given that the same can be said about transformers.
> We solve it in one shot
> Spacial patterns look very obvious
The human brain is very good at having 'intuition' to prune a search space. Though it may happen quickly, there are many (maybe infinite) possibilities for decisions humans can make, but yet we prune that down to just a few very quickly
@@ARCprize What do you think of MITs “Introduction to Program Synthesis” course as a starting point?
@@ARCprize this implies that multi modal models can do much better if there is an internal “for loop” that allows it to iterate through different solutions in the hidden space manifold before decoding into output. Only question becomes how to train such a model and what dataset / loss function objective can be used.
Alternatively, would we be able to imitate such a process by having the “thinking” happen in the decoded output in an auto regressive fashion? We human have the ability to “think out loud” as one option and having an LLM think in the decoded output space might make it easier and more familiar to train
(Basically, is chain of thought considered one crude form of discreet search?)
We don't do it in one shot. We go through 100s or 1000s of prediction updates to answer a question. We update our mental models in real time while this is happening.
This is how Stockfish (Chess Engine) works.
Not sure whether it will lead to AGI.
We'd love a submission that tried this approach to see how it goes - super interesting
Did Francois say he would like to see a solution which can solve these puzzles without having been trained on a lot of “ARC like” input output pairs?
This benchmark seems to exist as a subset of the permutation group of operations on coloured grids
(expanding/contracting grid, extruding masses, rotating masses, filling in holes, adding single colours, etc…)
If the “core knowledge” claim is true about ARC, then the discovery of the correct set of “core knowledge” matrix operations could be used to synthetically generate a dataset.
You could then sample thousands of games from a pre-trained policy network given only the first training input, and reinforce the trajectories closest to the revealed answer.
Then sample games given the 1st training input, the first training output, and the 2nd training input, and reinforce again before the test set.
Or is this not allowed?
Francois said he would *like* to see a solution that isn't trained on a bunch of input/output ARC tasks but there isn't a rule that says this isn't allowed.
If as long as your submission only makes 2 final attempts per task you can use it. This means you can test and iterate on the example pairs as much as you'd like.
@ARCprize Hey thanks for the reply! :)
You guys obviously have the generating operations locked away, so this method would only be able to train on "ARC like" operations,
So can I ask what his intuition is on a model being able to center on "core knowledge" priors from training on tasks orthogonal to ARC?
Thanks for making such a great opportunity for the community 👍
Perhaps we want the model to search actively as humans do by manipulating it's environment.
I don't think this problem is going to be easy to solve.
I dont know if he is nervous or what but his is totally red and looks like it is hurting alot.
Francois is on fire, with knowledge.
It's a good idea that the solution has to be made open source.
Awesome, yes, this is to counter act closed-AI research recently