Video

i might be tripping but i think this dude cloned his own voice and then layered it into the video. you can hear the typical elevenlabs lisp

Why is the train/evaluation set so small?

Mark Knoop is a very smart guy. Knows a lot about ML for being an outsider in that space and meche background

I am not entirely convinced about active inference. Training on unlabelled test data is an old trick on Kaggle to improve score but does it really say anything about intelligence?

Excellent.

third since no one called it :kappa:

Does your submission count if you make use of private models like gpt4 at some point in your algorithm?

So he is saying that LLMs guiding hyperparameter space will get us to AGI? Seems a little simplistic to me ... and there are better ways to search that space.

Upp

I'm confused...what are we supposed to attach with our API add-on secret?

I understand retries, but I am confuced with the two attempts. Do you always need to provide two? In which case they would have different data and both would be required for 100% correct prediction? I also missed the part in which the prediction and correct answers are matched and prounounced.

When you think about it, the optimal network should be like a physics simulator, every example has its own stable rules. My guess is that a recurrent network would have the best chance. Though the parameter count would need to be huge so we could perhaps make a Hypernet to generate the weights from scratch.

Second 😂

How did this change so much, so quickly. ARC-AGI-PUB now has code that gets ~42%. Looking at it now, and it's pretty awesome.

first.

Here is my little observation I noticed while playing with ARC. It is very dependent on the assumed geometry of the grid. Meaning, that if we imagine that each cell is a node of the graph and we do not know how the graph is connected we cannot build a program that describes the transformations of the particular task. So we need to assume some way it is connected - i.e. geometry of the grid. For each task the geometry is different and depending on how well we assumed the geometry the program that ‘solves’ the task might be either simple or complicated. There are several consequences that follow from this. Firstly, for each task there might be a huge amount of "correct" solutions that exist on the unfamiliar geometries but with very simple generating programs. Such solutions most likely will not be qualified as a correct one. Secondly, to solve ARC an AI should be aligned to the expectations of the human of how the proper task should look like and might be solved. And lastly, most of the ARC-like tasks generated via some algorithm that assumes random geometry of the grid and simple program would be completely unsolvable for almost any human in most of the cases. So, my main point is, ARC-like tasks exist in the much broader space that we tend to assume. And the only reason they are solvable for us at all is that the subset of them that was created by François reminds us of tasks we have already encountered and we can reliably guess what François had in mind while creating them.

Sorry this isn’t general intelligence. This is just reasoning. It is painful watching a whole industry trying to reinvent psychology when there is shady a century of research there.

So have you collaborated with any psychologists to make this test

what is the relation between the precentage of questions the AI gets correct and the iq of the models

It can't be that hard right

i think i might have unintentionally set the basis for solving this in a project i did a couple months ago

Is this related to P vs NP in an interesting way?

I have an idea now, thanks. I’ll probably check out ARC after my PhD qualifying exam. Finetuning is gonna be fun 🤩

The question is if someone manages to get high score in ARC style problems will they really share the code for such little amount of money rather than raising money from venture capital farms based on that for their startup.

There are also birds such as Sulphur Crested Cockatoos that have shown problem solving skills. Hopefully it's proof enough that a basic reasoning model won't require a trillion parameters.

IMO, solving ARC is necessary for AGI, though it may not be sufficient It's not obvious to me whether a solution to ARC would generalize to other problems Like Francois mentioned, no benchmark is perfect, and there may be ways to "cheese" ARC!

It's a good idea that the solution has to be made open source.

Lets get to the bottom of this. How much for getting 90 % accuracy on a free llm model? How much do i get for that?

I advise you to not get that cooky. Otherwise ill solve everything and demand my prize 😂

Did Francois say he would like to see a solution which can solve these puzzles without having been trained on a lot of “ARC like” input output pairs? This benchmark seems to exist as a subset of the permutation group of operations on coloured grids (expanding/contracting grid, extruding masses, rotating masses, filling in holes, adding single colours, etc…) If the “core knowledge” claim is true about ARC, then the discovery of the correct set of “core knowledge” matrix operations could be used to synthetically generate a dataset. You could then sample thousands of games from a pre-trained policy network given only the first training input, and reinforce the trajectories closest to the revealed answer. Then sample games given the 1st training input, the first training output, and the 2nd training input, and reinforce again before the test set. Or is this not allowed?

I dont know if he is nervous or what but his is totally red and looks like it is hurting alot.

Is there a way to know all the priors you embed into the puzzles? So far I’ve identified: 1. Translations - Shifting objects or patterns across the grid. 2. Rotations - Rotating objects or patterns at different angles. 3. Reflections - Flipping objects or patterns across a line. 4. Scaling - Changing the size of objects or patterns. 5. Repetition and symmetry - Repeating patterns or creating symmetrical designs. 6. Color changes - Altering the color of objects or patterns. 7. Compositions - Combining multiple operations or transformations. 8. Object addition or removal - Adding or removing elements within the grid. 9. Changes of the size matrices - Modifying the dimensions of the grid or the objects within it.

The video gives me such interesting vibes! I feel like it will bring thousands of smart minds all working on a single problem, and I can't wait to see the results! Humanity is truly amazing when we work together on something

These recent code diffusion models sound simmilar to this idea 🤔

Perhaps we want the model to search actively as humans do by manipulating it's environment. I don't think this problem is going to be easy to solve.

If u cant design an ai architecture to solve this problem, u arent as smart as you think.

This can be achieved fairly easily by employing a diffusion based LLM reasoning module that has accurate image captioning capabilities that ignore irrelevant details when feeding the llm during inference or test time?

chollet has been my favorite voice on AI for 5+ years and I don't see that changing!

This is how Stockfish (Chess Engine) works. Not sure whether it will lead to AGI.

But we, as humans, don’t do tree search when solving ARC. We solve it in one shot, almost immediately without trying / searching through different solutions, because the spacial patterns look very obvious, no? Large multimodal models with ability to input AND output images natively will be able to solve this in one shot. Case in point: gpt5 with native image output.

Love this

im glad somebody came there to share the misconception that most Models that exist nowadays are NOT ai but just an overly complex imitating machine even tho it can imitate a lot, learning skills is more of a natural thing, yet not described with code, witch is the goal here, witch is so fun to realize that its more like the process of creating artificial life more than anything

Some real brainpower behind this project. I've longed for a counter example that really signifies the shortcomings of current SOTA models, it was a gut feeling but now we can actually know with some certainty!

ok thanks, I'll try to build this solution with ChatGPT

children can solve these puzzles but i dont think LLM's can

Let's go! I'm all in on this, I will say: don't count out the power of one shot

Yeah I'm excited about this, but isn't it likely that the best solution to ARC will be to treat it like a specialized game like chess? I'm not sure you need intelligence to do well on the problem set. It's very well defined. I can see doing pretty well on the examples by treating each problem category independently. if(moving problem) do moving solution. if(filling out problem) do filling out pattern. If(color change problem) do color change solution. if(subset problem) take the subset. etc.

thx for demonstrations, this taks feel like arbitrarily single step arbitrary state transition in cellular automaton. It also looks like fun to play 😄

Thank you! I know this has been around for a while, but I'm happy to see a legitimate attempt at testing intelligence that isn't "It passed the Turing test." LLMs sound smart because they speak our language, but are they really doing anything more than regurgitating memorized information? This test shows that most likely not really.

Thank you for this.