fchollet

5 ms·

1.
by fchollet 6mo ago
It is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
2.
by fchollet 6mo ago
There are no tricks. Our approach to reducing the impact of targeting (without fully eliminating it) is described in the paper.
3.
by fchollet 6mo ago
I'm all for testing humans and AI on a fair basis; how about we restrict testing to robots physically coming to our testing center to solve the environments via keyboard / mouse / screen like our human testers? ;-) (This vers
4.
by fchollet 6mo ago
Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solv
5.
by fchollet 11mo ago
One interesting observation is that French-derived words in English tend to be fancier -- formal, sophisticated, higher-class -- while Germanic ones tend to be more casual, everyday vocabulary.
6.
by fchollet 1y ago
The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed. By the time OpenAI attempted ARC in 2024, a colossal amount of resources had already been expended trying to beat the benchmark. The Ope
7.
by fchollet 1y ago
You can easily convert these tasks to token strings. The reason why ARC does not use language as part of its format is that it seeks to minimize the amount of prior knowledge needed to approach the tasks, so as to focus on fluid intelligenc
8.
by fchollet 1y ago
The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. ARC 1 was released long before in-context learning was identified in LLMs (and designed befor
9.
by fchollet 1y ago
There have been some human studies on ARC 1 previously, I expect there will be more in the future. See this paper from 2021, which was one of the earliest works in this direction: https://arxiv.org/abs/2103.05823
10.
by fchollet 1y ago
It's useful to know what current AI systems can achieve with unlimited test-time compute resources. Ultimately though, the "spirit of the challenge" is efficiency, which is why we're specifically looking for solutions th
11.
by fchollet 1y ago
ARC 3 is still spatially 2D, but it adds a time dimension, and it's interactive.
12.
by fchollet 2y ago
> Who would be buying bitcoin right now? Well, maybe the US government? What if the US starts dedicating 10-15% of yearly federal receipts to serve as exit liquidity for Bitcoin holders?
13.
by fchollet 2y ago
What all top models do is recombine at test time the knowledge they already have. So they all possess Core Knowledge priors. Techniques to acquire them vary: * Use a pretrained LLM and hope that relevant programs will be memorized via expos
14.
by fchollet 2y ago
It is correct that the first model that will beat ARC-AGI will only be able to handle ARC-AGI tasks. However, the idea is that the architecture of that model should be able to be repurposed to arbitrary problems. That is what makes ARC-AG
15.
by fchollet 2y ago
I will never enter ARC Prize myself, since I'm organizing it. But the reason I made ARC in the first place was to work on it myself! I intend to solve it (outside of the context of the competition).
16.
by fchollet 2y ago
ARC was never supposed to grade LLMs! I designed the ARC format back when LLMs weren't a thing at all. It's a test of AI systems' ability to generalize to novel tasks.
17.
by fchollet 2y ago
I believe the MindsAI solution does feature novel ideas that do indeed lead to better generalization (test-time fine-tuning). So it's definitely the kind of research that ARC was supposed to incentivize -- things are working as intende
18.
by fchollet 2y ago
My go-to DL stack is Keras 3 + JAX. W&B is a great tool as well. I think JAX is generally under-appreciated compared to how powerful it is.
19.
by fchollet 2y ago
Keras is now standalone and multi-backend again. Keras weights files from older versions are still loadable and Keras code from older versions are still runnable (on any backend as long as they only used Keras APIs)! In general the ability
20.
by fchollet 2y ago
This roughly aligns with my timeline. ARC will be solved within a couple of years. There is a distinction between solving ARC, creating AGI, and creating an AI that would represent an existential risk. ARC is a stepping stone towards AGI, s
21.
by fchollet 2y ago
Yes to both.
22.
by fchollet 2y ago
Actually, `keras.distribution` is straightforward to implement in TF DTensor and with the experimental PyTorch SPMD API. We haven't done it yet first because these APIs are experimental (only JAX is mature) and second because all the d
23.
by fchollet 2y ago
Enjoy the book!
24.
by fchollet 2y ago
That's what I plan on doing -- so I would say yes :)
25.
by fchollet 2y ago
Thank you!
26.
by fchollet 2y ago
This is just the standard Google ladder. Your initial level when you join is based on your past experience. Then you gain levels by going through the infamous promo process. L8 represents the level of Director. Yes, there are conflicts of i
27.
by fchollet 2y ago
Thanks for the kind words -- glad Keras has been useful!
28.
by fchollet 2y ago
Right, one rather uninteresting line of approaches to ARC consists of trying to anticipate what might be in the test set, by generating millions of synthetic tasks. This can only work on relatively simple tasks, since the chance of task col
29.
by fchollet 2y ago
I will still be US-based for the time being. I'm seeing great things happening on the AI scene in Paris, though!
30.
by fchollet 2y ago
Keras 1.0 in 2016-2017 was much less flexible than Keras 3 is now! Keras is designed around the principle of "progressive disclosure of complexity": there are easy high-level workflows you can get started with, but you're alw

More ›