\n<\/p>\n
There are many ways to test the intelligence of an artificial intelligence\u2014conversational fluidity, reading comprehension or mind-bendingly difficult physics. But some of the tests that are most likely to stump AIs are ones that humans find relatively easy, even entertaining. Though AIs increasingly excel at tasks that require high levels of human expertise, this does not mean that they are close to attaining artificial general intelligence, or AGI. AGI requires that an AI can take a very small amount of information and use it to generalize and adapt to highly novel situations. This ability, which is the basis for human learning, remains challenging for AIs.<\/p>\n
One test designed to evaluate an AI\u2019s ability to generalize is the Abstraction and Reasoning Corpus, or ARC: a collection of tiny, colored-grid puzzles that ask a solver to deduce a hidden rule and then apply it to a new grid. Developed by AI researcher Fran\u00e7ois Chollet in 2019, it became the basis of the ARC Prize Foundation, a nonprofit program that administers the test\u2014now an industry benchmark used by all major AI models. The organization also develops new tests and has been routinely using two (ARC-AGI-1 and its more challenging successor ARC-AGI-2). This week the foundation is launching ARC-AGI-3, which is specifically designed for testing AI agents\u2014and is based on making them play video games.<\/p>\n
Scientific American spoke to ARC Prize Foundation president, AI researcher and entrepreneur Greg Kamradt to understand how these tests evaluate AIs, what they tell us about the potential for AGI and why they are often challenging for deep-learning models even though many humans tend to find them relatively easy. Links to try the tests are at the end of the article.<\/p>\n
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.<\/p>\n
[An edited transcript of the interview follows.]<\/p>\n
What definition of intelligence is measured by ARC-AGI-1?<\/p>\n
Our definition of intelligence is your ability to learn new things. We already know that AI can win at chess. We know they can beat Go. But those models cannot generalize to new domains; they can\u2019t go and learn English. So what Fran\u00e7ois Chollet made was a benchmark called ARC-AGI\u2014it teaches you a mini skill in the question, and then it asks you to demonstrate that mini skill. We\u2019re basically teaching something and asking you to repeat the skill that you just learned. So the test measures a model\u2019s ability to learn within a narrow domain. But our claim is that it does not measure AGI because it\u2019s still in a scoped domain [in which learning applies to only a limited area]. It measures that an AI can generalize, but we do not claim this is AGI.<\/p>\n
How are you defining AGI here?<\/p>\n
There are two ways I look at it. The first is more tech-forward, which is \u2018Can an artificial system match the learning efficiency of a human?\u2019 Now what I mean by that is after humans are born, they learn a lot outside their training data. In fact, they don\u2019t really have training data, other than a few evolutionary priors. So we learn how to speak English, we learn how to drive a car, and we learn how to ride a bike\u2014all these things outside our training data. That\u2019s called generalization. When you can do things outside of what you\u2019ve been trained on now, we define that as intelligence. Now, an alternative definition of AGI that we use is when we can no longer come up with problems that humans can do and AI cannot\u2014that\u2019s when we have AGI. That\u2019s an observational definition. The flip side is also true, which is as long as the ARC Prize or humanity in general can still find problems that humans can do but AI cannot, then we do not have AGI. One of the key factors about Fran\u00e7ois Chollet\u2019s benchmark… is that we test humans on them, and the average human can do these tasks and these problems, but AI still has a really hard time with it. The reason that\u2019s so interesting is that some advanced AIs, such as Grok, can pass any graduate-level exam or do all these crazy things, but that\u2019s spiky intelligence. It still doesn\u2019t have the generalization power of a human. And that\u2019s what this benchmark shows.<\/p>\n
How do your benchmarks differ from those used by other organizations?<\/p>\n
One of the things that differentiates us is that we require that our benchmark to be solvable by humans. That\u2019s in opposition to other benchmarks, where they do \u201cPh.D.-plus-plus\u201d problems. I don\u2019t need to be told that AI is smarter than me\u2014I already know that OpenAI\u2019s o3 can do a lot of things better than me, but it doesn\u2019t have a human\u2019s power to generalize. That\u2019s what we measure on, so we need to test humans. We actually tested 400 people on ARC-AGI-2. We got them in a room, we gave them computers, we did demographic screening, and then gave them the test. The average person scored 66 percent on ARC-AGI-2. Collectively, though, the aggregated responses of five to 10 people will contain the correct answers to all the questions on the ARC2.<\/p>\n
What makes this test hard for AI and relatively easy for humans?<\/p>\n
There are two things. Humans are incredibly sample-efficient with their learning, meaning they can look at a problem and with maybe one or two examples, they can pick up the mini skill or transformation and they can go and do it. The algorithm that\u2019s running in a human\u2019s head is orders of magnitude better and more efficient than what we\u2019re seeing with AI right now.<\/p>\n
What is the difference between ARC-AGI-1 and ARC-AGI-2?<\/p>\n
So ARC-AGI-1, Fran\u00e7ois Chollet made that himself. It was about 1,000 tasks. That was in 2019. He basically did the minimum viable version in order to measure generalization, and it held for five years because deep learning couldn\u2019t touch it at all. It wasn\u2019t even getting close. Then reasoning models that came out in 2024, by OpenAI, started making progress on it, which showed a step-level change in what AI could do. Then, when we went to ARC-AGI-2, we went a little bit further down the rabbit hole in regard to what humans can do and AI cannot. It requires a little bit more planning for each task. So instead of getting solved within five seconds, humans may be able to do it in a minute or two. There are more complicated rules, and the grids are larger, so you have to be more precise with your answer, but it\u2019s the same concept, more or less…. We are now launching a developer preview for ARC-AGI-3, and that\u2019s completely departing from this format. The new format will actually be interactive. So think of it more as an agent benchmark.<\/p>\n
How will ARC-AGI-3 test agents differently compared with previous tests?<\/p>\n
If you think about everyday life, it\u2019s rare that we have a stateless decision. When I say stateless, I mean just a question and an answer. Right now all benchmarks are more or less stateless benchmarks. If you ask a language model a question, it gives you a single answer. There\u2019s a lot that you cannot test with a stateless benchmark. You cannot test planning. You cannot test exploration. You cannot test intuiting about your environment or the goals that come with that. So we\u2019re making 100 novel video games that we will use to test humans to make sure that humans can do them because that\u2019s the basis for our benchmark. And then we\u2019re going to drop AIs into these video games and see if they can understand this environment that they\u2019ve never seen beforehand. To date, with our internal testing, we haven\u2019t had a single AI be able to beat even one level of one of the games.<\/p>\n
Can you describe the video games here? <\/p>\n
Each \u201cenvironment,\u201d or video game, is a two-dimensional, pixel-based puzzle. These games are structured as distinct levels, each designed to teach a specific mini skill to the player (human or AI). To successfully complete a level, the player must demonstrate mastery of that skill by executing planned sequences of actions.<\/p>\n
How is using video games to test for AGI different from the ways that video games have previously been used to test AI systems? <\/p>\n
Video games have long been used as benchmarks in AI research, with Atari games being a popular example. But traditional video game benchmarks face several limitations. Popular games have extensive training data publicly available, lack standardized performance evaluation metrics and permit brute-force methods involving billions of simulations. Additionally, the developers building AI agents typically have prior knowledge of these games\u2014unintentionally embedding their own insights into the solutions.<\/p>\n
Try ARC-AGI-1, ARC-AGI-2 and ARC-AGI-3. <\/p>\n","protected":false},"excerpt":{"rendered":"
There are many ways to test the intelligence of an artificial intelligence\u2014conversational fluidity, reading comprehension or mind-bendingly difficult physics. But some of the tests that are most likely to stump AIs are ones that humans find relatively easy, even entertaining. Though AIs increasingly excel at tasks that require high levels of human expertise, this does<\/p>\n","protected":false},"author":1,"featured_media":11351,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[50],"tags":[2372,4690,1564,350,4692,3698,1443,4691,3998],"class_list":{"0":"post-11350","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-environment","8":"tag-ace","9":"tag-ais","10":"tag-artificial","11":"tag-fail","12":"tag-general","13":"tag-humans","14":"tag-intelligence","15":"tag-pave","16":"tag-tests"},"_links":{"self":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts\/11350","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11350"}],"version-history":[{"count":0,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/posts\/11350\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=\/wp\/v2\/media\/11351"}],"wp:attachment":[{"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11350"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11350"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/naijaglobalnews.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11350"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}