AI makes stunning progress in logical reasoning (Patreon)

[This is a transcript with links to references.]

Google has unveiled a new artificially intelligent system, AlphaGeometry, that can solve problems of mathematical geometry. It’s the first computer program to surpass the average performance of participants at the International Mathematical Olympiad. That might sound like an incremental improvement, just one more thing that AI is really good at, but mathematics isn’t just one more thing, it’s everywhere. This makes Google’s recent development a significant step forward. Let’s have a look.

This new research was done by scientists at Google DeepMind and Google Research and just published in Nature. They tested their new program AlphaGeometry with a set of problems posed at the international mathematical Olympiad during the years 2000 to 2022.

After the program was trained, it was given 30 Olympiad geometry problems. The AI solved 25 of them correctly within the standard Olympiad time limit. This performance far surpassed the capabilities of the previous state-of-the-art system, which could only solve 10 of these geometry problems. To put this into perspective, the average participant at these Olympiads solves about 15 problems correctly.   A gold medallist typically solves almost 26. So, the new AI is better than the average, but not quite beating the smartest of the smart humans.

Clearly this AI is doing very well, but how does it work? AlphaGeometry uses what’s called a neuro-symbolic approach. That means it combines a neural language model,like ChatGPT, with symbolic deduction like that used by software like Mathematica.

The neural language models are good at identifying general patterns and relationships in data. This makes it possible to quickly come up with potentially useful ideas. These symbolic deduction programs on the other hand basically allow to infer logical relationships. The combination of both makes for a very, very powerful system. It’s indeed much more similar to how the human brain works than just neural networks because it combines "intuitive" ideas that we extract from input, with more deliberate, rational decision-making. Similar to Kahneman’s system 1 and system 2 thinking.

It’s like basically AlphaGeometry has what ChatGPT is missing, it understands logic.

One of the problems that has so far prevented AI from becoming good at maths in general or geometry in particular is the lack of training data. There are only so many proofs that humans have written down that you can train them on. The Google researchers have addressed this by first generating a vast pool of synthetic proofs that added up to as much as 100 million examples.  Thanks to this, AlphaGeometry could train without relying on human demonstrations.

Better still, AlphaGeometry doesn’t just spit out a result, it delivers a human-readable step-by-step proof. Basically it works by trying to find a sequence of steps that logically fit together and that also solves the problem. Though it isn’t irrelevant to mention that most of the proofs it found were considerably longer than those from humans.

But the significance of AlphaGeometry's achievement goes beyond solving geometry problems. It generally highlights AI's growing ability to reason logically, discover new knowledge, and verify solutions. And not only this, it can also explain how it arrived at the conclusion. Such a type of AI system has uses that extend way beyond just geometry. Not only can this achievement be generalized across various mathematical domains, but it will without doubt also come in handy in other areas where rational thinking and logical deduction is of use. Like, eh, everything.

Now throw in a bunch of data in addition to the reasoning skills and you have the perfect scientist, unbiased, clearheaded, able to digest huge amounts of data and draw logical conclusions from it.

That AlphaGeometry is able to explain how it arrived at a conclusion should also help alleviate the fears of people who think that AI will be a black box that no human can follow. You see, the thing with mathematical proofs or arguments in general is that it’s difficult to come up with correct chains of reasoning, but once you have the proof it’s much simpler to check whether its correct. Though simpler doesn’t mean simple…

These developments in AI also raise an interesting philosophical question, which is whether there is anything humans can do which AI will not eventually also be able to do. At the moment, the major protection for jobs from being taken over by AI is that a lot of human labour requires sensor input or physical skills that robots don’t have, or AI can’t cope with. Yet. Because robots are developing incredibly quickly, and if you combine this development with AI, there really won’t be much left for us to do.

Maybe we’ll all just make YouTube videos? Well, actually, Google is also working on a text-to-video system that isn’t yet publicly available but from which they shared some amazing examples last week. So soon enough I can replace myself with an AI which will make total sense because all my viewers will be robots.