Victor Taelin recently posted a challenge about the claimed reasoning abilities of AI/LLMs:

https://x.com/victortaelin/status/1844886809005687270?s=46

A short proof that LLMs (even o1) still can't reason:

Consider the problem of inverting a perfect binary tree. That's an old, entry-level interview question that humans and LLMs can solve easily. Now, let's add just 3 key nuances to make it new and unique:

1. It must invert keys ("bit-reversal permutation")

2. It must be a dependency-free, pure recursive function

3. It must have type Bit -> Tree -> Tree

These small changes are enough to move this problem out of the "memorized solution zone". It isn't on the internet. And, guess what? This is enough to make it completely intractable to modern AIs. All of them fail miserably at it, no matter how you prompt it.

This is very relevant, because the problem is still easy to a human researcher, and being capable of solving it is clear pre-requisite to contribute to CS research. Yet, all modern AIs fail miserably. As much as I love LLMs, truth is: they do NOT reason, and they will never do CS.

Some prompts for you to try:

gist.github.com/VictorTaelin/4…

I'm willing to give $10k to anyone who shows any AI capable of implementing this function correctly. It just won't work, no matter how long it thinks. (The solution is 7 lines of code!)

This market will resolve YES if anyone produces a demonstration that wins this $10k by the end of 2025. I will wait for confirmation from Victor before resolving YES. This market will resolve NO at the end of 2025 otherwise.