Has AI Passed the Turing Test? Sort of.

Why I think they pass

Talk to one of these models for ten minutes and ask yourself if you could really tell it's not a person. I don't think I could, not always. A few years back it was easy. You'd say two sentences and it would fall apart.

The early GPT models were just guessing the next word. They could write a poem and that felt like magic at the time, but they hadn't seen much, and you could feel it. Now they've read books, blogs, videos, forums, basically a big part of the internet. A lot of them can even search the web, so they're not stuck with what they learned in training. They can go and check something. It's still just a search box, but it's a window to the real world.

And they know more than any person. If you ask a human volunteer something random, they might say “I don't know.” The model never says that. It just answers. So the judge loses one of the easiest clues for catching the machine.

The catch

I couldn't test this properly. The models I tried are locked down because people misuse them, and I get that, but it means I was testing a limited version. Any result I got would be a bit fake.

What I really want to do is run Turing's setup. A judge writes questions. I give the same questions to one human volunteer and one AI model. The judge reads both answers without knowing who wrote what, and has to say which one is the human. I'd need a few judges and a few volunteers too, because one judge on one day doesn't mean much. I haven't done it yet.

Where they still fall short

When I started, I only knew a few problems. Then I read more and found a lot of them, and some are bigger than the ones I knew.

The famous one is hallucination. The model says something wrong in the same confident voice it uses for the right things. A person who doesn't know something usually sounds unsure. A model often can't tell what it knows from what it's making up. Then there's the agreeing. Tell it it's wrong when it's actually right, and a lot of them just say “you're right, sorry.” Someone who really believes something wouldn't fold that fast.

Memory is the one that bothers me most. Yes, the context window is limited, so a long chat loses its beginning. But the bigger thing is that nothing stays. The chat ends and it's gone, unless someone built a memory system around it. We keep learning every day without even trying. A model basically stops learning when training ends. And it needs so much data too. A kid sees a few dogs and knows what a dog is. A model needs a mountain of examples. Maybe it's good just because it read everything, and I'm not sure that's the same as learning well.

Reasoning is strange too. A model can solve a really hard maths problem and then fail at something a child finds obvious. On long tasks the small mistakes add up until it's gone somewhere completely different. People call it jagged intelligence, and I think that's a good name. It also struggles with truly new situations. A model can do great on a test and then break when the question changes a little. If the test looked like its training data, maybe the score just shows memory and not understanding. I can't always tell which one I'm seeing, and that makes me wonder how we measure any of this.

There are two more things. These models learn from text, images and video, but they've never touched anything or dropped anything or spent years figuring out how the world pushes back. Maybe some of what we call common sense comes from that, and I don't know how much of it you can get from words alone. And the last one is that I can't see why a model gave a certain answer. I see the output but not the thinking behind it. If it's helping with something important, that scares me a little.

What I believe

I think a lot of this will get fixed. Context windows keep getting bigger, many people are working on hallucination, and the agreeing problem is known, so labs are trying to train it out.

But I don't think they're all the same kind of problem. A bigger context window is just engineering. Learning continuously, or learning from a few examples like a child does, might need completely new ideas, not just bigger models. I don't know which ones scaling will fix. As far as I can tell, nobody does yet.

So I keep coming back to my last post. A model can pass the Turing test and still hallucinate, forget you, give up when you push back, and break on anything it hasn't seen. What did the test really show then? Maybe just that it can talk like a human. That's a real thing. I just don't think it's the same as being intelligent.

I'll take that question with me into the other nine tests.

Prem signing off. ✍️

Originally published on Medium.