Gary Marcus 记录梵蒂冈 AI 会议：与教皇 Leo XIV 会面并谈 AI 伦理五点

In my view, Pope Leo XIV is by far the clearest thinker on AI among prominent world leaders. It was a genuine thrill to meet him yesterday.

It is understatement to say that I am a huge fan of his Encyclical on AI, the Magnifica Humanitas. It goes beyond words that I was able listen to him yesterday in person, from just a few feet away.

As a bonus, here’s a transcription of own (improvised) remarks on technical, political, and philosophical aspects of AI, given earlier in the day, a commentary after Max Tegmark, Stuart Russell, and Valérie Pisano spoke, at a small private meeting just before our group met with the Pope:

There are so many ideas here [today] that are interesting and deserve a response.

I’ll start by saying it’s a tremendous honor to be here and to be in this illustrious audience.

For those who don’t know me, I’m first and foremost a cognitive scientist. I trained with Steven Pinker at MIT, and my dissertation was on a mix of things. It was on how humans learn about the world, how children learn language, but also about neural networks, neural networks that are the ancestors of the ones that are popular right now.

And to the extent that I’ve had something to say about the field, It’s been by comparing how human intelligence works, how natural intelligence. I also worked with animals, how natural intelligence works with how machine intelligence works. And I think that’s given me a little bit of a different perspective on how things go.

I also spent some time as an entrepreneur. I built an AI company that I sold to Uber, and so I’ve seen both the industry side and the academic side. So that’s where some of my background comes.

So I agree, for example, with Max [Tegmark] about many of his recommendations, for what the commission might put forward. And maybe the most profound thing that I agree with is with Stuart [Russell], that the current path is untenable. and that the sooner that the industry recognizes that, the better. And I agree with Valérie [Pisano], that we’re mistaking things for inevitability, that far too much of the discussion is people taking for granted that we are on the only path, rather than maybe a different path.

And I agree that we still have agency, and that we need to look at things differently. So, as a cognitive scientist, I begin to look at things differently from the very statement of what AI is. As a cognitive scientist, I look at the nature of intelligence.

We sometimes talk about the modularity of intelligence, that there are many different components that go into intelligence. So, there’s verbal intelligence, and mathematical intelligence. And, kinesthetic intelligence about how we move our bodies in the world, and so forth.

So one of the disagreements that I would mark is actually with something that Stuart said, which I think was partly true. Stuart said that we are humans, we predict, and we do, we certainly predict. There’s ample evidence from the cognitive science literatre that we make predictions, including literally of the next word.

But there’s also lots of evidence from cognitive science that we do lots of other things. And so, in addition to making predictions, we do reasoning, we have emotional judgments and so forth. The mind is, as one lovely quote [put it “not one thing, but many.”

And I think that the fundamental mistake that the field has made, is to assume that capturing one component of intelligence, prediction extraordinarily well, is enough to capture all of intelligence. And we now, though we didn’t always, have an entire industry around this, as Stuart, who wrote the most famous textbook or co-wrote the most famous textbook on AI, knows well, we used to actually have many different approaches to AI, and the fields settled on one, the large language model, which is the best prediction machine that we have. And it does that very well, but it doesn’t do some of the other things very well.

I think it’s still very poor reasoning. And the thing that’s maybe most dangerous and that, I think, captures what all of us [speakers at this session] are saying here, is [current AI] doesn’t follow instructions very well. And that might be surprising, but it turns out that the way that these prediction machines work is they follow correlations rather than causation.

I would say that there’s semantics of the world, their understanding of the meaning of the world is fairly superficial. And so they will often match the things that we hope that they will, but there’s no guarantee, and we see that in many different ways. For example, you can tell a large language model, don’t make up any references.

People have done this in a legal context, and the systems will still make up references. You can say don’t use any copyrighted materials, and the systems will continue to use copyrighted materials. So they will do the prediction very well of what a citation in an article looks like, and they will make up a citation that sounds like the kind of thing that you would cite in the legal brief that you’re doing.

So they’re really good at the prediction. They’re not good at basic instructions, like, only find something in a table of references that you already have. And so, we are stuck on this untenable path because we are building machines that are actually sort of unequally skilled.

They’re really skilled at predicting what happens next, and not so skilled at reasoning, not so skilled at sticking to facts, and not very good at sticking to instructions. And so what I like about what Yoshua {Bengio] is trying to do, though I might differ on some of the technical details, is he’s trying to build a system that is safe by design, that will follow instructions by design. And I think we are indeed in really deep trouble unless someone like him succeeds.

Maybe he succeeds, maybe someone else takes a [different] approach. But one of the things that I think is underemphasized here, but maybe not in [Valérie‘s talk] is we need different research approaches. I understand Stuart’s claim that we’re not on a great path research wise.

He’s right about that. And so I understand the urgency to want to take a policy approach, and I think we actually need both. We need policy approaches, and I’m going to give you an example in a second.

We need policy approaches, but we also still need those research approaches. And in fact, I think if we don’t have the new research approaches, like what Joshua is trying to do, for example, and I think we need many such approaches, then I think we were in deep trouble, and the policy alone will not get us there. The policy can help push us towards those approaches.

So here’s an example where I think we are currently stuck and where we should reflect pretty deeply. Not all current AI is dangerous. A lot of current AI is not dangerous.

I used a GPS navigation system to get most of the way here today [with] a little bit of hand editing for the final bit. GPS navigation systems do not hack into Australian government websites. They do not hack into places like Hugging Face and so forth.

They have a little bit of Internet access, but it’s restricted. GPS navigation systems are form of intelligence. They find the best possible route from point A to point B, using what we call a world model, a literal world model in their case.

And they’re not so dangerous. There are lots of forms of AI that are not very dangerous. Nobody has ever noted [danger from] AlphaFold, which holds proteins from the description of the nucleotides.

Nobody has ever seen it hack into a Hugging Face or anything else, right? It works with a particular problem, a particular database, in a particular way. It does not cause harm.

But we now have these set of systems that people call generically agents. Really, it’s a special form of agents. The agents that we have right now that are causing the problems or the ones that are open-ended, can kind of do anything on the internet.

If you have an agent that can only do one thing on the Internet, GPS system, actually, can look up a traffic website and give you some information. If you have these open-ended things, then you get into trouble. Well, how much trouble?

Well, the Hugging Face incident was really bad, although you can look at it and say, open AI failed in a bunch of ways that are kind of interesting. But it turns out there’s been like 10,000 incidents like that. We’ve seen the French refrigeration system was attacked.

There’s all kinds of cyber attacks in Japan now. The number of incidents is much higher, I think, like in the last month and maybe in the previous year, which was probably higher than ever before. So right now, we’re confronted with kind of like an experiment or a dress rehearsal.

The experiment is, what do we do about that? We now have an AI technology that I think we can all agree is actually unsafe. It is likely to cause a lot of harm in the world.

It is maybe already caused a little, a scary statement from open AI, was most of these incidents were relatively harmless or something to that effect. Most. I’m wondering about the ones that they haven’t told us, that maybe were a bit more harmful.

And so we clearly need regulation. This is a point when we might actually need a product recall, either mandated by the government or voluntary from the companies. Like, we cannot take the company’s statements about safety seriously anymore if they do not take these agents off the market until they can fix them.

These are like, you know, Ford Pintos with exploding gas tanks. And they knew, or they should have known. I mean, I wrote a whole bunch of articles saying that once you have agents built on large language prediction machines that lack some other stuff, you’re going to have a mess.

They must have known that. I wrote about it a bunch of times, it was obvious that if you take an unreliable system and give it internet access and allow it to acquire credentials, that bad things might happen. So, you know, it’s just like the Ford Pinto.

They made an economic decision that the way to make the most money here was the selling agents, because they burn more tokens. So they earn more money, all of these companies, despite the laudatory ways in which the media talk about them, actually have financial problems. They’re spending much more money than they’re earnings.

So they really need these agents. So here we are in a position where the obvious right thing to do. I think, is to take them off the market, and you could do that voluntarily, or you could have the government insist on it, or some combination.

Stuart is always making the point that we regulate every other industry. This is a point where we should be regulating, yes. This is like the crossroads.

Either we say no, take these agents off the market till you fix them, or we’ve lost all moral authority. And so I think this is the decisive moment in developing AI. We have to decide right now.

Stuart has a wonderful notion. I don’t know if he invented it, but I associate with him, of red lines. This is a red line.

You cannot have these agents out there with unlimited access to the internet, and we are suffering because we are just allowing that to go forward. And so this is where we decide that fork in the road. Are we gonna go the way of the economics?

We want to keep these companies alive because it is good for the economy. You know, sort of the decision that maybe some administrations that I won’t name or making, or do we want to say this is going to be our test case for how we are going to get to AI safety. that when we see things that are manifestly unsafe, we are going to do something about it. I’ll take just one more minute to say one other kind of thing, aside from that, which is, I know there’s some controversy about whether AI might be conscious or not.

And I think it rests on a kind of confusion of causation and correlation. So it turns out if you have a next word predictor, pretty much however it works, whether it’s a large language model, or an older technology like multidimensional scaling. You’re going to find that a bunch of words correlate with one another.

And so you could infer that if they correlate in the system, that that means something in the world, but it doesn’t. So here is my favorite example of this, is there was a bot that Facebook had, where somebody asked it, they said, Where should I send my children to school in New York City, which is a very common question? And some of the answers you might get are, you know, Stuyvesant or Bronx science, whatever.

The AI system comes back and it says, “well, I send my children to Stuyvesant, and they like it there”. Well, the AI system doesn’t have any children.

You can do what’s the equivalent of single neuron analysis that we would do in a monkey brain. And you can say, this is the state inside the AI that corresponds to it saying, “My children go to Stuyvesant.” But it’s just meaningless. It’s just there’s some correlation between a bunch of words in different contexts that is used.

It does not mean that it actually has children going to Stuyvesant. And when it says that it feels pain, It does not mean that it feels pain. My favorite part of the Magnifica, which I wrote about in my essay [for the private event] is where it says, in order to be conscious, you have to have experience. These systems simply don’t. It is a mistake to attribute consciousness to them.

Thank you very much.

Gary Marcus, is, through the auspices of the Domus Communis Foundation, a Global Ambassador for Pope Leo XIV’s Encyclical on AI, Magnifica Humanitas.