The ‘But China!’ Dilemma Driving the A.I. Race

This is an edited transcript of “The Ezra Klein Show.” You can listen to the episode wherever you get your podcasts.

Look, I don’t know if you are following every article, every tweet, every blog post from the A.I. labs right now, but we are in a frightening place.

There is a growing sense that we actually need to do something — we need to pace the frontier, we need to slow all this down.

But if you talk to anyone in Washington or at the A.I. labs about this, you just crash into the shoals of: Well, what about China?

If we slow down, we will lose the A.I. race to China. And as dangerous as it is to build these things we can’t control, it is even more dangerous to have them in China’s hands if they’re not in ours.

We’re on the eve of talks between Donald Trump and Xi Jinping. Behind these will be talks between Scott Bessent and his Chinese counterpart that are more tightly focused on A.I.

The expectations for these talks are not very high, both because of the broad relationship between the U.S. and China and because neither side really seems to know what they want to do.

But these talks are at least a beginning — of relationships and maybe frameworks and approaches. If things continue to get crazier and action is needed, maybe they are a platform we can stand on.

So I wanted to talk to somebody today who’s an expert on China and A.I. — how they regulate it, how they approach it, the relationship between China and America on this topic and also somebody who has thought about what talks like this could achieve — what is realistic within the operating frameworks of the two superpowers.

My guest today is Matt Sheehan, a senior fellow at the Carnegie Endowment for International Peace. He has been closely following and studying China’s regulations and governmental structure on A.I. He has been involved in U.S.-China A.I. talks. He also has a great Substack on these topics. Sheehan is the author of the 2019 book “The Transpacific Experiment: How China and California Collaborate and Compete for Our Future.”

This conversation was recorded on Sept. 10.

Ezra Klein: Matt Sheehan, welcome to the show.

Matt Sheehan: Thanks very much for having me.

The dominant metaphor, for the relationship between China and America on A.I. that exists in Silicon Valley, that exists in Washington, is this metaphor of the race. The ending of this race is superintelligence — that some company, or some country, is going to have the moment where their well-aligned, safe model moves into recursive self-improvement and goes [FOOM!]!

This used to get called, in the rationalist community, the FOOM moment. And at varying levels of explicitness, people in Washington, to me, seem to have this model in their head that we are racing China toward this kind of supremacy.

Does China buy this race model? Is that how they see it? And do you buy this race model? Is that how you see it?

In terms of: Does China buy this race model? — it’s definitely not the dominant paradigm that has been informing A.I. policy across the country writ large. And it doesn’t have the chokehold that it does in the U.S.

In China, it’s like: Huh, OK, that could happen. That’s a potential technical path forward. We’re kind of looking for evidence on this. We see that America is very concerned about this.

But China has not taken the steps that you might think they would take if they were ultimately laser focused on that type of thing. China is very constrained on compute. They have far, far less compute than the U.S. Some estimates are that they have one-eighth the compute of the U.S., maybe one-tenth of the compute of the U.S. does as of next ——

These are the chips, the G.P.U.s that all of these A.I.s are trained on and then run on.

Exactly. And most people think that one of the key determinants of how powerful your model is how much compute, how many chips, you are using to train it.

With China being so compute constrained, if they were really laser focused on this massive takeoff scenario, you might expect them to start consolidating all that compute, make your bet on DeepSeek or another company, and go from there.

And we haven’t seen that. Actually, in terms of the major A.I. policy documents that have come out, they’ve taken a very diffuse approach to compute.

They’ve said things like: Our No. 1 concern is A.I. applications, and we want to incentivize every mayor, every governor, every state-owned enterprise to look for ways to apply A.I. to manufacturing, apply A.I. to traffic lights, apply A.I. to upgrading the robotics industry.

And those actions, of focusing on applications and really diffusing your compute throughout the country, are not what you would expect for a government that is laser focused on this takeoff.

It’s possible that changes. It could change very quickly, and I think, as some people in America beat this drum louder and louder, you have to imagine that it’s going to seep into their consciousness in that way, or seep into their beliefs about the way this is going. But so far, we have not seen that evidence.

Every single conversation I have with politicians, with A.I. lab leaders, about regulating the frontier of A.I., always falls apart on this “But China!” problem.

Maybe there are things we could do to regulate the pace of the frontier here in America — but China will race forward. But China will create recursive self-improving A.I., and either we have the same dangers that we would have had were it here — but now it is under control of a competitive foreign country with a very different political system than ours.

So how do you see the “But China!” conversation and problem?

There’s a reality to it. This is a competition. These are the two leading countries — the only two countries that really matter at this point in time.

China is not that far behind, and they have outperformed all of our expectations along the way. So the idea that you just totally surrender competition — you surrender the playing field to another country that’s a geopolitical rival and that probably has less safe A.I. practices than you — that’s not a good idea to just abandon the field.

But there’s also an irony in this, especially when we’re talking about regulation of A.I.: China has had the world’s strictest, most comprehensive, most burdensome A.I. regulations on its companies for three or four years at this point in time. And it’s during this period when they had these heavy and maybe burdensome regulations that they did a lot of their catching up.

So the idea that this is just a total binary of any obligations you put on companies automatically puts you behind this totally wild, unconstrained Chinese juggernaut that is just not true. That is not based in reality.

You said two things there that can sound like they’re in conflict: One is that China’s A.I. practices are less safe than ours. The other is that China has a much more burdensome, severe, intrusive regulatory apparatus.

So tell me a bit about what they are doing that is so much stronger than what we are doing from a regulatory perspective. And then, why do you also say that they are in a less safe place than we are?

Most Chinese A.I. regulations — the early ones, especially — were really focused on online content, on information. When the C.C.P. encounters a new information technology, the first question is always: How is this going to affect our controls on information?

So when A.I. came into the picture, that’s what they looked at. They looked at recommendation algorithms, and they said: Why is everybody getting their own news feed? Why can’t we set the news agenda?

So they regulated recommendation algorithms. They looked at deepfakes — with obvious implications there. They regulated deepfakes. They looked at generative A.I., and they did the same thing.

And these do impose real costs on the companies. The companies have to do mandatory predeployment testing. They have to file their safety report cards with the main regulator in China.

It’s a real burden of time, money and effort on the companies. But most of that work, especially, say, 2022 through 2024, was really focused on securing the content environment — what we would call censorship, obviously.

From 2024 on, they’ve kind of expanded the scope a little bit, and they’ve brought in new concerns. They’ve started regulating A.I. companions, so they’re concerned about the psychological impact on kids. They’re concerned about overreliance and self-harm.

But all of this, so far, is not focused on the type of frontier A.I. safety risks that are really the focus of a lot of people in Silicon Valley — on loss of control, on bio-uplift, chemical and biological weapons, stuff like that.

That’s coming into the Chinese conversation now, but it’s coming in much later. It’s a much less mature ecosystem over there, and it really needs to get up to speed.

Something you’ll hear, at least in America sometimes, is that much of the closeness in the race comes from China in different ways — generously: building atop our models; less generously: stealing them. The key term here is “distilling.” There are ways to train a model on the answers another model gives.

So if that is true, then it’s not just like a race in which the people are tied together. So the faster America runs, the faster China is going to run.

Because it’s actually amazing — in America, too — how close a lot of the different labs are. They’re just always a month or two around each other.

How much do you buy that everybody has bunched up — because, in fact, the race is governed by the leader dragging everybody else with them?

I think it definitely plays a role, and maybe a pretty significant role. And just for the audience to visualize this, I saw a great meme of a speedboat pulling a wake surfer. — you know, the person behind, who’s essentially trailing behind the boat, going over the waves. And the people on the boat are like: They’re so close, we need to go faster. We need to go faster — because it’s pulling them along with you.

I mean, this is one of the arguments people are making. When all these A.I. labs in America are like: Well, we can’t possibly slow down because China will speed up.

Well, China is going so fast because you’re going so fast. Maybe if you slowed down, China wouldn’t be going so fast, either.

Yeah. At least in part, I think, we can say that distillation probably plays a significant role in cutting into the U.S. lead.

My mental model for it is that China is so short on compute and that distillation is probably a way for them to essentially train more efficiently, increase intelligence more efficiently, given that they have so little compute. So it’s about making up for one of their biggest shortcomings.

I don’t think that if we suddenly found a way to block all distillation, the Chinese labs would just stagnate. In A.I., in nuclear weapons, in almost every technical field over the last 30 to 40 years, China has consistently outperformed expectations. They just do things that we don’t think they should be able to do, given their level of economic development and their capabilities.

They have an amazing A.I. research ecosystem over there. One of the reasons we’re ahead is because we keep taking Chinese A.I. researchers and employing them in our labs. We are siphoning off a lot of their top talent. So they have a really thriving ecosystem of their own.

But I do think distillation plays a big role. It might be the difference between six months and a year. It might be the difference between six months and two years. We don’t know. But I think there’s pretty strong evidence that the Chinese labs are doing it, and they wouldn’t be doing it if it wasn’t to their benefit.

How does China see us on A.I.? How do they see what our goal actually is, what our goal is vis-à-vis them?

We talk about this question of whether these countries can cooperate if they need to. What is China’s perception of America’s A.I. industry?

I think their No. 1 perception is that the U.S. wants to hold China down and wants to constrain China, especially with the export controls.

Which came under Joe Biden, I should say.

Yeah. Export controls, under Biden, on these advanced chips.

China sees itself as being boxed in by this hegemon that wants to keep China in a permanent position of subservience. That’s a meta-narrative across Chinese modern history, and it’s one that has crystallized in A.I. So I think, in some ways, that’s the first thing.

Another element is they see us often as being pretty irresponsible, deregulatory — just let it all rip, let it all hang out. They see chaos within our government ——

That’s crazy because it looks so orderly from here.

[Laughs.] In the run-up to these potential A.I. talks that might be happening in the next couple of weeks, China is issuing Op-Eds by its state media, where it lays down its markers and tries to position itself in advance of the talks.

And one of the markers that they lay down is: America wants to lecture us. They want to tell us what is a safety risk and what isn’t. They want to define all this stuff unilaterally, and they don’t even impose any requirements on their own companies. So don’t come to us with that stuff unless you’re going to take care of your own house.

That doesn’t seem totally unreasonable to me.

Not totally unreasonable. Self-serving in a way, but, yes. I mean, to ——

But I think it’s interesting. This is a point others have made. To China, we look like the ones who are not regulating A.I.

There might be this whole discourse, and the countries need to cooperate. But, in fact, what they see is us racing forward, trying to attain A.I. supremacy before them. And kind of in a weird, diffuse way, calling for regulation of something that might destroy all of humanity, but we’re not actually doing any serious regulation of the thing that might destroy humanity.

When I read some of these state Op-Eds you’re talking about, the way they end up framing it is insincerity.

When I talk to people in the Chinese government, their sense of American politics is actually not often as sophisticated as I would imagine it to be. Maybe I don’t get to talk to the right people, but I think sometimes they look at us and assume that the things that are said have a more orderly structure, in the way that everybody has to use Xi’s language there.

But if you look at a thing that doesn’t make sense, and you come from their perspective, well, maybe the reason it doesn’t make sense is the counterparty is not serious. They’re just making a bunch of different moves that are all different forms of a strategy to stay ahead in the race.

As a macro-perception, I see this all the time. Talking to Chinese about the U.S. political system, talking to Americans about the Chinese political system is: If you don’t understand the system at a pretty kind of ground level — if you don’t have an intuitive feel for the two systems — the tendency is to look at the other side and to connect a bunch of dots and see a grand conspiracy. And it usually is a conspiracy against you.

The Chinese Communist Party has a very conspiratorial view of the world. They see conspiracies everywhere. And there have been times when the United States and other countries have conspired to hold down China. But the way they will connect dots that, from a U.S. perspective, are just wildly unconnected, is worrying.

And we basically do the same thing over there. We do not have the ability to see through rhetoric that’s like: That’s just what they have to say. That’s what they have to say to start, and then the real protein, the real meat of this conversation is here. That’s the real signal.

So that’s kind of a permanent issue in U.S.-China mutual perception.

From the Chinese perspective, and frankly, from the American perspective — as somebody who does have a good ground-level intuition for who is saying what in our system and why — the Biden export controls on chips were, quite explicitly, an effort to maintain A.I. supremacy for America. Which is not a crazy thing to do for a country, but if you’re the country that is being denied the exports, I think that’s a little bit provocative.

And then Donald Trump comes in, right? Orients his entire trade war against you. Dario Amodei, the leader of arguably the most important American A.I. company, in Anthropic, had this big essay, “The Adolescence of Technology.”

And he says of China:

So if I’m China, and I see the leader of the frontier A.I. lab saying that it’s an existential imperative to keep China down, I really worry about this atmosphere being the one in which these momentous technologies are potentially being developed.

Because when I talk to American policymakers, they don’t really feel like they understand what is happening in China. There’s a lot of skepticism that an agreement would necessarily be verifiable or followed.

And I think China has pretty good reason to be skeptical of what our real intentions are. Are our intentions to make A.I. safe for everyone — or are our intentions to make sure America is the first one with the A.I. that ensures American dominance of the global order for another 100 or 200 or 500 years?

That miasma of mistrust is a tough space for negotiating.

It’s a very tough space. This is arguably one of the worst times for a technology this momentous to be coming online —** **at a real moment of deep geopolitical competition between two superpowers that distrust each other, that have interests that are fundamentally in conflict in some areas.

I think one of the key points here is that, yes, trust is good. It is the lubrication that can ease things along in a negotiation. But it’s not going to be the thing that makes this work or not.

The thing that makes this work or not, in my opinion, is going to be whether both sides, for their own reasons, genuinely believe that this is a potentially catastrophic risk and believe that they need to take action on that for their own safety and security.

The Chinese technical A.I. community needs to believe deeply in its bones that if we push into this area without the right safeguards and testing and evaluation, then we run a real risk of losing control of this technology. And Xi Jinping and other Chinese leaders do not want that in their own country for their own reasons.

And so I think that is the area of mutual interest, as opposed to mutual trust, that things have to be built on.

So when I’m looking at U.S.-China interactions in this space, I think that one of the most important things, at least as a starting point, is: Can we build up a mutual understanding of the risk? Can we share information? What are we seeing about emerging risks, and how do we test for those risks? What are the best practices for securing a model that has cyber capabilities that you don’t understand? What are the best practices for defanging a model that might have bio-capabilities that you don’t want?

That work in the United States is just much more mature. The regulation in the U.S. is much less mature, but within the companies, within the labs, this has been work that they’ve been doing seriously, and investing a lot of money and a lot of people and resources in, for a long time.

And I think that’s one of the misunderstandings in China. They look, entirely, at our regulatory ecosystem, and they say: You’re not doing anything. Whereas, the labs here are voluntarily doing far more on this than the Chinese labs are doing in a mandatory regulatory environment.

So with all that work and that knowledge that we’ve gained, can we find ways to safely share some of that with China to essentially seed, bolster and help grow their existing A.I. safety ecosystem, their technical A.I. ecosystem over there, that is concerned, wants to do good work on this, but is just starting five-plus years later and just has invested far less people, money, computing resources in that work.

You have some personal experience here. You’ve hosted some of these China-U.S. dialogues on A.I. — not the official high-level U.S. government ones, but these more informal ones that are, in some ways, supposed to help lay long-term groundwork for this.

What have those felt like? What have you learned from interacting with Chinese colleagues and counterparts? Give me some of your texture on this.

These conversations are normally very technocratic, maybe a little bit boring, productive but calm affairs. And one thing I’ve seen a couple times, when you get a little bit of heat — like a little bit of spark in the conversation happens — oftentimes when the Americans are pointing at these trend lines in A.I. and bio, or these emerging safety issues scaling, they’ll say: Look at this. Why aren’t you more concerned about this? Why aren’t you doing more on safety?

And you’ll see the Chinese side getting really frustrated and being like: You guys don’t get it. We are doing a ton on safety. We have these A.I. companion regulations. We have mandatory labeling of A.I.-generated content. We have all of these rules and regulations. They’re just not the exact thing that you want.

And I think that a lot of the disconnect between the two sides is that the Chinese side feels like they have been doing serious work for a long time, and that’s just not understood, or not respected, outside of the country.

I think talking about this requires some sense of how China’s A.I. industry differs from ours. How would you describe the difference between the culture and approach of the frontier A.I. makers in China, compared with Anthropic, OpenAI, Google DeepMind here?

Among the most frontier labs, I’d say that’s where the cultural similarities are the closest. And it’s much more of the broader A.I. industry and the policy ecosystem where the differences are much wider.

I’ll start with that broader ecosystem and then bring it into the frontier.

I think, in many ways, in the U.S., a huge portion of the A.I. industry and the policy world really started from this idea that one day we will reach superintelligence. That is the goal that will bring with it catastrophic risks that will require heavy focus on safety.

And it has been this magnet that’s, like, drawing us into this future. That’s, of course, especially true at Anthropic and at OpenAI and DeepMind, but I think that’s a pretty significant portion of the policy ecosystem here, too. So it’s almost this teleological thing of we’re endlessly being drawn toward that.

And in China, there are a couple of people who lead the frontier labs who have that take. But in terms of a broad culture throughout the industry, the investors, the engineers, the policy people, the government, that has not been this magnet drawing them into the future in the same way.

It’s more like they’ve been developing an industry, developing applications of it. As they develop a new application, they develop a new policy to deal with that.

It’s a pretty fundamental difference in the way the ecosystems have kind of grown and expanded.

I’ve thought about this a lot in the American context. When American policymakers are like: Where do you start? — I sometimes say: Well, you start by starting. You learn how to regulate things, you learn how to legislate on them by regulating and legislating on them.

The Chinese approach — they are already learning, day by day, how to interface with their labs, how to craft regulations and revise them. They are building up practical regulatory experience that then, if or when they need to come in with things that are much more potent, they sort of know how to do that.

And so it’s not that their regulations are what the American frontier labs believe are needed, or what I believe are needed. They’re not. Although frankly, I would like us to be more thoughtful about A.I. companion bots than we’ve been.

But we’re actually behind. And practical experience here is meaningful. In some ways, it’s more meaningful than, like, neat conceptual arguments about the worst-case outcomes.

One of the characteristics of Chinese policymaking, but especially in A.I., is that it’s very iterative. They’ll roll out a regulation, they’ll see how it’s working, they’ll roll out a technical standard that specifies it. It’s not quite achieving the end that they want, and they’ll roll out another regulation that basically just overlaps the first one. And with each one of these, they’ve built up these reusable regulatory tools.

So the main one is this registration system for A.I. models and that the C.A.C., the Cyberspace Administration of China — the main regulator there — needs to be able to read, needs to be able to understand, in some cases maybe do the tests on their own. And when they first started this in 2021, the regulators were totally out of their depth.

The C.A.C. is traditionally an internet regulator. It’s focused on content and political content, stuff like that. But that was four years ago.

It has been focused, initially, all on this controlling content. It has now been expanded to these other areas about emotional dependency, around labeling of content: Can you impose and then remove a label on A.I.-generated content? Stuff like that.

But they have been constantly in touch with the labs for about four years. That’s a lot of regulatory practice and regulatory muscle.

Part of the question, with these frontier safety risks, is whether this is something that you can just easily tack on to. Is it just another test that they run? Or is it something significantly more complicated?

And I think it’s kind of in between the two. They have a lot of mechanisms, a lot of habits and touch points that are very good. But they do need to increase their technical capabilities in these specific areas of frontier risk and control.

Tell me about the way in which China has evolved to emphasize open-weight models versus our main models. Anthropic and OpenAI are closed weight.

And maybe begin, for people who don’t know those terms, by defining them.

Sure. When you use ChatGPT or Claude or Gemini, those are closed models, as in you interact with it on the company’s terms through their portal. You cannot edit the model. You cannot download it to your computer and run it yourself.

An open-weight model can be downloaded from the internet, and if you know how to do it, you can play with it. You can tweak it. You can remove safeguards. You can add new capabilities. You can tailor it to your own purposes.

If you need to use a model — you need to make thousands or tens of thousands of calls every single day to run your own start-up — you do not want to be paying Anthropic and OpenAI for every single one of those tokens every time you ask the model a question.

And this has been a divide that has really emerged, starting especially in 2024 or so, where it wasn’t always a given that this is how the two ecosystems would develop. But the way it has developed is that Chinese labs primarily release their models open weight, and the U.S. labs primarily release them closed weight.

So at the very beginning, in the aftermath of ChatGPT, when China was first regulating generative A.I., they actually started off taking a relatively cautious approach to open-weight models, and putting regulatory burdens on them that would have made it much harder to use open-weight models in China.

The reason they were doing that is, at the time, the leading open-weight model was Llama — from Meta, from Facebook. And China was worried: Are Chinese developers going to take in Llama, build their applications on top of it, and it’s going to kind of poison the ecosystem with their information that we don’t want?

But over the next year or so, we saw a couple of the leading Chinese labs decide to release their models open weight. DeepSeek was really the big kaboom moment in this, in that when they released it open weight, it took the world by storm. The entire global A.I. community was able to actually play with it and look at it and see that it really is that impressive.

Since then, it has kind of snowballed from there. And I think, in some ways, the C.C.P. might have stumbled into this outcome, but I think they’re pretty happy with it. And it makes sense, both for the companies, to a certain extent, and for the government.

Doesn’t it make it harder to control these models? This is a debate in the American A.I. ecosystem, where Dario Amodei and Sam Altman often fight with Mark Zuckerberg over this.

There’s a view that when you get these very powerful models, something like Mythos, which has these incredible cyber-hacking implications, you don’t want anybody to be able to just download Mythos and do what they want with it. I mean, these are potent things. You need to have some control over them.

The C.C.P. has a more aggressive regulatory stance and is more control obsessed than the U.S. government tends to be — and yet, China is the center of the open-weight ecosystem. How do those things hold together?

I think one factor is: What was needed for the companies to be seen as globally competitive? I think if DeepSeek, in late 2024, had just announced to the world: Hey, we’ve got a great model, and feel free to use it. It will go to Chinese servers, and we’ll give you back the answers — I think there would have been a level of suspicion about that.

I don’t think it would have seen this rapid global proliferation because people have a certain distrust of Chinese technology. And by releasing it open weight, they can essentially say: Hey, you look at it, you change it, you do whatever you want to it. It’s that good, and you will see that, and then we’ll figure out how to make money in other ways.

So I think that’s part of the business aspect to this. It’s hugely reputation enhancing for Chinese companies, and now for China’s A.I. ecosystem as a whole, to release these open-weight models and therefore overcome some of the suspicion that normally falls on Chinese companies when they go global.

So that’s one part of it. Another part is that, frankly, these are probably undermining the future valuation of Anthropic and OpenAI.

I don’t think that this was a scheme going back to 2023 or 2024, when this world we’re in wasn’t totally foreseeable. But now that they’re here, I think it says: Well, that is to our benefit in terms of long-term competitiveness.

Do the Chinese models give extremely different answers, or come up with very different approaches, than the American models? Is a more fundamentally different worldview detectable if you run testing across the two? More skepticism of democracy, generally?

I think American models very much do have an American outlook on the world. Do you see the models as being very different when Americans are talking to DeepSeek?

It depends a little bit on how you’re using the model. Like, the most censored version of a Chinese model will be when you’re using it through the app, or you’re using it through the A.P.I., when you’re going to deepseek.com and asking it questions.

That’s the version that has the most controls built into it. If you download the model, an open-weight model, you download it, you run it on your own computer, you will have a different set of safeguards, not entirely removed, but a different set of them, and you can also tweak those. You can remove some of those. You can add new training data. You can change the way that the model functions.

And some American companies, like Cursor and others, feel that they can get the models into a place where they are not propaganda machines for the C.C.P.

There are a lot of countries — Singapore, Southeast Asian countries — that are building sovereign A.I. models on top of these open-weight models. Say, add your own language data, add your own cultural data to post-train these models in a way, and tweak them to your needs.

So that’s part of China’s pitch to the world, to the Global South: America is the technological hegemon that wants to restrict your access to this technology. It wants to impose its own values. It wants to blot out your own local culture, and it won’t let you in any way adapt or play with their models.

We’re just giving you the model, and you can do with it what you want. You can change it, you can adapt it, and you can run it for just the cost of the cloud computing that you’re using.

That’s the pitch. I don’t think it’s 100 percent honest. I don’t think it’s actually going to play out in that exact way. But that’s the divide that China has been trying to pitch to the rest of the world.

I find this really interesting. The way the internet developed, and a lot of the modern mega-online platforms developed, China and the U.S. have pretty separate digital ecosystems. I mean, you’re not using a lot of Google search in China. We’re not using WeChat here.

We are much more integrated on A.I. than we are on what came before. But a pretty large number of American companies, which are consuming A.I. tokens at a level where you actually have to pay real money to keep going, they’re using Chinese models. Airbnb and Coinbase are famously using Chinese models for significant parts of their A.I. infrastructure.

So this is not just the Chinese ecosystem over there and the American ecosystem over here. They’re already somewhat combined.

I don’t know how much OpenAI or Claude are allowed in China, because I assume they’re not censoring in the way the C.C.P. would want them to. But the Chinese models are here and in widespread commercial use.

Yeah, this is really one of the great ironies of this current A.I. moment that we’re in. The firewall really came down and kicked out the American technology companies — Google, Facebook, Twitter, etc.

In 2008 to 2010, we had very separate product ecosystems. They were building their own products. We were building our own products. The products didn’t really cross over.

But we always actually had pretty integrated technology ecosystems. You had a huge flow of Chinese people coming to the U.S. and working in companies. A lot of them would go back and cross-pollinate the two ecosystems with ideas.

We had a lot of American money going into Chinese start-ups, a lot of Chinese money going into American start-ups. It was all quite integrated, outside of the product layer, up until about 2017, 2018.

That’s when we began the American project of technology decoupling with China. We want to pull apart these connections because we think this is how China is catching up. It’s catching up because they’re stealing. It’s catching up because they’re learning at our universities, etc., etc.

And so, the first Trump administration — and to a certain extent the Biden administration — did a lot to cut down the flows of people, to cut off the flows of money, to reduce the flow of ideas between the two ecosystems.

And that was relatively successful. I think it probably would have continued to be quite segmented in this way, except for the fact that the Chinese model is going open weight. And so it’s almost like the open-weight ecosystem has reintegrated these ecosystems in a way that I don’t think anybody could have foreseen three, five, 10 years ago.

Xi Jinping recently gave a pretty big speech on A.I. What seemed new to you in that speech?

So this speech was at the World A.I. Conference, which is China’s premier A.I. event every year. They try to get the whole world to come out.

It’s a big to-do, and this is the first year that Xi Jinping has attended in person and given a speech there. So it’s really his biggest A.I. speech, maybe ever. So people were watching very closely.

I think a good portion of it was China’s pitch to the rest of the world. It’s the one I outlined earlier.

And then the other part that stuck out to me was the conclusion. It ended with some pretty striking metaphors using an ancient Chinese idiom — that I don’t have off the top of my head — about how the wise adapt to circumstances, and they do not get stuck on one path.

And I think one of the key terms was that we need to be able to forestall loss of control of A.I. There’s a long-term concern in the West: Does A.I. get out of our human control? It’s a long-term concern in China, but one that has taken a bunch of different forms.

What do they mean? Are they talking about party control? Are they talking about the control of an operator? Or are they talking about human control over A.I.?

And so he put down a marker there, around loss of control, and I read it as leaving this space open. These are all signals to people in the system. When he says “forestall loss of control,” that means that A.I. researchers all throughout China, when they’re applying for their next grant, they’re going to use that term.

If you use a term that was in a big Xi speech, you’re just more likely to get grant funding. Like these things are markers that everyone, the policymakers, are interpreting.

They’re trying to figure out: How can I do that in my area? Researchers looking for funding are adopting it. Companies are looking for signals about what will and won’t be in bounds. So the words really matter, and I think that conclusion was at least putting down some markers that are showing that China is shifting pretty quickly on a couple of these fronts.

My model of this is that political pressure, political possibility, doesn’t build linearly — and it particularly will not on A.I. That what happens is you have issues, they stagnate, they are not at the front of the agenda, and then something happens, and the window of possibility blows open.

So I think in America here, the OpenAI-Hugging Face hacks — almost more consequentially, the fact that OpenAI systems hacked OpenAI and took over part of their research clusters; the kind of social engineering and effort to upload malicious code from frontier Anthropic models; the swarm behavior, the peer behavior — this summer of weird A.I. incidents has blown this open a bit in America. And now, all of a sudden, we’re talking about pacing the frontier.

I guess the question is: China knows these things are happening. So how are they responding to these same events that are transforming our conversation?

So you’re right that they’re taking it in. There’s tons of coverage, in Chinese state media, about the Hugging Face incident, about pretty much all the major safety developments.

Recently, there was an Anthropic researcher who resigned and issued these pretty dire warnings. That was in state media today. I was reading that.

And so they take it all in. They are much more attuned to our conversation than we are to theirs.

I think part of it is that they react more incrementally than we do. And that’s, in part, I think, due to this long-term different relationship to, say, superintelligence and catastrophic risk.

For a lot of Americans who have been thinking about this for a decade — they’ve been predicting this will happen, and then this happened. It is the ultimate illustration that they were right all along, and it’s happening.

For the Chinese side, this is just newer. They’re taking it on board, and they take the data point. It’s like: OK, that’s interesting. It hacked out of a system. Was this an issue with the safeguards? With the tooling? Could this have been constrained with pretty mundane security measures — or is this a sign of something bigger?

And it’s been really interesting to watch over the last year — really a year and a half, at this point — as a lot of this safety terminology has worked its way into important Chinese government documents and important technical standards, documents by their lead regulator.

I think in early September, China’s main A.I. regulator, the Cyberspace Administration of China, issued a public statement on its top-five A.I. risks. And No. 2 on that list included what they call “extreme loss of control.” That’s the first time that I’ve seen that specific phrase — extreme loss of control.

They’ll talk about controllability. In the past, when they talked about that, it was more party-state controllability.

But around 2021, and then really in 2023, they started talking about human control over A.I. And now it’s, essentially, working its way into more and more practical and specific A.I. policy and technical documents.

So they’re taking it on board. I think they are moving in the right direction on a lot of this. And to me, the big, open question is: Do they move fast enough?

There was another incident that made headlines here but has not been greeted as such a big deal — even though, to me, it was quite scary. So frontier A.I. models were able to find a vulnerability in WeChat, which — maybe you can describe for an American audience the centrality of WeChat to the Chinese digital ecosystem. And they were able to build this attack on it that they dubbed WeWorm, and it would have given control of somebody’s phone just by calling that phone.

Now this was then conveyed to Tencent, the developers of WeChat, and according to them, the vulnerability has been patched. But it seemed like a hell of an example to China that the hacking capabilities here are at the point where they could compromise major foundational Chinese digital infrastructure.

How has that been covered?

I haven’t seen that much coverage of it in mainstream media. And that might be because when an American company finds a huge vulnerability in the central digital platform in China, it’s not really seen as in everybody’s interest to publicize that a ton. So that might be part of it.

I think another part of it is that this is a question of offensive hacking capabilities. And I think, for them, the real wake-up moment for that came with Mythos — when the U.S. develops a system that they’re not releasing to the public, that they’re only releasing to a set number of companies and also the N.S.A. And China has to assume at that point that it is being deployed far and wide against Chinese systems.

So a lot of the discussion in the aftermath of that was: How do we harden our own system against these types of cyberattacks? In some sense, this type of cyberwarfare between the two countries is inevitable and long-term, and it’s almost like we shouldn’t take it too personally.

China shouldn’t take it too personally when we hack them in a bunch of ways. We shouldn’t take it too personally when they do. That’s kind of our job and their job. It’s the question of, when something happens that ——

A lot caught up in there. I’m not going to question it, just putting a pin on it.

Yeah, a lot. It’s the job of the N.S.A. and it is the job of the Ministry of State Security to try to hack each other. There should be limits, critical infrastructure, all that kind of stuff.

But in some ways, I think that’s baked into both countries’ worldview, that we’re both going to be using it intentionally against each other. The issue is, when it’s something that’s not being done intentionally by a state, when it’s happening by a nonstate actor that neither of us wants these tools in the hands of, when it’s out of control and neither of us has the ability to sort of understand or control it — those are the areas where I would expect some level of overlap. And that’s a newer phenomenon that China is grappling with.

What about recursive self-improvement? Sometimes we talk about loss of control like it is a passive thing. I don’t intend to lose my keys, but I do it all the time. Loss of keys.

Recursive self-improvement is handing control over to A.I.s, right? Recursive self-improvement is where the A.I. systems autonomously build the next system — that they are now moving faster at improvements than human beings can possibly keep up with. We are dependent on the A.I. system to tell us what it is doing. We are dependent on those descriptions of what is happening being correct.

We have seen A.I.s exhibiting deceptive behavior. We see the frontier lab saying that our capacity to monitor is already degrading. We are seeing A.I. labs that are currently racing toward recursive self-improvement, expressing very high levels of concern about what it will mean to achieve this thing that they are desperately trying to achieve.

It is a very strange situation. And then, of course, when you say maybe you shouldn’t do this, you get: But China!

Xi also keeps talking about how A.I. should be developed by humanity and should be under humanity’s control. Recursive self-improvement is the simplest way to give up human control of A.I.

To me, that’s a place where some international standards seem really needed. And not like we can wait five years on that because, I think, the American A.I. labs think they’re going to hit R.S.I. in the next 18 months or so.

Is there the possibility of cooperation on this or constructive dialogue on this — or is something that is outside a crisis point not even plausible within the conversation?

So R.S.I. is somewhat newer in China. I listen to a lot of Chinese tech podcasts, and they just started talking about R.S.I. this summer, like mid-late summer. Whereas, I think this has been in the conversation in Silicon Valley for much longer than that.

Now they say: Wow, this is the next thing! This is where things are going. Because in many ways, as creative and innovative as the Chinese ecosystem is, they still do a lot of times look to Silicon Valley for these types of directional shifts: What is the next paradigm?

And so, with so many of the U.S. labs beating the drum on R.S.I. and saying that this is where it’s going, I think the Chinese labs are kind of following into that space. I don’t think they have as much experience with it. I don’t think they’ve done as much technical work with it, or maybe even thought as much about the risks of it.

I do think that this is one of those places where we might just have to draw a line. And whether it is done bilaterally, at the exact same time, or whether it is something done unilaterally, with the expectation or intense negotiation to try to get China to agree to the same limitation, that might be the point. This is a core tenet.

Would we be more likely to get them to agree to it if we drew that line unilaterally?

If we do it unilaterally, it increases the chances that China does it. It also increases some risks that you do it unilaterally, and then China catches up or forges ahead. That’s a double-edged sword, and I won’t pretend that it’s like the solve-all for us to do it unilaterally.

But a lot of this is a matter of sending costly signals. You talked about all of the misinterpretation and conspiratorial thinking between the two sides. And so, when we just say a bunch of stuff about the dangers of R.S.I., and we talk about it, but we don’t actually have any regulations, we don’t do anything about it, we are not sending any costly signals.

And our lead in this technology allows us to have access to information, to see threats and to see risks that they just haven’t seen yet.

They’re not going to trust everything that we say. They’re not going to trust all the information that we share. But that’s a card that we can play in these areas.

And whether it’s a unilateral pause or it’s an information-sharing mechanism that we set up now — something where we share information on incidents like the Hugging Face incident — if the U.S. and China are going to sit down and talk about A.I. in the coming weeks, that’s a great opportunity to put on the table a lot of information that’s not sensitive. In the sense that it doesn’t undermine the U.S. lead but lays out, very clearly and in deeply technical terms: This is what we saw, and this is why we’re worried about it. Do with that what you want, but this is what we saw.

I think that’s the space that we want to be working in, and then — other than just sharing information on the risks — we want to be trying to plant seeds or enhance the technical A.I. safety capabilities within China. They really need to catch up. The practices there are just much further behind the leading U.S. labs.

The capabilities aren’t that far behind, but the safety practices are further behind. And so it’s in our interest — loss of control, if something goes out of control in China, it doesn’t stop at the borders there — for them to have good safety practices, and they have a lot of catching up to do.

So it seems we’re on the cusp of there being talks. They would be led, on the U.S. side, by Treasury Secretary Scott Bessent. He’s got a Chinese counterpart.

There’s also going to be the Trump and Xi Jinping meetings coming up.

You’ve been pouring a little bit of cold water for people on what to expect out of these. But what, to you, is a constructive outcome here — the sort of beginning of a space in which possibilities could emerge? And what, to you, would be a negative signal about what’s possible?

A negative signal would be a statement that sounds good, and gestures at something that nobody has a problem with. So if the two sides get together, and they say: Look, we both care about A.I., and we both care about child safety, and so we each affirm our commitment to child safety and A.I. It’s like: Sure, yeah, that is an important issue, but that is not the key issue between the two countries. So I think that would be a sign that we didn’t really have traction yet, at least.

What would be positive to me is, one, establish this as an ongoing, recurring dialogue that will have staff, that will sort of build over time. Maybe the U.S.-China strategic A.I. dialogue that meets every four months. In the past, we’ve established these on security issues, on economic issues, and you need to have a real structure in place where it’s not just a one-off.

The next thing I’d like to see — maybe two things: One would be a working group between the leading technical A.I. safety people within the U.S. government and the leading technical A.I. safety people within the Chinese system.

In the U.S., there’s a lot of bureaucratic fighting over this, but the Center for A.I. Standards and Innovation, the CAISI, is really, I think, the center of knowledge when it comes to testing frontier models. China has a new working group, called Working Group 9, which is essentially tasked with developing technical standards related to A.I. safety, broadly defined, but also starting to look at catastrophic risks.

I think something that creates a space where those two teams can safely talk to each other and exchange best practices — say: This is what we’re seeing, this is what we’re worried about, this is how we test for it, and this is how we mitigate it.

Maybe the second thing would be a crisis communication line — a good way for the U.S. and China to get in touch if something emerges rapidly that is an A.I.-driven crisis, that could get spun even further out of control because of U.S.-China dynamics.

So the Hugging Face incident. Somewhat luckily, OpenAI hacked Hugging Face. They were able to more or less get in touch with each other and sort it out, and it wasn’t a big deal.

And the hack was somewhat untangled by Hugging Face using a Chinese open-weight model, which the Chinese state media quite enjoyed.

Absolutely. Imagine just a couple different twists on that. What if that is a DeepSeek model that’s hacking Hugging Face? Or what if it’s an OpenAI model that, for whatever reason, decides it really needs to acquire more compute resources?

And, oh, I can find this, you know, insecure compute cluster that happens to be in Zhejiang Province in China. What happens if it takes over a compute cluster there? How is China going to read that signal? What’s their response going to be?

Or even, if you take it out of just a purely bilateral context, if we start to get information, intelligence, that a swarm of A.I. agents are draining bank accounts in Pakistan, and we don’t know what their intentions are, we cannot read their communications, and we cannot shut it down right away — we need to be in touch with the other leading A.I. superpower on that issue.

So a way that these two countries can get in touch, share information in a crisis situation. It’s a very fraught issue. We have had a lot of these crisis communication lines on military issues, and the U.S. complaint is always: Oh, the Chinese side doesn’t pick up the phone when we call them.

And that’s a real issue. I think one mitigation to that is — somewhat ironic — not to use a phone but to use a fax machine.

Literal faxes?

Literal faxes. It has a logic to it, too. Because the political system there is not a system of empowered individuals. It’s a system of committees and a system of documents.

So when our treasury secretary — someone who feels very empowered on the U.S. side — picks up the phone and is like: Give me some answers, He Lifeng — or other Chinese counterpart — they’re kind of like: Eh, not really ready to give you answers on the fly.

Much better to send a document over to their system that they can review, they can bring it to their committee, they can come up with their understanding and response and send something back.

So something in that vein, that at least puts a little bit of a safety net on these incidents that — I think something like that is pretty likely to happen in the next year.

Everything you’re saying here makes sense to me — about the fax-not-phone dynamic and documents. But man, when the whole thing we’re facing down is the acceleration of A.I. incidents and things happening at faster than human speeds, and now we’re dealing with governments that work at, frankly, slower than human speeds, it really creates quite a mismatch.

The big language right now is “pacing the frontier,” but we don’t even have a plan to keep the frontier from accelerating. Like, forget pacing it at the moment. We are currently accelerating the frontier. It’s concerning.

It’s deeply concerning. And if it’s a matter of a race in decision making between an agent and a person, the agent is going to win that race. We, hopefully, won’t be engaged in that very specific race.

And I think the Chinese system, it’s interesting because, in some ways, it can be so slow and incremental, and then it can be so fast. Their response to Covid-19 was initially so halting. It was screwed up by information gaps where the local officials don’t want to report the bad news to the higher officials.

It has all these kinds of neuroses and idiosyncrasies. But when they decide, like: We need to shut down this city, we need to wall this city and not let anybody in or out — that happens pretty fast.

And I’m not saying that’s the solution on A.I. That had a ton of human costs. When China takes action like that, it always has a ton of human costs.

But each side has its strengths and weaknesses in this area, and I think there is a chance that, while China is moving pretty incrementally now, as the evidence builds, I think there is a chance they will shift gears pretty quickly.

What is the relational context, between the leadership, that these conversations are coming into?

Trump rose in politics with a very skeptical, to say the least, take on China. In his second term, after the beginning tariffs on Liberation Day, they tried to pivot toward a trade war with China. China fought back. We functionally backed down. And since then, for all the bluster you sometimes hear, Trump seems to be trying to build a better direct relationship with Xi.

So to what degree is the current status of the U.S.-China relationship — and particularly the Trump-Xi relationship — maybe more flexible than one might assume, just knowing where it was at the beginning of Trump’s second term?

Yeah, Trump does both extremes when it comes to China. He really changed the direction of American policy in a much more hawkish direction, taking seriously harmful, aggressive actions against China.

At the same time, he seems to personally really like Xi Jinping. He seems to admire him. He seems to see a kindred spirit, in some way, in these two strong leaders of countries.

And I think that affects a lot of U.S. policymaking. There’s a lot of evidence that different, potentially aggressive actions against China have been watered down, because Trump doesn’t want to screw things up ahead of the meeting.

During the Biden administration, I think a fair number of people were sort of thinking ahead to saying: Maybe we do need to be engaging China on A.I. safety. Maybe we do need to be sharing information.

But they felt very constrained by the idea that, well, if Democrats do that — if the Biden administration does that — we’re going to get roasted as soft on China, and we’re going to be seen as giving away the store on A.I.

Trump just creates his own political gravity, his own political environment, where that same action will be read in a very different way — that he doesn’t have to share the same concerns as past administrations.

So I think that’s a dynamic. On the Chinese side, I think Xi is a much more systematic thinker and much less reliant on these individual relationships and seeing this as a structural long-term contest between two systems, between two countries.

And the day-to-day wavering of: We love China, we hate China, we’re blockading, we want to have double the investment — I don’t think he sees that as a meaningful change in the overall trajectory between the two countries.

And so while I think the one-to-one relationship — you know: Does Xi like Trump? Not all that relevant. But the changes in the Overton window of what we think is possible, when it comes to engagement, I think that is meaningful.

You’re a pretty calm-seeming person, temperamentally. If you’re talking honestly to Chinese counterparts who are regulating but not on the most profound set of risks — or American counterparts who are worrying but not actually doing all that much — what’s your real level of alarm?

What would you tell them about the moment we’re actually in? Not what you think is possible, not what you think is likely to happen in the bilateral talks, but if everybody was where you were, the way they would see this issue right now?

I think that the moment, this period of time, is terrifying. We should be terrified on a certain level.

But I’ve been working in A.I. policy, one way or another, since about 2017, and I’ve been hearing these warnings since then. And I’ve always tried to maintain some type of neutrality on how real these risks are.

I’m like: These scientists say this, these scientists say that. I’m not the one to adjudicate this. I’m not going to be the one who solves it, so I’m just going to try to sort of keep both these things in mind and work forward from there.

But the evidence is mounting. The evidence is growing that the people who have been making some of the most dire warnings for the longest time — they have probably been right, at least about a lot of things. And the warnings that they are issuing are increasingly dire and increasingly on short timelines.

For someone who has been looking at this for a while, and has tried to maintain a position of not panicking and neutrality, it’s very worrying.

I think that’s the place to end. Always our final question: What are three books you’d recommend to the audience?

I’ll do two books on China and one fun one.

The first China book is “Country Driving” by Peter Hessler, the New Yorker staff writer. In a lot of ways, for people of my generation who went over there and lived there, he’s kind of like the godfather. He’s the guy who inspired me to become a journalist, to just get out into the country, meet people, get such incredible, beautiful portraits of Chinese society at the micro level that I think we’re missing. We’re missing that in so much of policy today.

And I hope young people today will start going back over there and getting in the mix. We need that textured understanding. So that’s one.

Another one, a little bit more obscure, is called “From the Soil: The Foundations of Chinese Society.” It’s a book by a Chinese sociologist in the 1930s and 1940s, a guy named Fei Xiaotong, who was trained in the West, went back to China and applied Western sociological paradigms to studying Chinese villages and agriculture. It’s just one of the most insightful books about Chinese culture, so I’d encourage people to seek that one out. I read it every three or four years.

And the last one, just for fun, is Zadie Smith’s “On Beauty.” You had Zadie on the show. I think she’s the GOAT. I think she’s the best.

“On Beauty” is just a hilarious novel of an academic family, and her ability to pierce into the psychology and the insecurities of each of us, and put that on blast in a way, is just — I don’t know. It just brings me a lot of joy. So “On Beauty.”

Matt Sheehan, thank you very much.

Thanks for having me.

You can listen to this conversation by following “The Ezra Klein Show” on the NYTimes app, Apple, Spotify, Amazon Music, YouTube, iHeartRadio or wherever you get your podcasts. View a list of book recommendations from our guests here.