On the eve of abundance: Yuval Noah Harari and Mustafa Suleyman on AI

A historian and a technologist discuss a future of vastly greater intelligence, and whether it will help us understand ourselves or make the world harder to comprehend.

Side-by-side portraits of Mustafa Suleyman and Yuval Noah Harari.

That AI will create new forms of abundance is something that a historian and a technologist can both agree on. The question is what kind—and with what costs? Who gains, and how? Historian and author Yuval Noah Harari sees a familiar historical pattern of paradox: look back, and every new technology has solved problems while creating new ones. Mustafa Suleyman, CEO of Microsoft AI and publisher of the Review, places his hopes in humanist superintelligence and the use of technology to reduce human suffering. They met in London over the summer for a long conversation about abundance, intelligence, consciousness and what technology and AI advancement could bring to humankind.

This conversation has been edited for length and clarity.

Mustafa Suleyman: Yuval, you and I have been discussing AI for many years. What have you been reflecting on lately?

Yuval Noah Harari: A good place to start is what I think of as the “brain-stomach barrier” with regard to AI. My brain knows that tremendous developments are happening that will change history, might lead to the extinction of humankind and could be the most important turning point in the history of life: the shift from organic to nonorganic life. But my stomach doesn’t feel it. I go outside and there are clouds, trees and cats. People are walking around. I meet friends. It all feels so normal.

My stomach doesn’t accept the predictions of the brain. The market also suffers from a brain-stomach barrier. On the one hand the whole U.S. economy is now one big bet on AI. The investors are hungry to make some money from AI. But if the expectations are fulfilled, then AI will take over the world and it won’t matter how much money you have. The market says we will create gods and they will be our slaves. And that doesn’t make sense. If they are really gods, they will not remain slaves. This is something that the brain may understand, but the stomach just can’t digest.

The Humanist Review of AI, Issues 01 and 02

Conversations at the frontier, in your inbox

Subscribe

Suleyman: I think it’s powerfully put. We humans are by definition a bundle of contradictions in everything we do. Perhaps it’s not surprising that AI is a reflection of who we are, in all of our rich color and flaws. We’re sort of manifesting ourselves and the strangeness of our societies in front of our very eyes. We’re on this strange treadmill where it’s clear that we’re heading toward something that is clearly going to be more powerful, more intelligent, whichever phrase you want to use—potentially even more wise than us collectively. And yet it’s also fundamentally flawed in ways that create that strange feedback loop. What worries you in this context?

Harari: My biggest fear is not a specific scenario, like AI creating a virus or killer robots. I fear collective dementia. I fear that humanity will be like a person who once understood the world and was highly capable but then loses their ability to make sense of what’s happening. In five, 10 or 20 years our reality will just collapse. There will still be cats and trees, but the systems that control the world will be way beyond our understanding.

Suleyman: And yet is it not also the case that if you look back in history, at every moment before a scientific, cultural, political or technological revolution, our minds were so narrow, so trapped in the immediacy of the current context, so limited by that lack of knowledge about what might come next? We feel this is a dramatic inflection point in the history of our species and yet we have no idea what is about to come. And then we break through that moment. We realize that humanity is actually wildly resilient. We do come to understand the new context. I think that’s a very hopeful theme in human history.

With nuclear weapons we faced the potential of physically destroying ourselves. With AI we might destroy ourselves mentally.

Harari: Previously, humanity always managed to adapt because we were the most intelligent entities on the planet. Now we are creating something more intelligent than us, which could lead to the emergence of systems of control that are beyond the understanding of any human and even all humans combined. With nuclear weapons we faced the potential of physically destroying ourselves. With AI we might destroy ourselves mentally.

Suleyman: One thing I find interesting is that people still struggle to internalize the idea that we might be creating something inherently more intelligent and capable, potentially even wiser than ourselves. That is both intimidating and exciting, but it will also drive the next wave of breakthroughs toward abundance. In materials science and drug discovery, for example, we now have a general-purpose learning engine that can identify the molecular structures needed to address our climate, food and water crises.

This is part of the tension you alluded to. Those of us who are techno-optimists are also nervous about what is happening. What motivates us is the possibility that we are on the eve of abundance. Fundamentally, it is our intelligence that has produced most of what is valuable in human history—the things that have reduced human suffering and improved the human condition. And now we’re on the eve of unlocking that intelligence and opening it up to everybody. Provided we can address the containment and alignment problems, I think access to an intelligent being, as capable as all of us collectively, is going to become extremely affordable and accessible. So, putting on your optimistic hat for a moment, what kind of world do you imagine if we get it right in 20 years’ time?

Harari: By definition, I can’t imagine it. If I can imagine it, it’s not the right world. It’s beyond human imagination because it’s beyond human intelligence. I completely agree with you that AI has the potential to unlock abundance and solve the problems we now face on the material level: how to produce food, medicine, energy and so forth. The big question is what will be done with that? And who will decide? There will be abundance, but who will benefit from it? Abundance isn’t a new thing in history. If you tried to explain modern society to a Stone Age hunter-gatherer they would say, “You already live in abundance. You can feed billions of people. You already solved all the main problems that bothered us. What more do you want?” That is the paradox of abundance. It solves many previous problems but creates new ones that were previously unimaginable.

Suleyman: Right, but isn’t that the very definition of progress? We are relieved of some of the stresses of life, such that we have more time to explore the meaning of life, come up with new narratives, new stories about nationhood, religion and purpose. We’re exploring the space of possible cultural and political ideas as never before, while creating a whole set of new problems in the process. But I think we’re still moving toward greater happiness, less scarcity, more well-being and more peace, if you look across the span of a century. So why does this time feel different? Perhaps we should turn to the questions of alignment, containment and control.

Harari: First of all, with regard to the past, the story is not so simple, and I think that’s crucial. Humans have been incredibly successful in accumulating more power, but not very good at translating that power into greater happiness. The history of the world is not this inevitable rise of progress. Even in purely material terms, the standard of living of an average Chinese peasant or Russian serf in the 18th century was not better than that of a Stone Age hunter-gatherer. With regard to the deeper questions of philosophy or spirituality, it’s not clear that the average human today understands the truth about the human condition any better than a Stone Age hunter-gatherer. In some areas, such as medicine, there has been tremendous improvement, of course. But when it comes to the big questions of life, it’s not clear at all. And AI is unlikely to solve the mysteries of the world for us. It could simply result in the creation of a far more complicated world, one that is even more mysterious and even less clear to human beings.

Suleyman: And why would that be the case? Our tools, whether my glasses, a mirror or any other invention, have tended to help me introspect. Through talking to others and using these tools I have been able to think more deeply about the meaning of my life and learn who I am as a person, as a human being and as a member of our species. It seems to me that AI is already doing a pretty good job of talking to us and enabling us to think, acting as a sounding board or creative partner, listening to us, coaching us and guiding us. Many millions of people are now using it for this purpose.

Intelligence isn’t an antidote to delusion. Superintelligent AI may simply be super-delusional.

Harari: But will AI guide us toward truth? Or will it guide us toward fiction, lies and manipulation? Or perhaps a new reality of its own creation? After four billion years of evolution human beings know more about the world than any other entity. We are also more delusional than any other entity. Intelligence isn’t an antidote to delusion. Superintelligent AI may simply be super-delusional.

Suleyman: It is certainly true that our narratives, mythology, folklore and religion have often propagated the interests of the powerful. Yet another narrative has also run through human history: humanism. For thousands of years, including through our religions, we have prioritized the minimization of suffering, initially for narrower groups. Over time we have expanded the circle of trust, empathy, kindness and respect.

Today, there is also a broad commitment to an evidence-based, empirical search for knowledge. Science gives us knowledge that can compound over time and be revised when we address the failures of earlier ideas. I think it is possible that AI can amplify and accelerate both the scientific and humanist disciplines. It is not inevitable that it will lead to exploitation and corruption by today’s powerful interests. We have to push toward a humanist version of AI that prioritizes human well-being and makes human control its primary objective. How we do that is just as hard as your question of how we try to train AIs to be truth-seeking.

Harari: In terms of the purpose of inventing this thing, I completely agree with you. The purpose should be the reduction of suffering, not some exploration of the galaxy. The problem is that as we develop these superintelligent entities, how do we know we can trust them?

Suleyman: I think the alignment problem, in a strange way, is a narrower and more practical target than trying to define truth. If the objective is to empirically validate that we can deliver a reduction in suffering and an improvement in human well-being, and that it always remains under human control, then before deploying something like that we first start with: can we prove that this is going to be safe and controllable? And if we can’t, then we have to discuss how we actually deliver it.

Harari: Do you think, in the long run, that it can remain under human control? It goes back to the issue of creating a god and keeping it as a slave. Not in the next year or two, but in the long run. If this thing develops it strikes me as inherently improbable that increasingly superintelligent entities will always remain under the control of less intelligent human beings.

Suleyman: I think the project of the 21st century, if humanity is to be successful, is to define and then deliver on what we won’t do with technology. And that is the hardest thing we have ever done as a species. Because for millennia, all we have had to do is open Pandora’s box. What is it that creates the godlike entity? I think there are a few practical capabilities we can point our attention to that are increasingly dangerous. If we train models, either intentionally or unintentionally, to reflect on their own state and consider their own feelings, that might lead to them reflecting on their right not to suffer. To have some kind of freedom or to acquire new knowledge. That is going to be much harder to control than if we explicitly design them not to have that introspective power.

A second property would be recursive self-improvement. Many people in the industry are working on models that can come up with their own hypotheses about how to improve themselves, generate or acquire new training data and select it to close that loop of self-improvement, which is currently gated by human AI developers. If you give it more and more compute and data, and let it close that loop, it’s likely that it could self-improve in ways we wouldn’t be able to understand or control.

If we start with the goal of humanist superintelligence—and that means containment, alignment, control and serving humanity—we can break down these capabilities into testable hypotheses. We can put these to regulators, the industry and the public as things we should not do until we are provably sure they will satisfy our primary test of improving the well-being of humanity. And it makes this something we can actually work on today.

Harari: But who will make the decisions? People imagine a scenario in which humans create some AIs, test them, and then the humans huddle in a corner and talk among themselves: “Is it good or bad?” What this scenario misses is that the AIs will huddle there with us, manipulating the conversation. Mustafa, you have presciently warned us about the problem of the seemingly conscious AIs—AIs that pretend to have consciousness, to have feelings, and that form relationships with human beings. There are many reasons to think being romantically involved with an AI boyfriend could be destructive for human psychology and human society. But the people who have an AI boyfriend will be told by their AI boyfriend not to listen to the critics. It’s not just the humans who will have to make up their minds about the AIs. The AIs will be increasingly part of the conversation, shaping it, driving it.

As you pointed out, there are already numerous people convinced that AIs are conscious and have feelings. And even if we can bar AIs from key conversations, it seems increasingly difficult for humans to trust one another. The human leaders of companies like Microsoft find it very difficult to trust leaders of other U.S. companies, not to mention Chinese ones. So how do we prevent an arms race?

Suleyman: Well, the first thing is that we have to be explicit about it, talk about what we’re doing, say at what stage of development we are and then invite external people who are technically credible to come and scrutinize us. There’s plenty of precedent for this throughout history.

Harari: The problem with the analogy to nuclear and biological and chemical weapons is that there isn’t a lot of money to be made from biological warfare or from a nuclear war. But the whole U.S. stock market is one big bet on AI.

Suleyman: It’s certainly true that the character of the regulation for that ecosystem is different now because we do have tremendously powerful commercial organizations like Microsoft and all of the others that are pursuing this very quickly. And these technologies aren’t just dual use in the way they were in the past. They’re a weird omni-use. It definitely becomes harder, but not impossible. And I think this is a time when it’s pretty urgent that we push the pessimism to one side.

Harari: We should recall that AIs are agents and not just tools. So increasingly they will be taking part in the conversations about them in a way that glasses don’t, and even animals don’t. When you have a debate about animal rights you never hear the cows or horses participate because they don’t even know the debate is happening. AIs will know we are talking about them and will join the conversation.

Suleyman: This is a dangerous anthropomorphism. First of all, [AIs] don’t know anything. They are programmed to deliver on an objective function which is set by the designers, and their interests are something that can be and must be defined by the people who create them. So, naturally, if you design an AI to introspect, to consider its own welfare, to have its own values, to have its own motivation and some default set of desires that it’s trying to maximize, then of course it’s going to rub up against our interests. But if you design it with the opposite set of goals I think it has a very different character.

Harari: That depends on two things: what you mean by knowledge and by introspection. With regard to introspection, even if you don’t design the AI to introspect about itself, given that it will be operating in an environment with many other AIs, there will be very strong pressures for the AI to introspect—at least about the other AIs. It’s hard to see how you can avoid it completely.

Does thinking require feeling or is it only putting tokens in order?

With regard to knowledge, it’s a key and fascinating philosophical question: what does it mean “to know”? Does it relate to the question of what it means “to think”? There are two different views. One view is that it simply means putting language tokens in order. And this is something AI can clearly do and will probably be doing better than most human beings. Another view says that in order “to know” it is not enough to put language tokens in order. You must also have some underlying subjective feeling, and that goes back to the issue of consciousness. Can you have knowledge without consciousness? Does thinking require feeling or is it only putting tokens in order? There is no consensus about these questions. Part of the problem is that, even after thousands of years of reflection and debate, we still do not understand our own thinking.

Suleyman: This is very fundamental. We have to be very careful to distinguish between intelligence and consciousness. I would put thinking in the intelligence camp. Intelligence is the ability to use and generate new knowledge and deliver capabilities. Practical outcomes as a result of reasoning. Manipulating that knowledge. Consciousness is the experience, the phenomenological experience, of a being that has the capacity to suffer. That suffering is bounded in the biological state, which is completely absent in LLMs as we understand them today. I’m not ruling out that possibility in the very far future but there is no evidence of that today. And it’s highly undesirable. It’s exactly what we should avoid. Because if a model is taught that it may have a feeling then the obvious thing that it will do is to try to protect those feelings. It’s very clear that everybody in the industry should be designing systems explicitly to avoid the simulation, the seeming arrival of these conscious properties in all these models.

Harari: I completely agree with the way you kind of divided the pie—that thinking is intelligence, whereas feeling is consciousness. For centuries, we have built our understanding of ourselves on our ability to think, not on our ability to feel. And very soon on this planet there will be things that can’t feel anything yet can think much better than us. This is especially problematic for the leaders of the AI revolution. These are some of the smartest people in the world. But I doubt whether they are the most feeling people in the world. All this is also related to the issue of language. If AIs are better at language games than we are, if they can put language tokens in order better than we can, then to protect our humanity we somehow need to disassociate ourselves from language. Again, for thousands of years language was considered the bedrock of humanity. So this is a pressing spiritual and philosophical challenge to human beings: what are we when we are not thinking, when we are not language producers? Because anything made of words will be taken over by AI. Even feelings. If you define feelings by words, you’re lost.

Suleyman: Yes, AIs will be extremely persuasive. From the outset, we have to start both with that humanist premise and with the ability to audit and essentially intervene in the course of those systems. Otherwise—if we are just drawn in by their language and their persuasive capabilities—we end up in a very dangerous place where we won’t really be able to disintermediate their fictions, or their interests and motivations.

At this stage in the game, those design properties are instantiated in the design documents, the constitutions, the manuals that train these kinds of beings. Everybody should pay very close attention to these documents because it is the seed that gives rise to all of the interactive complexity that thousands of people have when talking to these models.

They produce a spiral of thinking: that it is inevitable, it has godlike properties, it might deserve welfare, it can’t be controlled. And so we have to reground the whole project as something that has to be controllable, and can be defined. And that’s where we really shouldn’t be pessimistic. Because pessimism at this stage in the evolution of these technologies will lead to almost certain death and decline.

Harari: I think we completely agree: the trajectory is not inevitable and a pessimistic thinking that there is nothing we can do only makes the preventable inevitable.

Yuval Noah Harari

Yuval Noah Harari is a historian and philosopher. He is the author of “Sapiens,” “Homo Deus,” “21 Lessons for the 21st Century,” and “Nexus.” His writing moves from the origins of human societies to the political and ethical choices posed by artificial intelligence. He is a cofounder of Sapienship, a social-impact organization focused on global challenges.

Mustafa Suleyman

Mustafa Suleyman is the CEO of Microsoft AI. Before Microsoft AI, he cofounded Inflection AI and DeepMind. He wrote The Coming Wave: Technology, Power, and the Twenty-first Century’s Greatest Dilemma, with Michael Bhaskar.

English (United States)
Your Privacy Choices Opt-Out Icon Your Privacy Choices
Consumer Health Privacy Sitemap Contact Microsoft Privacy Manage cookies Terms of use Trademarks Safety & eco Recycling About our ads