When AI knows better, who gets the last word?

Following an assistant’s advice can make life easier. Brendan McCord asks what happens when choosing for ourselves becomes a habit we no longer practice.

At the start of 2025, a viral meme introduced us to the Claude Boy: a teenager who outsources every decision to Claude, from the moment he wakes up to the moment he goes to bed. He asks Claude what to eat or how to reply to friends and then carries out Claude’s advice in deadpan obedience. The story was made up, but the meme spread because it felt plausible. Accounts followed up by posting things like “Claude-willing” or “on Claude,” and some people began acting it out, still half in jest, by asking Claude what to do and doing what it said.

Having been a teenage boy once, I get it. At 16, I did not feel especially qualified to make wise decisions, and I am confident Claude would have handled many social situations better than I did. And it is hard to say, precisely, what is wrong with the Claude Boy’s behavior. Anyone handing their decisions to an AI system chooses to do so freely and can stop at any time. They are not being coerced or deceived. If they are making better decisions and their life feels better for it, what actually is the objection?

Someone in a leadership role at a frontier lab once told me that, as these systems get better, refusing to act on their advice makes that person “a rebel and a fool.”

The view that we should defer to AI rests on two serious arguments. The first is that what matters morally is consequences, and we are responsible for bringing about the best outcomes we can. If following AI’s advice reliably produces better outcomes than following your own judgment, then choosing for yourself merely because the judgment is yours is vanity. This is especially irresponsible when the decision impacts other people’s outcomes.

The Humanist Review of AI, Issues 01 and 02

Conversations at the frontier, in your inbox

Subscribe

The second argument is that deferring to AI can help you realize, or approximate, your best self. The legal scholar Cass Sunstein describes “choice engines” as tools that take information about our preferences and circumstances and use those to help us make better decisions. Such a system could nudge us toward the choices we might have made if we had better information, fewer biases or a full night’s sleep. In doing so, AI would help us live up to our own considered purposes.

Now imagine that kind of assistance made abundant and extended across an entire life by systems that reliably judge better than we do. There is no Tuesday on which you are the better judge. The first argument says AI knows better than you. The second says it knows the better you. Followed to the limit, either one makes Claude Boys of us all.

Choices form choosers, and so do the choices we hand off.

So far, the case for deference has turned on the quality of our decisions. But those choices also change who we are and the way we make choices in the future. Choices form choosers, and so do the choices we hand off. If we run the Claude Boy forward 10 years, who will be choosing then?

Aristotle observed that we become builders by building and brave by doing brave acts. Our actions produce both an outcome in the world and a capacity in the person who does them.

In an article published in The Atlantic, the mathematician Terence Tao likens AI tools to “taking a helicopter to drop you off at the site,” with the result that “you miss all the benefits of the journey itself.” We can produce output without going through the skill development that came along with undertaking the journey to get there. A finished result can still teach, just as a proof in a textbook is learning material for a student if they work through it, but the journey becomes optional, and optional practice is easy to skip. Earlier technologies made the journey optional for particular skills, such as arithmetic and navigation. AI could do the same for judgment itself.

We often welcome technologies that spare us mental effort. As Alfred North Whitehead put it, “Civilization advances by extending the number of important operations which we can perform without thinking about them.” In practice, we already offload a great deal of our judgments to experts and organizations. If the edifice of civilization was built by freeing people up to do more worthwhile activities, why stop at the judgments we make about our own lives? If AI is the better judge however much we practice, then the case for practice must rest on what judging makes of us.

A calculator costs me opportunities to practice my arithmetic, but when I use one, I already know what problem I am trying to solve. A hard decision about how to act is different, given that I may not yet know what kind of problem I am facing. Imagine our Claude Boy has a friend who cheated on a test, and half the class did the same. When the friend asks him to stay quiet, he consults his AI assistant, which recommends first giving the friend a chance to come clean and then, if he doesn’t, telling the teacher.

The assistant’s advice may be sensible. But its response frames the problem as one of disclosure. Another frame might start from loyalty, asking what he owes his friend and how that is in tension with what he owes his other classmates who did their own work. It could look at whether his reluctance comes from his allegiance to his friend or a fear of confrontation. Working through these tensions helps him get to a better understanding, even if he ends up following the original recommendation. Simply accepting the answer gives him a fraction of that understanding.

A single deferral of this kind might be fine, and so might the periodic, bounded deference a student gives a teacher. Teachers have always framed problems for their students, and a good teacher frames them in order to leave students better able to frame problems themselves. An assistant can be built to serve that aim, but it is usually measured instead by whether it completed the task and satisfied the user. Neither shows whether the user’s judgment grew. Teachers are also scarce, with finite amounts of time and expertise. Even the most overbearing one covers a limited number of hours and subjects, while an assistant is always there, instantly responsive, even before you’ve fully formed your own questions. When it answers first, day after day, deference may become like second nature.

Our Claude Boy can stop any time and reject any piece of advice. He keeps the right to question the next answer, or to think through the problem for himself, but he may have long lost the habit of doing so.

He may be much more effective by virtue of his approach, obtaining better test scores and having smoother conversations. This is a gain in agency, meaning the power to get what you want and act effectively in the world. But bit by bit, one micro-abdication at a time, he may be ceding his autonomy, or the ability to decide what is worth wanting and doing in the first place. Someone can become better at getting what they want while exercising less of their ability to self-rule. As the systems improve, the pull toward a life of high agency and low autonomy could strengthen. The products of judgment could become abundant while the occasions that once required us to think for ourselves grow scarce.

An alternative to the Claude Boy is someone I like to call the “Free Agent”, someone who can take the help of powerful AI while retaining the ability to judge what goals or directions that help should serve.

The temptation for deference will grow more powerful in tandem with AI systems, but receiving the benefits of good judgment does not make us capable of exercising it. It must instead be developed through deliberation, action and reflection that cannot be done by anyone but us. Its exercise is partly constitutive of the good life—living well requires choosing for ourselves what it means to live well—regardless of whether we’re making the best decision available to us.

John Stuart Mill observed in “On Liberty” that his contemporaries, even in their private choices, did not ask “What do I prefer?” They asked what was “usually done by persons of my station and pecuniary circumstances.” Following custom, even when it leads to good decisions, gives people little practice in judging for themselves. People who never follow their own nature, he writes, end up with “no nature to follow.” Mill acknowledges that such a person “might be guided in some good path, and kept out of harm’s way,” but he asks what that individual’s “comparative worth as a human being” would be. “It really is of importance,” he writes, “not only what men do, but also what manner of men they are that do it.” For Mill, human nature is “not a machine to be built after a model, and set to do exactly the work prescribed for it, but a tree, which requires to grow and develop itself on all sides, according to the tendency of the inward forces which make it a living thing.”

The same concern applies to Sunstein’s choice engine. It can serve your goals, but if the engine increasingly just decides for you, it can leave you less able to challenge its suggestions, no matter its intentions. Knowing what you would choose with better information and a clearer head is one achievement. Becoming someone who can understand and revise that choice is another.

Moving beyond the individual case, a nation could enjoy greater abundance while its people became less practiced in governing themselves.

We have never governed ourselves unaided, and choosing whom to trust can itself be an act of self-rule. But what would it mean for a whole country to defer its judgment freely to superintelligent machines? Moving beyond the individual case, a nation could enjoy greater abundance while its people became less practiced in governing themselves.

Alexis de Tocqueville, who studied the young American republic in the 1830s, feared in “Democracy in America” that democratic peoples would willingly raise a “tutelary power” above themselves. It would resemble a parent, except that parents prepare children for adulthood, whereas this power would keep them in a “perpetual childhood.” It would provide for their needs and pleasures until it spared them, in his words, “all the care of thinking and all the trouble of living.” They would remain busy, but with less occasion to exercise their own will, like “a flock of timid and industrious animals.” He was writing about the state, but the argument holds for anything we may freely defer to in return for convenience.

For Tocqueville, self-government was a practiced habit of mind first and a constitutional order second. American town meetings, he wrote, “are to liberty what primary schools are to science.” In them, and through voluntary associations with each other, citizens learned self-government through managing shared affairs with differing priorities. AI could bring more people into those affairs, helping a resident understand a zoning dispute well enough to argue their case at a town meeting. It could also handle the dispute for them, drafting the comment and brokering the compromise so that no one has to sit across from a neighbor who disagrees. Either way, the dispute gets settled, but only the first leaves the town with citizens who have practiced settling one. Choices form choosers, and choosers form republics.

Before there even was an American republic, Benjamin Franklin set about forming its choosers. In Philadelphia, books were scarce and few people had access to material to ignite their thinking. As a first step, members of Franklin’s Junto, a club devoted to self-improvement, pooled the books they owned. And later Franklin helped establish America’s first successful subscription library. He credited such libraries with making “the common tradesmen and farmers as intelligent as most gentlemen from other countries” and with contributing to the colonies’ stand in defense of their liberties.

Franklin built places to exercise judgment because books were scarce, but we now need them because answers are abundant.

Franklin built places to exercise judgment because books were scarce, but we now need them because answers are abundant.

AI can provide such an environment. It should help strengthen our judgment while sparing us needless effort. The philosopher Philipp Koralus argues that reasoning is driven by two forces: a strong pull to answer a question as directly as possible and a weaker pull to test the answer against further questions. An assistant that immediately answers any question, no matter how half-formed, addresses the former pull, but not the latter. A better assistant would leave room for the weaker pull, raising a question when one could change our minds without making us question everything, which would end in paralysis.

Suppose that when the teenager brings the exam problem to his assistant, it asks why he is torn instead of simply telling him what to do. He says that reporting his friend would be unfair when half the class cheated. When asked what would be fair, he proposes that he and his friend approach the teacher together about what happened across the class. The assistant can help him test his arguments against the pushback he might get from his classmates, including whether it is fair to the students who did their own work.

Together they can examine the proposal, asking whether he is using the widespread cheating to excuse his friend or has found a fairer way to address it. He may still end up following the original recommendation, but he can now understand why he is following it. After that, the assistant can help him prepare for the conversation.

His judgment and reasoning need to be able to withstand whatever happens once he finishes the conversation with the assistant. For example, what happens if his friend disputes his account of what happened? What if he challenges the idea that the proposal is fair? A good assistant should leave him ready to hear these objections, weigh them dispassionately and, if he needs to, adjust his original judgment. The exchange with the assistant shouldn’t be a rehearsal for a conversation with a foregone conclusion. The ideal relationship is one in which we are both better informed and better practiced in deciding what to do.

A good assistant should not make us reconsider every question every time we use it, since such a product would quickly become unusable. Instead, it should work from judgments we have already made to form a set of working assumptions. Koralus calls an environment arranged to help us reach answers that will help us generally withstand further questions an “erotetic home” (“erotetic” meaning “concerned with questions”). If you stock your fridge with food that matches your diet, you don’t have to think from first principles every time you make your lunch. But it’s important that these defaults remain provisional and open to change.

An assistant should be transparent about what it remembers about our past preferences and judgments and allow us to revise them when circumstances, evidence or our minds change. The convenience of past judgments shouldn’t become invisible constraints on future decisions. While showing us what it remembers doesn’t guarantee that we’ll notice when something has changed, it gives us occasion to exercise the capacity the Claude Boy risks losing.

A consequentialist might reply that a good assistant should promote human development as one of its goals. Mill himself believed it was important. But then, to judge whether an assistant is succeeding in this goal, we can’t simply judge the quality of the immediate decision. Instead, we have to examine what happens to the chooser over time.

A research study could compare forms of assistance that produce similarly good immediate decisions, following people from the start of sustained use for a year or more. Some tests would present advice that is sound but framed too narrowly and ask whether people noticed that it was incomplete. Another test could ask people to state their priorities at intervals and explain whether, and why, the assistant’s advice changed them. Since people should follow sound advice, the evidence would come less from how often they disagreed with the assistant and more from how they directed the assistance and what they could still do on their own.

Most AI evaluations still judge an assistant by the quality of its answers and the satisfaction of its users rather than what is happening to the person. The specifications guiding model behavior should make the development of human judgment an explicit aim and test how sustained model use affects it.

Ten years from now, AI may be vastly more capable than it is today. Who will we have become alongside it? A good assistant is one that helps the Claude Boy grow up.

Brendan McCord

Brendan McCord is the founder and chair of Cosmos Institute, which supports research and technology development focused on human autonomy and a free society. He previously served as founding CEO of two AI startups and authored the U.S. Department of Defense’s first AI strategy. A graduate of MIT and Harvard Business School, he was a visiting fellow at St Catherine’s College, Oxford.

English (United States)
Your Privacy Choices Opt-Out Icon Your Privacy Choices
Consumer Health Privacy Sitemap Contact Microsoft Privacy Manage cookies Terms of use Trademarks Safety & eco Recycling About our ads