The Price of Consciousness

On borrowed intelligence, the capacities we lose, and the new senses we may gain

I gave a version of this talk in New York last week. The recording is unfortunately nearly inaudible, so rather than publish a transcript, I reconstructed the argument here alongside the slides—and some of what happened in the room afterward.

Alan Watts wrote, in The Wisdom of Insecurity, that “there is a price to be paid for every increase in consciousness.”

I have been thinking about that sentence a lot, because right now we have a fairly extraordinary increase in consciousness. We can ask questions we couldn’t ask before. We can move across fields we don’t yet understand. We can perceive patterns across enormous amounts of information. We can reach forms of intelligence that do not actually live inside us.

The most interesting word in that sentence is price. A price implies that we are getting something.

The obvious question is whether AI is making us stupider or smarter. But I think the better question to ask is what is the price of this increase in consciousness, and what does it buy us?

A year ago, I called one part of that price the fragility of borrowed intelligence.

I was interested in what happens when more and more of our capability becomes something we can access without having to reproduce it independently. I called one version economic fragility: capability becomes external, so we stop maintaining the human expertise that produced it. The other was epistemic fragility: answers become external, so we stop maintaining the judgment required to evaluate them.

I still think both are valid. But I now think I underweighted the other side of the trade.

Not every lost capacity is a loss.

Human beings have been externalizing capacity for the whole of civilization. One way to describe technological progress is as the movement of things we once had to carry inside us into the environment outside us, and that movement is exactly what makes us more capable.

I don’t know if civilization would be improved if we still had to navigate by the stars, memorize everything we read, or do long division by hand. My favorite example is glasses, because it is the most literal one. We built infrastructure around a biological limitation, and now poor eyesight no longer determines whether you survive.

Civilization is quite literally built on borrowed intelligence. Intelligence was infrastructure long before artificial intelligence.

What is different this time?

For a long time, what we reduced was the physical capability we had to carry in our own bodies. Machines replaced muscle and infrastructure protected us from the outside environment. Specialization meant none of us had to know how to make everything we used. The result was that human economic value moved, steadily, into cognition. We became less physically self-sufficient and increasingly specialised in the one thing we thought was only ours: thinking.

That is why this transition feels so different. The capacities we migrated toward are now becoming infrastructural themselves: writing, coding, synthesis, research.

While these capabilities allowed us to become capable in the first place, there is another way to interpret what is happening.

Alice Albrecht recently proposed that working with AI may cause us to grow new senses. Her argument is that AI systems perceive and organize information in ways very unlike ours, but for now we force most of that intelligence back through language: we ask a question, and the machine throws words at us.

What if language is only a primitive interface? What if AI eventually allows us to perceive patterns that are currently inaccessible to us altogether?

She gives a lovely example of navigation: instead of looking at a map, you might simply feel a directional tug in your body toward where you need to go. The larger point is that AI may replace old capabilities while also expanding the range of things a human can perceive.

So perhaps externalizing intelligence does not simply make the individual less capable. Perhaps the relationship between the person and the tool produces capabilities neither has alone.

I may only be able to perceive some pattern because a borrowed intelligence is coupled to me. At that point, it may become less useful to ask whether the intelligence is “mine” or the machine’s. The better question may be: what does the boundary cost?

This is the trade as I currently see it. I am definitely not saying that any of this is set in stone. I am asking us to hold the tension in the trade-off instead of collapsing it. After all, consciousness emerges as a dynamic state of tension.

“This, then, is the human problem: there is a price to be paid for every increase in consciousness. We cannot be more sensitive to pleasure without being more sensitive to pain. By remembering the past we can plan for the future. But the ability to plan for pleasure is offset by the “ability” to dread pain and fear the unknown.”

Which forms of human development are worth preserving even when they are no longer economically necessary?

A live test case

Two hours before I gave this talk, OpenAI announced that an internal system had produced a proposed solution to the Navier-Stokes problem, one of the Clay Millennium Prize problems.

The system reportedly used roughly 10,000 concurrent agents, 2.7 million agent messages, 130 million output tokens, and 88 hours of wall-clock time before arriving at the result. What had been produced at that point was only a formalization; the prize requires publication and a waiting period.

For roughly ninety years, humans had been unable to answer a fundamental question about fluids. Now a company was saying that its internal model had found a proposed answer. No individual mathematician could reproduce the process by which the result was generated. Humans could inspect the proof at the end.

So: what are we losing, and what are we gaining, with this?

Is this fragility, or a new sense?

One reaction to the announcement called it an extraordinary win for human–AI collaboration. My friend Benjamin Parry asked: have we already decided the AI gets to determine what is true?

This is exactly the tension. If we can reach mathematics humans could not reach on their own, that expands our range. But no individual human can reproduce the search process that produced the result. We can inspect the proof. But if the machine produced the proof and another machine ultimately becomes necessary to verify it, where does human judgment sit?

And then we argued about it

At the end of the talk, we split the room around a motion:

The human capacities we lose to AI are a price worth paying for the new capacities that we gain with AI.

What is the trade-off?

AGAINST : This is a problem that stood for ninety years with no solution. What we’ve done with AI is essentially brute-force a solution. But not every problem is susceptible to brute force. If you try to solve all problems that way, you lose the capacity for creativity, insight, and intuition that the other kinds of problems need.

FOR : An open problem for ninety years was done in eighty-eight hours. In a year this may be a twenty-dollar subscription and you can go solve an open math problem.

FROM THE ROOM: So you’re saying the trade is worth it.

FOR: In this one case.

Who gets to keep the mistakes?

AGAINST: One thing this machine cannot replicate is our capacity to make mistakes, which is where a lot of creativity stems from.

FOR: Why do we have to give up our mistakes? You can keep them if you want. Nobody’s forcing you.

AGAINST: In theory. But it gets complex when corporations mandate AI use, especially in creativity. That’s where it stops being a personal choice. The pace quickens and more is expected in less time.

“You can keep them if you want” is the answer most of us give ourselves. The losses look optional until the tool is mandated, the pace changes, and the capacity you would have kept by choosing friction is no longer yours to keep.

What should I do?

One of the most interesting arguments from the room was about the shift from asking AI “how do I do something?: to asking “what should I do?”

The first outsources method. The second begins to outsource judgment.

AGAINST: Most of us in this room have already made the shift to asking AI: what should I do? Not how do I do something — what should I do? That normative frame, externalizing our own decision-making, is the first step of the erosion.

What happens when friction disappears?

There is an optimistic version of this future in which we hand the scut work to machines and use the space that opens up to paint, dance, look at the stars, and become more human.

The reply from the room was less confident:

Humans have a remarkable capacity, in the face of friction, to create friction, and created friction is often far more damaging than the naturally occurring kind. The idea that we’ll shed meaningless work and fill the space with what’s true to humanity — we’ve demonstrated as a society, time and again, that’s not how it works.

When we really tried to decide which capacities were worth losing for the ones we might gain, the room did not arrive at a clean answer. The “no” side, arguing for the trade not being worth it technically won. They’d argued that some of the capacities we lose are too foundational to trade away.

There is a price to be paid for every increase in consciousness. Which prices should we be willing to pay?

---

With thanks to Tusk&Quill for publishing the original essay and organizing this event.