TL;DR
What is worth wanting is not an intelligence utopia but a society that can keep improving — and intelligence alone cannot supply what that requires.
- Technological abundance and worsening distribution can happen at once. Productivity answers how much value a society creates; distribution answers who gets it. Ownership, competition, public policy and organisational structure sit between them.
- Intelligence is not the only factor of production. If a step must wait on a physical experiment that cannot be accelerated, reasoning a hundred times faster will not compress the cycle a hundredfold. But do not conclude that returns to intelligence necessarily diminish — stronger intelligence may invent its way around the bottleneck.
- AI cannot substitute for political legitimacy. Predictive accuracy and normative justification are different things, and governance needs legitimate procedure as well as correct outcomes. Handing decisions to an AI does not dissolve the question of power; it moves power to whoever designs and controls the AI.
- Science fiction is valuable not as prediction but for changing a world's basic rules so you can see the assumptions you stopped noticing. An institution that works in a story is not thereby workable in reality.
- A future civilisation has to solve four problems at once: cognition, coordination, power and distribution. They are linked but not interchangeable — stronger intelligence will not fix bad incentives, and better procedure will not substitute for accurate facts.
1. The reasonable part of technological optimism
In October 2024 Anthropic's CEO Dario Amodei published Machines of Loving Grace, taking up a question the risk narrative often obscures: assuming powerful AI is kept under reasonable control, how much value could it actually create?
The essay focuses on five areas: biology and physical health, neuroscience and mental health, economic development and poverty, peace and governance, and work and meaning. What they share is that each directly shapes quality of life and has long been constrained by knowledge, research capacity and social resources.
Take medicine. We have accumulated enormous knowledge about disease, proteins, genes and human systems, but the path from identifying a candidate target to completing validation and clinical trials remains long. If AI materially improves hypothesis generation, experimental design, molecular simulation and data analysis, parts of that path can be accelerated.
That is a reasonable direction.
But a scientific discovery is not a medical outcome. A new drug still has to pass safety and efficacy testing; manufacturing, regulation, reimbursement and clinical adoption do not complete themselves because a model reasons faster.
So growth in intelligence does not mean all social progress grows at the same rate.
2. Intelligence is not the only factor of production
Economics offers the useful frame: real production depends on complementary factors — not only knowledge and compute, but capital, energy, physical equipment, experimental resources, organisational capacity and the institutional environment.
If a production step must wait on a physical experiment that cannot be accelerated, then even a hundredfold gain in theoretical reasoning may not compress the development cycle a hundredfold. That is a bottleneck constraint.
A simplified model: final output depends jointly on intelligence, data and experimental conditions, physical resources and institutional execution. When one factor becomes the binding constraint, adding more of the others yields less.
And immediately the counter-argument: real factors are not in fixed proportion. Stronger intelligence may relieve a bottleneck by inventing new methods, improving organisation or restructuring the experimental process. So it is wrong to assert that returns to intelligence must diminish.
The question worth studying is: in which technical fields can intelligence replace a bottleneck, in which can it only improve one, and which remain bound by physical or social conditions that do not move easily.
That analysis is worth far more than a blanket forecast that "AI will make every industry ten times more efficient".
3. Abundance and fairness are not connected by necessity
Suppose some future AI delivers high-quality education at nearly zero cost. Technically, an enormous advance.
But the actual distribution of educational opportunity still depends on internet access, language support, educational institutions, family circumstances, and whether society recognises the resulting credentials.
Likewise, if AI greatly improves financial research and investment decisions, it does not follow that every investor earns more. Prices reflect participants' expectations — as more accurate information becomes widely used, some of the informational edge that existed disappears.
Capability can diffuse while the resulting economic gains do not distribute evenly. Returns may flow disproportionately to whoever holds the scarce capital, data, platform access, energy or other complementary resources.
An important distinction:
| Question | What it answers |
|---|---|
| Productivity growth | How much value society can create |
| Distributive institutions | How much of it each party receives |
A society can experience a technology boom and worsening income distribution at the same time. That does not prove technical progress is useless — it shows that ownership, market competition, public policy and organisational structure sit between capability and welfare.
4. Institutions have to be tested against reality too
If Machines of Loving Grace mainly asks what powerful AI could create, an equally important question is: through what institutions should a future society organise those capabilities?
One point deserves particular attention: an institution must not only let different parties coordinate; it must itself be testable against real outcomes.
A governance system that aggregates public opinion well but cannot tell whether a policy achieved what it intended still has a serious defect.
This connects directly to epistemology:
- Individuals revise beliefs against evidence
- Organisations revise decisions against outcomes
- Institutions likewise need to revise rules against feedback from reality
An institution that cannot learn will repeat its errors however much compute it has.
5. Why AI cannot substitute for political legitimacy
Suppose a superintelligence can predict economic growth, public health and social welfare. It can simulate tax policies, forecast the efficiency of infrastructure, even propose better allocations of public resources.
Should we simply hand government decisions to it?
First, predictive accuracy and normative justification are different. A policy may raise aggregate output while imposing disproportionate losses on some groups; another may reduce short-term efficiency while protecting long-run fairness, social stability or basic rights. These are choices about the ordering of values, which no predictive model settles.
Second, governance needs legitimate procedure as well as correct outcomes. Citizens may demand to be informed, to appeal, to participate and to supervise, even where an expert system is statistically more accurate than human decision-makers.
Third, any AI governance system still has to answer the question of power:
- Who sets the model's objective?
- Who supplies the training data?
- Who can change the system's parameters?
- Who can review its predictions?
- When the public disagrees with the model, who decides?
Handing decisions to an AI does not make these questions disappear. It may simply move political power to the institutions that design, train and control the system.
So there is a real institutional boundary between AI assisting governance and AI replacing it.
6. Why fiction belongs in technical thinking
Yudkowsky's Harry Potter and the Methods of Rationality brings scientific reasoning into a world with magical rules, and the thought experiment it offers is: on entering an unfamiliar environment, how do you come to understand its mechanics through observation, experiment and inference?
Scott Alexander's UNSONG imagines a world where language, religious mysticism and physical regularity connect strangely. When rules once treated as mysterious become discoverable, usable and industrialisable, how does the relationship between knowledge, power and institutions change?
These novels are not predictions and are not empirical evidence about future technology. Their value is in changing a world's basic rules so a reader re-examines assumptions they had stopped noticing in their own.
Economists use simplified models to study incentives; computer scientists use simulated environments to study algorithms. Fiction can run a wider thought experiment: alter the technical conditions, the cognitive capacity and the institutional constraints, and watch what a society might become.
But literary thought experiments have limits. An institution that functions in a story is not thereby workable in reality; a technology that has some capability in a novel is not thereby engineerable.
Only when a thought experiment becomes an explicit hypothesis, a formal model and a testable claim does it become part of serious technical research.
7. Four problems for a future civilisation
Taken together, a future civilisation has at least four problems to solve simultaneously:
| Class | The core question |
|---|---|
| Cognition | How do we tell whether information is true? How do we measure the quality of predictions? How do humans and AI keep correcting themselves? |
| Coordination | When parties pursue their own interests, how do we design cooperation that avoids an equilibrium bad for everyone? |
| Power | When agents execute independently, how do we bound their permissions so behaviour stays supervisable, auditable and revocable? |
| Distribution | As the cost of producing intelligence falls, how do markets, organisations and public institutions turn that into broad welfare? |
These are linked and not interchangeable:
- Stronger intelligence will not automatically fix bad incentives
- Better governance procedure does not substitute for accurate judgment about facts
- Reliable permission control reduces specific execution risk but cannot by itself decide which social goals are worth pursuing
- Rising welfare needs technical capability and institutions that convert capability into real improvements in life
So designing a future is not a single technical objective. It needs a whole set of cognitive and institutional infrastructure that constrains and checks itself.
8. Not an intelligence utopia
Discussion of the future tends to collapse into two poles: that technology will eventually solve everything, or that increasingly powerful technology must bring uncontrollable danger.
Reality is likely more complicated than either. Powerful AI can create enormous value and new systemic risk at the same time. What decides the long-run outcome is not only model capability but how we set objectives, allocate power, establish accountability and correct errors through feedback.
An AI society worth wanting is not merely the one with the strongest models, the fastest automation and the highest output. It is one that lets people question technical decisions, lets different institutions be compared, lets errors be found and fixed, and lets individuals keep real choices as the technology advances.
Civilisation's achievements are not only more powerful tools. They include gradually learning how to limit power, organise cooperation, accumulate knowledge, and change ourselves when we are wrong.
Superintelligence may accelerate that process enormously. It cannot decide for us which future is worth having.
The future will not be created by intelligence alone. It will be shaped by intelligence, values, institutions and the choices people keep making.
Fourth in a four-part series on the problems of the AI era, on institutions and distribution. The others: cognition, coordination, power.