14 min read
Civilization's Hidden Dependence on Bounded Optimization
Civilization contains a surprising amount of slack.
People trust strangers. Companies offer generous refund policies. Websites expose interfaces that could be abused. Friends speak without considering every possible strategic interpretation of what they say. Small mistakes are forgiven. Privacy often exists because nobody has the time or motivation to investigate you closely. Rules are written with the expectation that people will mostly interpret them as intended.
Many of these arrangements are not robust against arbitrarily determined optimization.
They work partly because humans are limited.
Finding an exploit takes time. Understanding a complicated institution takes expertise. Coordinating many actions is difficult. Keeping track of thousands of weak signals is exhausting. Small loopholes are not worth pursuing. Most people get bored. Even unusually malicious people fail to notice most of the opportunities available to them.
We normally think of these limitations as defects. Greater intelligence lets us solve problems that were previously too difficult.
But some of our limitations may also be load-bearing.
Call this protective incompetence: valuable equilibria that remain viable partly because exploiting them requires more cognition, persistence, coordination, memory, or attention than humans usually supply.
This is closely related to ideas like security mindset, optimization pressure, and slack. Strong optimization searches for failures that ordinary use never encounters, while slack gives systems room for experimentation, mistakes, and freedom from binding constraints. The additional point I want to isolate is about what happens after the optimizer arrives.
My claim is not that AI necessarily causes more successful attacks. It is that cheap machine optimization can make previously unnecessary defenses necessary, and the equilibrium defense may itself destroy valuable slack.
The central question is therefore not just:
Who wins after attackers and defenders both get better AI?
It is:
What kind of society must the defender construct in order to win?
Imagine an online service with a loophole worth $3.
Finding it requires forty minutes of careful reading, understanding an obscure interaction between two policies, filling in several forms, and taking some risk that it will not work.
Almost nobody bothers.
The loophole persists for years.
What protected the company?
Not robust design. The exploit was simply below the human exploitation threshold.
Now imagine that an AI agent can discover the loophole automatically, execute it for almost no marginal cost, and search thousands of similar services for variants.
The institution has not changed.
The exploit has not changed.
The economics have changed.
A slightly richer toy model captures what matters.
Suppose discovering an exploit costs . Executing it once costs . Each successful use produces value , and there are available instances.
Exploitation is worthwhile when the total obtainable value exceeds the discovery and execution costs:
AI can change several terms simultaneously.
It can lower by making search and reasoning cheaper. It can lower through automation. And it can increase the reachable enormously by applying the same strategy across many cases.
This is why apparently trivial vulnerabilities can become important under automation. A human may not spend an hour discovering how to extract $3. A machine may spend the equivalent of a few cents discovering a method that can then be executed one million times.
This pattern already appears in mundane forms.
Spam became economically significant because sending another message became almost free, forcing email providers to build increasingly sophisticated filtering systems. Automated bots cause websites to impose CAPTCHAs, rate limits, account verification, and anti-abuse systems on everyone. Generous promotional or refund policies are often tightened once automated or systematic abuse makes them expensive.
None of these examples requires advanced AI. They merely show that the mechanism exists:
lower exploitation costs → more exploitation → defensive adaptation → costs imposed on benign users.
More capable AI could extend the mechanism into domains where the relevant search is currently too cognitively difficult rather than merely too labor-intensive.
There is another asymmetry.
For many systems, the average user barely matters.
What matters is the strongest motivated attacker.
Suppose almost everyone uses advanced AI harmlessly. They study, write, program, organize their lives, and automate boring work.
A tiny minority instead spends months asking machine systems to search for obscure, high-leverage strategies.
For some systems, one sufficiently capable actor is enough.
A cryptographic system is not secure because 99.999 percent of users cannot break it. A catastrophic software vulnerability is not harmless because only one person discovers it. A dangerous technique does not become safe because almost nobody is motivated to pursue it.
The relevant part of the distribution is often the adversarial tail.
AI could make this tail much more important without changing human motivations at all.
Today, a determined malicious person may lack technical expertise. Someone technically skilled may lack time. Someone persistent may lack specialized knowledge.
Machine intelligence lets people borrow some of these missing capabilities.
The most motivated actors can borrow cognition.
This means that widespread access to the same AI does not necessarily produce symmetric consequences. Differences in motivation, patience, willingness to violate norms, and appetite for unusual strategies may matter more once competence becomes cheap.
The effect also need not require superintelligence in the dramatic sense. Millions of persistent, moderately capable agents can create optimization pressure very different from that produced by a small number of human specialists.
None of this establishes that offense wins.
Defenders also get better AI.
Perhaps that dominates everything I have said so far. Fraud detection improves. Software becomes easier to secure. Institutions discover vulnerabilities before attackers do. Powerful systems allow much finer-grained authorization. Better models distinguish benign eccentricity from actual abuse. Privacy-preserving technologies improve enough that institutions can verify what matters without learning everything else.
A high-intelligence world could conceivably support more freedom than ours because the defense becomes less crude.
So the important distinction is not simply offense versus defense.
It is between two ways defense can improve.
An invariant defense changes the structure of the system so that a broad class of attacks no longer works.
Good cryptography has this flavor. You do not separately prohibit every possible password guess. You design the system so that the relevant attack class is infeasible.
Memory-safe programming can eliminate broad categories of memory-corruption vulnerabilities. Least-privilege systems can ensure that compromising one component does not grant arbitrary authority.
The defense compresses. One structural improvement replaces a long list of patches.
Sometimes no sufficiently cheap invariant is available.
The defender instead reduces the set of actions people are allowed to take.
An activity is usually harmless but has a rare dangerous use. The institution cannot cheaply distinguish benign from malicious cases, so it restricts the whole category.
This is defensive contraction.
As optimization pressure rises, institutions therefore face a race:
Can we discover sufficiently general defenses, or must we keep removing degrees of freedom?
I do not know which side wins in general. It probably varies dramatically by domain.
But the distinction determines much of what follows.
Most offense-defense analysis naturally focuses on whether the attacker ultimately succeeds.
I am interested in another variable:
What is the cost and character of the equilibrium defense required to stop them?
Suppose AI makes some class of abuse far easier.
Institutions adapt successfully. Almost every serious attack is prevented.
But doing so requires persistent identity verification, tighter controls on anonymity, increased monitoring of transactions, restrictions on access to powerful tools, narrower permissions, and automated review of suspicious behavior.
It would be misleading to say that nothing bad happened because the attacks failed.
Threats impose costs without succeeding.
Military deterrence is expensive even when no war occurs. Locks cost money even if nobody attempts burglary. Anti-spam systems inconvenience legitimate senders even when spam is filtered successfully.
The same can happen socially.
A permissive institution can survive while exploiting it is uneconomical. Once optimization makes exploitation cheap, the institution must either become genuinely robust or become less permissive.
The second option is often much easier.
A platform that cannot distinguish automated abuse from legitimate high-volume use imposes rate limits.
A service that cannot distinguish genuine customers from promotion farming adds identity requirements.
An institution that cannot distinguish unusual benign behavior from an attack may standardize the behavior it accepts.
The harm is not only whatever exploitation gets through.
It is the contraction of the space everyone else is allowed to inhabit.
This is why I think the concept of slack matters.
Consider several things that look unrelated:
These are all partly sustained by limits on optimization.
Some cooperative arrangements survive because people leave gains on the table. Some privacy survives because inference is expensive. Some generosity survives because exploiting it is inconvenient.
Slack is often inefficient in a narrow sense. But removing every inefficiency does not necessarily produce a better human environment.
A world in which every action is evaluated for strategic advantage is not obviously desirable. Neither is a world in which every mistake is permanently searchable, every institution assumes adversarial intent, and every informal arrangement must be hardened against the most capable possible user.
The basic causal chain is:
bounded optimization → sustainable slack → valuable freedoms
Cheap machine optimization can instead produce:
cheap exploitation → defensive adaptation → defensive contraction
The optimistic route is:
cheap optimization → invariant defense → reconstructed slack
The goal is not to preserve incompetence. It is to replace the valuable protections that incompetence supplied accidentally with mechanisms that survive much stronger optimization.
Suppose an institution becomes worried about increasingly capable adversaries.
The obvious response is surveillance.
Observe more. Combine more data. Detect suspicious behavior earlier.
This can work. But observability is not the same as verifiability.
Seeing what someone does is not the same as understanding what it means. Understanding the current action is not the same as predicting its long-run consequences. Predicting danger is not useful if intervention comes too late.
Strategic behavior makes the distinction worse.
Ordinary institutions frequently rely on statistical safety. If some behavior is harmless in 99.999 percent of cases, permitting it may be sensible.
An adversary is not sampled randomly from those cases.
It searches for the remaining 0.001 percent.
The relevant question gradually shifts from:
Is this normally harmless?
toward:
Is there a strategically chosen version of this that causes serious harm?
This can push institutions from average-case governance toward worst-case governance.
And if the consequences become sufficiently irreversible, retrospective monitoring stops being enough.
The old model is:
act → observe → punish if necessary
The defensive model becomes:
verify → authorize → act
That is a qualitative change. It replaces some forms of permission by default with mediated access.
There is, however, a much better possibility than trying to make every person completely legible.
Verify the boundary, not the person.
Instead of proving that Alice has benign intentions, design the relevant interface so that Alice cannot obtain catastrophic leverage through it regardless of her intentions.
Computer security often works this way. We do not perfectly understand a user's psychology before allowing them onto a machine. We limit what the account can do.
The ideal civilizational analogue would preserve a huge interior space of human freedom while making a much smaller number of high-leverage interfaces robust.
People could remain private and strange. They could argue, experiment, form unusual communities, make mistakes, and change their minds. What would be unavailable is an obscure route from ordinary action to catastrophic consequence.
If advanced AI makes this kind of discrimination dramatically better, it could increase rather than decrease sustainable freedom.
The difficulty is that many important human goods are not cleanly formalizable.
"Does this credential authorize this file?" is a relatively crisp question.
"Was this interaction coercive?" is not.
Neither are questions like:
Human social life is full of context, ambiguity, history, and contested interpretation.
If increasingly powerful defense requires increasing verifiability, there may be pressure to redesign messy human activities into forms that are easier to verify.
The danger is not merely universal surveillance.
It is that the world gradually reorganizes around what can be verified.
Perhaps AI solves this by understanding human context well enough that crude standardization becomes unnecessary. That is the optimistic possibility.
But if it does not, defensive contraction could reach surprisingly deep into ordinary social life.
There is an even stranger possibility: some human goods may themselves depend on bounded optimization.
Much ordinary privacy comes from practical obscurity. Information exists, but nobody looks, remembers, combines, or infers enough from it.
Forgiveness partly depends on the fact that past errors are not converted into perfectly persistent strategic information.
Half-formed ideas can develop because they are not instantly exposed to exhaustive evaluation.
Local cultures can remain locally strange because no universal optimizer continuously models them.
If advanced machine intelligence removes these accidental protections, we may need deliberate replacements: artificial forgetting, information firewalls, limits on inference, protected private spaces, and constraints on how high-leverage systems interact with personal information.
Technology normally removes constraints.
A sufficiently mature technological civilization may need to manufacture some of them.
This is also why ordinary notions of AI alignment seem incomplete.
Imagine that every AI perfectly obeys its user.
Companies get extremely competent profit optimizers. Political movements get extremely competent persuasion systems. Governments get extremely competent strategic advisers. Individuals get extremely competent negotiators. Criminals get extremely competent planners.
No AI needs to be misaligned with its principal for the equilibrium to become undesirable.
Individual alignment does not imply equilibrium alignment.
The broader question is what happens when many actors can cheaply exert much stronger optimization pressure on one another.
One organization's use of AI can impose defensive costs on everyone else. Competitors must respond. Regulators must adapt. Customers need protection. Previously tolerable weaknesses become strategically relevant.
In this sense, intelligence can create an externality.
The user captures the local benefit of becoming more capable. The surrounding ecosystem may bear part of the cost of adjusting to that capability.
This suggests a different set of questions for AI governance.
Most importantly:
What kinds of civilization remain stable once powerful optimization becomes cheap?
There may be very optimistic answers.
Perhaps advanced AI makes invariant defense so powerful that we can have more privacy, more freedom, and more experimentation than today. Perhaps crude security measures such as mass surveillance eventually look primitive because intelligent systems can protect dangerous boundaries without making ordinary life legible.
There may also be less pleasant answers.
Perhaps some freedoms remain viable only while exploiting them is difficult. Perhaps some technologies require pervasive mediation. Perhaps the space of stable institutions narrows as increasingly obscure forms of exploitation become accessible.
I do not think we currently know.
But this seems like a question worth separating from the more familiar issue of whether AI "favors offense" or "favors defense."
The relevant variable is not only whether defense ultimately succeeds.
It is what successful defense requires society to become.
Protective incompetence is not the claim that stupidity is desirable.
It is the claim that human boundedness is an implicit part of our institutional threat model.
Civilization contains privacy, trust, forgiveness, ambiguity, and slack partly because exploiting all of them is expensive. Human beings have limited attention, patience, memory, coordination, and strategic depth. Many possible attacks never become economically or cognitively relevant.
Cheap machine optimization may change that.
The obvious danger is that attacks become stronger.
The less obvious danger is that even successful defense leaves us in a worse equilibrium.
If every permissive institution becomes an attack surface, one response is to make institutions less permissive.
The better response is to build systems whose safety does not depend on adversaries remaining weak.
We want privacy that survives powerful inference, trust mechanisms that remain viable under strategic pressure, and freedom that does not expose catastrophic leverage. We want invariant defenses that remove classes of attacks rather than defensive contraction that removes classes of ordinary behavior.
The goal is not to preserve incompetence.
It is to preserve the valuable equilibria that incompetence accidentally supported.
Superintelligence is usually discussed in terms of what enormously capable minds could accomplish. There is another question sitting beside it:
What kinds of civilization remain possible when powerful optimization is cheap?
If the things we value admit sufficiently general defenses, we may be able to replace protective incompetence with genuine robustness.
If they do not, then increasing intelligence may force a tradeoff between tolerating exploitation and contracting the space of human freedom.
Understanding which world we are in seems important before we find out by experiment.