Essay

If Every User Is a Potential Threat

People are not becoming dishonest. They are becoming game-theoretically optimal for the environment they have been placed in.

/ Originally published on The Crumple Zone

Somewhere in the last decade, the presumption of innocence stopped being the default setting for dealing with an institution.

For centuries, the social contract was built on a default setting of trust. You were assumed to be who you said you were, and you were assumed to be acting in good faith until you broke a rule. The burden of proof lay on the accuser.

That architecture has been replaced.

In the digital world, the default setting is now “Zero Trust.” This is a cybersecurity concept that has leaked into governance. In a Zero Trust architecture, no entity — inside or outside the perimeter — is trusted by default. Every request must be verified. Every interaction is conditional. Every user is treated as a potential threat until they perform the “proof of work” required to establish a temporary state of innocence.

The change came out of arithmetic. When you operate at the scale of billions, the “tail” of bad actors — spammers, fraudsters, state-sponsored hackers — is numerically larger than the entire population of most countries. To a system administrator, the volume of attack traffic is indistinguishable from the volume of legitimate traffic.

The Asymmetry of Trust

The defining feature of this new regime is Asymmetric Irreversibility.

Modern institutions have automated the power to suspect, but they have not automated the power to exonerate. A fraud filter can freeze a bank account in milliseconds. A content moderation bot can ban a decade-old creative business in the time it takes to render a pixel. These systems operate at machine speed, with zero marginal cost per decision.

But correction operates on human time — slow, expensive, and uncertain.

What it produces is administrative homelessness: a class of people who are biologically alive and systemically dead. They cannot rent because of a zombie eviction record. They cannot work because of a background check error. They cannot bank because of a fraud flag. They are trapped in the database’s blind spot, unable to prove they exist.

flagged t = 0 suspended t + 40ms, automated restored if the appeal is ever heard administrative homelessness — cannot rent, cannot work, cannot bank correction runs on human time, at human cost, with no guarantee
The same round trip on one axis, to scale. The machine owns the first two ticks; everything after them is the part nobody automated.

The Rationality of Bad Behavior

This is the architecture that explains the shift in how people behave inside it. What looks like a collapse in standards is a rational adaptation to a system that opens by assuming you are an enemy.

Cheating becomes rational

When the official recovery form is a dead end, but a bribe to a “dark web” consultant works, honesty becomes a survival disadvantage. If the system doesn’t play by the rules — it bans you without cause and refuses to hear your appeal — why should you? Cheating is no longer deviance; it is the only remaining form of agency.

Rage becomes pointless

Anger is a social signal intended to shame the community into enforcing standards. But you are screaming at a mechanism. A “No-Reply” inbox cannot feel shame. A chatbot cannot be intimidated. Rage requires a recipient, and the system has designed itself to be un-reachable.

Politeness becomes humiliating

To get help, you must perform submission. You must say “please” and “thank you” to the very entity that is destroying your livelihood. You must validate the system’s “Theater of Care” — the empathy scripts, the apology emails — just to keep your ticket open. It feels humiliating because it is humiliating: you are performing compliance to your own abuse.

Withdrawal starts to seem like the only sane move

If every interaction carries a small chance of an uncorrectable “death penalty” — a lifetime ban, a frozen wallet — the only winning move is not to play. People stop posting. They stop verifying. They stop experimenting. They standardize their lives to fit the machine’s narrow template of a “good user.”

The Mirror

Correctness is becoming a luxury good, available to whoever can afford the “premium” tier where a human still answers.

For everyone else, there is only the algorithm, which sees the world through a single, paranoid lens.

If the system treats every user as a potential threat, eventually the users return the favor. We stop trusting institutions. We assume every form is a trap, every delay is a lie, and every rule is a weapon.

We are not losing our values. We are mirroring the machine.

designsystemstrust

View on The Crumple Zone

← Back to writing