The internet of bad ideas

For much of the web's history, putting an idea into working software required significant effort. That effort filtered out some bad ideas and delayed plenty of good ones. Neither outcome was inherently desirable, but the cost forced some questions to be asked before a system was built.

As building becomes more accessible, we can act on a hunch sooner. A manager sees low engagement and orders a recommendation engine. A neighborhood group wants a membership system and receives a generated application. A founder has an attractive model of an opportunity and commissions the whole product before speaking to customers.

The implementation may be impressive. That tells us very little about whether the idea deserves to survive.

A release can pass its tests. The business case can still be unproven. A model can faithfully summarize a document that contains an error. A workflow can correctly execute an action that should not have been authorized.

These are different claims, and collapsing them into one green status is dangerous. It also makes honest failures difficult to learn from: the team either reports success because the build shipped or declares the whole effort a failure because its commercial result was disappointing.

The NIST AI RMF Playbook^{[1]NIST AI RMF Playbook} asks organizations to assess whether a system's intended benefits justify deployment and whether it is the right solution at all. A valid implementation can be the wrong intervention.

IllustrationThe loop and the missing question

1. What happensA prototype works
2. What it missesUsefulness is untested
3. What could changeLet real-world feedback change the plan

Suppose a community publication wants to rank local organizations by how responsive they are to residents. It gathers public comments, processes them with language models, gives each organization a score and publishes an attractive directory.

The software may work as specified. Yet the data could overrepresent the loudest participants, confuse organizations with similar names, and punish groups that lack a professional communications team. The problem is not the aesthetics of the directory or the accuracy of the code. It is the unsupported leap from available material to a public judgment.

A verifier should distinguish observation from inference and inference from an authorized decision. The team should be able to retract an unsupported result without pretending the whole source history never existed.

We don't want to replace easy creation with bureaucratic permission for every harmless prototype. There is enormous value in trying things cheaply and discovering new uses. The distinction should depend on consequence.

A private sketch can be revised freely. A public claim should be attributable and correctable. A financial transaction needs proper authority. A decision affecting another person's rights deserves a higher standard of explanation and review.

This is a direction worth pursuing for Aurekai's publishing and application systems. They should let people explore without confusing exploratory output with verified fact, or verified fact with permission to act.

A healthier abundant internet will not be one without bad ideas. It will be one where bad ideas are easier to challenge, improve, abandon and learn from before they become automated institutions.

See What happens when anyone can make an app? and When everyone agrees, who is thinking?.