What can we learn from insecure domains?

[Logan Zoellner]

Cryptocurrency is terrible. With a single click of a button, it is possible to accidentally lose all of your funds. 99.9% of all cryptocurrency projects are complete scams (conservative estimate). Crypto is also tailor-made for ransomware attacks, since it makes it possible to send money in such a way that the receiver has perfect anonymity.

typical web3 experience

Similarly, Cyber Security is terrible. Basically every computer on the internet is infected with multiple types of malware. If you have ever owned a web-server with a public IPV4 address, you undoubtedly have had the pleasure of viewing a log file that looks like this:

Every IPV4 address on earth is under constant attack by malware

In a few months, the world is about to be introduced to a brand new insecure by design platform, the LLM agent:

No one worth taking seriously believes that Microsoft Copilot (or Anthropic, or any other LLM agent) is going to be remotely secure against prompt injection attacks.

One fascinating thing (to me) about these examples is that they all basically work fine^[1]. Despite being completely broken, normal people with normal intelligence use these systems routinely without losing 100% of their funds. This happens despite the fact that people with above-average intelligence have a financial incentive to take advantage of these security flaws.

One possible conclusion is along the lines of “everything humanity has ever built is constantly on fire. We must never built something existentially dangerous or we’re already dead.”

However we already did:

And like everything else, the story of nuclear weapons is that they are horribly insecure and error prone.

What I want to know is why? Why is it that all of these systems, despite being hideously error prone and blatantly insecure by design somehow still work?

I consider each of these systems (and many like them) a sort of standing challenge to the fragile world hypothesis. If the world is so fragile, why does it keep not ending?

1. ^

   If anyone would like to make a bet, I predict 2 years from now LLM agents:

   1. will be vulnerable to nearly trivial forms of prompt-injection

   2. Millions of people will use them to do things like spend money that common-sense tells you not to do on a platform this insecure by design

[ProgramCrafter]

99.9% of all cryptocurrency projects are complete scams (conservative estimate).

On first skim, I agree with the estimate as stated and would post a limit order for either side. I’d also like to note that “crypto in general is terrible” instead of “all crypto is terrible”, as there have been applications developed that do not allow you to lose all funds without explicit acknowledgement.

Similarly, Cyber Security is terrible. Basically every computer on the internet is infected with multiple types of malware.

It is presumably terrible (or, 30%, result of availability bias), and I’ve observed bugs happen because functionality upgrade did not consider its interaction with all other code. However, I disagree that every computer is infected; probably you meant that it is under constant stream of attack attempts?

The insecure domains mainly work because people have charted known paths, and shown that if you follow those paths your loss probability is non-null but small. As a matter of IT, it would be really nice to have systems which don’t logically fail at all, but that requires good education and pressure-resistance skills for software developers.