To answer the question posed, I’d say the answer to the question posed is “Because our security isn’t nearly as on fire as people think, plus you are usually able to error-correct such that a first mistake isn’t fatal.”
For the computer security point, there’s a pretty large bias towards treating people that predict disaster as having more credit on the evidence then they do.
lc points out the general reasons for why, and anonymousaisafety points out that a large class of security bugs do not in fact work as useful exploits in practice, contra what Rowhammer worried people say:
For the cryptocurrency example, it’s fundamental basis is flawed, because it doesn’t have the properties a good money should have, and the money component is in practice enables only speculation about it’s value, and all of the good things crypto does have in reality would likely involve removing the money component, or using only play/fictional money that is allowed to increase.
Somewhat more generally, a big portion of the answer is that we can in fact learn from failures, and slowly based on empirical experience figure out ways for them to succeed, and a lot of proposed blockers to empirical testing turn out not to be nearly as much of a blocker as people say it is.
plus you are usually able to error-correct such that a first mistake isn’t fatal.”
This implies the answer is “trial and error”, but I really don’t think the whole answer is trial and error. Each of the domains I mentioned has the problem that you don’t get to redo things. If you send crypto to the wrong address it’s gone. People routinely type their credit card information into a website they’ve never visited before and get what they wanted. Global thermonuclear war didn’t happen. I strongly predict that when LLM agents come out, most people will successfully manage to use them without first falling for a string of prompt-injection attacks and learning from trial-and-error what prompts are/aren’t safe.
Humans are doing more than just trial and error, and figuring out what it is seems important.
To answer the question posed, I’d say the answer to the question posed is “Because our security isn’t nearly as on fire as people think, plus you are usually able to error-correct such that a first mistake isn’t fatal.”
For the computer security point, there’s a pretty large bias towards treating people that predict disaster as having more credit on the evidence then they do.
lc points out the general reasons for why, and anonymousaisafety points out that a large class of security bugs do not in fact work as useful exploits in practice, contra what Rowhammer worried people say:
https://www.lesswrong.com/posts/xsB3dDg5ubqnT7nsn/poc-or-or-gtfo-culture-as-partial-antidote-to-alignment
https://www.lesswrong.com/posts/etNJcXCsKC6izQQZj/pivotal-outcomes-and-pivotal-processes#ogt6CZkMNZ6oReuTk
For the cryptocurrency example, it’s fundamental basis is flawed, because it doesn’t have the properties a good money should have, and the money component is in practice enables only speculation about it’s value, and all of the good things crypto does have in reality would likely involve removing the money component, or using only play/fictional money that is allowed to increase.
Somewhat more generally, a big portion of the answer is that we can in fact learn from failures, and slowly based on empirical experience figure out ways for them to succeed, and a lot of proposed blockers to empirical testing turn out not to be nearly as much of a blocker as people say it is.
This implies the answer is “trial and error”, but I really don’t think the whole answer is trial and error. Each of the domains I mentioned has the problem that you don’t get to redo things. If you send crypto to the wrong address it’s gone. People routinely type their credit card information into a website they’ve never visited before and get what they wanted. Global thermonuclear war didn’t happen. I strongly predict that when LLM agents come out, most people will successfully manage to use them without first falling for a string of prompt-injection attacks and learning from trial-and-error what prompts are/aren’t safe.
Humans are doing more than just trial and error, and figuring out what it is seems important.
Yes, I underrated model-building here, and I do think that people sometimes underestimate how good humans actually are at model-building.