plus you are usually able to error-correct such that a first mistake isn’t fatal.”
This implies the answer is “trial and error”, but I really don’t think the whole answer is trial and error. Each of the domains I mentioned has the problem that you don’t get to redo things. If you send crypto to the wrong address it’s gone. People routinely type their credit card information into a website they’ve never visited before and get what they wanted. Global thermonuclear war didn’t happen. I strongly predict that when LLM agents come out, most people will successfully manage to use them without first falling for a string of prompt-injection attacks and learning from trial-and-error what prompts are/aren’t safe.
Humans are doing more than just trial and error, and figuring out what it is seems important.
This implies the answer is “trial and error”, but I really don’t think the whole answer is trial and error. Each of the domains I mentioned has the problem that you don’t get to redo things. If you send crypto to the wrong address it’s gone. People routinely type their credit card information into a website they’ve never visited before and get what they wanted. Global thermonuclear war didn’t happen. I strongly predict that when LLM agents come out, most people will successfully manage to use them without first falling for a string of prompt-injection attacks and learning from trial-and-error what prompts are/aren’t safe.
Humans are doing more than just trial and error, and figuring out what it is seems important.
Yes, I underrated model-building here, and I do think that people sometimes underestimate how good humans actually are at model-building.