In my life, I tried to guess a password O(10) times. I succeeded on the first try in two cases. This would seem to make this more feasible than you think.
Here there are two selection effects working against my argument:
I won’t even try to guess a password if I don’t think I have a chance for some reason;
I’m answering to this point because I’m someone who pulled this off, while people who never happened to guess a password on the first try will stay silent.
However, selection plays in favor of the hypothetical AI too: maybe you are confident you picked your password in a way that makes it unpredictable via public information, but there are other people who are not like that. Overall, about the question “Could it happen at least once that an important password was chosen in a way that made it predictable to an ASI, even assuming the ASI truly constrained in a box?”, I don’t feel confident either way right now.
> “guess a password on 1st try”
In my life, I tried to guess a password O(10) times. I succeeded on the first try in two cases. This would seem to make this more feasible than you think.
Here there are two selection effects working against my argument:
I won’t even try to guess a password if I don’t think I have a chance for some reason;
I’m answering to this point because I’m someone who pulled this off, while people who never happened to guess a password on the first try will stay silent.
However, selection plays in favor of the hypothetical AI too: maybe you are confident you picked your password in a way that makes it unpredictable via public information, but there are other people who are not like that. Overall, about the question “Could it happen at least once that an important password was chosen in a way that made it predictable to an ASI, even assuming the ASI truly constrained in a box?”, I don’t feel confident either way right now.