Note that beyond not-being-mentioned, such arguments are also anthropically filtered against: in worlds where such arguments have been out there for longer, we died a lot quicker, so we’re not there to observe those arguments having been made.
This anthropic analysis doesn’t take into account past observers (see this post).
My current belief is that you do make some update upon observing existing, you just don’t update as much as if we were somehow able to survive and observe unaligned AI taking over. I do agree that the no update at all because you can’t see the counterfactual is wrong, but anthropics is still somewhat filtering your evidence; you should update less.
(I don’t have my full reasoning for {why I came to this conclusion} fully loaded rn, but I could probably do so if needed. Also, I only skimmed your post, sorry. I have a post on updating under anthropics with actual math I’m working on, but unsure when I’ll get around to finishing it.)
This anthropic analysis doesn’t take into account past observers (see this post).
My current belief is that you do make some update upon observing existing, you just don’t update as much as if we were somehow able to survive and observe unaligned AI taking over. I do agree that the no update at all because you can’t see the counterfactual is wrong, but anthropics is still somewhat filtering your evidence; you should update less.
(I don’t have my full reasoning for {why I came to this conclusion} fully loaded rn, but I could probably do so if needed. Also, I only skimmed your post, sorry. I have a post on updating under anthropics with actual math I’m working on, but unsure when I’ll get around to finishing it.)