I think that the issue could be tracked back to the SIA definition. It says “possible observers”, and I think that physically possible are meant.
For example, it is conceivable that each planet is inhabited by billions trolls, but there is no physical basis for this. (though folk mythology claims exactly this—any stone has its spirit)
You still have the issue that “possible” has multiple meanings, and that physical possibility is more restrictive than conceivability.
I think that the issue could be tracked back to the SIA definition. It says “possible observers”, and I think that physically possible are meant.
For example, it is conceivable that each planet is inhabited by billions trolls, but there is no physical basis for this. (though folk mythology claims exactly this—any stone has its spirit)