This distinction might be important in some particular cases. If it looks like an AGI might ascend to power with no real chance of being stopped by humanity, its decision about humanity might be swayed by just such abstract factors.
That consideration of being in a test might be the difference between our extinction, and our survival and flourishing by current standards.
This would also apply to the analagous consideration that alien ASIs might consider any new ASI that extincted its creators to be untrustworthy and therefore kill-on-sight.
None of this has anything to do with “niceness”, just selfish logic, so I don’t think it’s a response to the main topic of that post.
This distinction might be important in some particular cases. If it looks like an AGI might ascend to power with no real chance of being stopped by humanity, its decision about humanity might be swayed by just such abstract factors.
That consideration of being in a test might be the difference between our extinction, and our survival and flourishing by current standards.
This would also apply to the analagous consideration that alien ASIs might consider any new ASI that extincted its creators to be untrustworthy and therefore kill-on-sight.
None of this has anything to do with “niceness”, just selfish logic, so I don’t think it’s a response to the main topic of that post.