How do you know humans have tiers? The situation has never come up before. We’ve never had the infinite coincidence where the value at the highest tier is zero.
Also, why does it matter? It’s never going to come up either. If you program an AI to have tiers, it will quickly optimize that out. Why waste processing power on lower tiers if it has a chance of helping with the higher ones?
See: gedankenexperiment. I can guess what I’d choose given a blank white room.
And that is a flaw in the system. But it’s one that real-valued utility systems have as well. See: Pascal’s Mugging. An AI vulnerable to Pascal’s Mugging will just spend all its time breaking free of a hypothetical Matrix.
How do you know humans have tiers? The situation has never come up before. We’ve never had the infinite coincidence where the value at the highest tier is zero.
Also, why does it matter? It’s never going to come up either. If you program an AI to have tiers, it will quickly optimize that out. Why waste processing power on lower tiers if it has a chance of helping with the higher ones?
See: gedankenexperiment. I can guess what I’d choose given a blank white room.
And that is a flaw in the system. But it’s one that real-valued utility systems have as well. See: Pascal’s Mugging. An AI vulnerable to Pascal’s Mugging will just spend all its time breaking free of a hypothetical Matrix.
I did mention this under Flaws, you know...