This proposal doesn’t depend on mugging the AI. The proposal actually gets the AI more resources in expectation due to a trade.
I agree the post is a bit confusing and unclear about this. (And the proposal under “Can we get more than this” is wrong. At a minimum, such AIs will also be mugged by everyone else too meaning you get get huge amounts of extra money for basically free.)
This doesn’t seem as a fair trade proposal to me. This is a bet where one side has disproportional amount of information and uses it to its own benefit.
Suppose I tossed a fair coin, looked on the outcome and proposed you to bet on Heads with 99:1 odds. Is it reasonable for you to agree?
This proposal doesn’t depend on mugging the AI. The proposal actually gets the AI more resources in expectation due to a trade.
I agree the post is a bit confusing and unclear about this. (And the proposal under “Can we get more than this” is wrong. At a minimum, such AIs will also be mugged by everyone else too meaning you get get huge amounts of extra money for basically free.)
This doesn’t seem as a fair trade proposal to me. This is a bet where one side has disproportional amount of information and uses it to its own benefit.
Suppose I tossed a fair coin, looked on the outcome and proposed you to bet on Heads with 99:1 odds. Is it reasonable for you to agree?