Yes, I agree. An upper bound on the complexity of a friendly superintelligence would be the total information content of all human brains, but you could probably compress that a lot.
False. The upper bound on the complexity of a Friendly superintelligence is [(total information content of all brains) + (minimum complexity of a what it takes to identify a superintelligence with a defined goal of deriving its objective from a set of agents according to some mechanism that represents what it would mean to be ‘friendly’ to them)].
ie. Just having all the information content of all human brains is not enough. You can’t avoid defining Friendliness in the search program.
I agree with wedrifid here. We don’t seem to have a valid argument showing that “an upper bound on the complexity of a friendly superintelligence would be the total information content of all human brains”. I would like to point out that if the K-complexity of friendly superintelligence is greater than that, then there is no way for us to build a friendly superintelligence except by luck (i.e., most Everett branches are doomed to fail to build a friendly superintelligence) or by somehow exploiting uncomputable physics.
Technically you can cheat by using the information in human brains to create upload-based superintelligences along the lines of Stuart’s proposal, make them do the research for you, etc., so it seems likely to me that the upper bound should hold… but I appreciate your point and agree that my comment was wrong as stated.
When talking about upper bounds we cannot afford to just cheat saying humans can probably figure it out. That isn’t an upper bound—it is an optimistic prediction about human potential. Moreover we still need a definition of Friendliness built in so we can tell whether this thing that the human researchers come up with is Friendliness or some other thing with that name. (Even an extremely ‘meta’ reference to a definition is fine but still requires more bits to point to which part of the humans is able to define Friendliness.)
Upper bounds are hard. But yes, I know your understanding of the area is solid and your ancestor comment serves as the definitive answer to the question of the post. I disagree only with the statement as made.
You are most likely not disagreeing with the intended meaning, so using words like “false” to motivate the clarification you were making is wrong.
No Vladimir. My disagreement was with the statement and the statement was, indeed false. That doesn’t mean I disagree with cousin_it’s overall philosophy. It just means I am calling a single false claim false.
False. The upper bound on the complexity of a Friendly superintelligence is [(total information content of all brains) + (minimum complexity of a what it takes to identify a superintelligence with a defined goal of deriving its objective from a set of agents according to some mechanism that represents what it would mean to be ‘friendly’ to them)].
ie. Just having all the information content of all human brains is not enough. You can’t avoid defining Friendliness in the search program.
I agree with wedrifid here. We don’t seem to have a valid argument showing that “an upper bound on the complexity of a friendly superintelligence would be the total information content of all human brains”. I would like to point out that if the K-complexity of friendly superintelligence is greater than that, then there is no way for us to build a friendly superintelligence except by luck (i.e., most Everett branches are doomed to fail to build a friendly superintelligence) or by somehow exploiting uncomputable physics.
Technically you can cheat by using the information in human brains to create upload-based superintelligences along the lines of Stuart’s proposal, make them do the research for you, etc., so it seems likely to me that the upper bound should hold… but I appreciate your point and agree that my comment was wrong as stated.
When talking about upper bounds we cannot afford to just cheat saying humans can probably figure it out. That isn’t an upper bound—it is an optimistic prediction about human potential. Moreover we still need a definition of Friendliness built in so we can tell whether this thing that the human researchers come up with is Friendliness or some other thing with that name. (Even an extremely ‘meta’ reference to a definition is fine but still requires more bits to point to which part of the humans is able to define Friendliness.)
Upper bounds are hard. But yes, I know your understanding of the area is solid and your ancestor comment serves as the definitive answer to the question of the post. I disagree only with the statement as made.
You are most likely not disagreeing with the intended meaning, so using words like “false” to motivate the clarification you were making is wrong.
No Vladimir. My disagreement was with the statement and the statement was, indeed false. That doesn’t mean I disagree with cousin_it’s overall philosophy. It just means I am calling a single false claim false.
You are wrong.