General response: I think you should revise the chances of this working way downwards until you have some sort of toy model where you can actually prove, completely, with no “obvious” assumptions necessary, that this will preserve values or at least the existence of an agent in a world. But I think enough has been said about this already.
Specific response:
Is the point you’re making about the unpredictability of the outcome of optimizing for f? Because the abstract patterns favored by f will look like noise relative to physics?
“Looks like noise” here means uncompressability, and thus logical shallowness. I’ll try again to explain why I think that relative logical depth turns out to not look like human values at all, and you can tell me what you think.
Consider an example.
Imagine, if you will, logical depth relative to a long string of nearly-random digits, called the Ongoing Tricky Procession. This is the computational work needed to output a string from its simplest description, if our agent already knows the Ongoing Tricky Procession.
On the other hand, boring old logical depth is the computational work needed to output a string from its simplest description period. The logical depth of the Ongoing Tricky Procession is not very big, even though it has a long description length.
Now imagine a contest between two agents, Alice and Bob. Alice knows the Ongoing Tricky Procession, and wants to output a string of high logical depth (to other agents who know the Ongoing Tricky Procession). The caveat is Bob has to think that the string has low logical depth. Is this possible?
The answer is yes. Alice and Bob are spies on opposite sides, and Alice is encrypting her deep message with a One Time Pad. Bob can’t decrypt the message because, as every good spy knows, One Time Pads are super-duper secure, and thus Bob can’t tell that Alice’s message is actually logically deep.
Even if the Ongoing Tricky Procession is not actually that K-complex, Alice can still hide a message in it—she just isn’t allowed to give Bob a simple description that actually decomposes into the OTP and the message.
This is almost the opposite of the Slow Growth Law. Slow Growth is where you have shallow inputs and you want to make a deep output. Alice has this deep message she wants to send to her homeland, but she wants her output to be shallow according to Bob. Fast Decay :P
Yes, I agree a toy setup and a proof are needed here. In case it wasn’t clear, my intentions with this post was to suss out if there was other related work out there already done (looks like there isn’t) and then do some intuition pumping in preparation for a deeper formal effort, in which you are instrumental and for which I am grateful. If you would be interested in working with me on this in a more formal way, I’m very open to collaboration.
Regarding your specific case, I think we may both be confused about the math. I think you are right that there’s something seriously wrong with the formulas I’ve proposed.
If the string y is incompressible and shallow, then whatever x is, D(x) ~ D(x/y), because D(x) (at least in the version I’m using for this argument) is the minimum computational time of producing x from an incompressible program. If there is a minimum running time program P that produces x, then appending y as noise at the end isn’t going to change the running time.
I think this case with incompressible y is like your Ongoing Tricky Procession.
On the other hand, say w is a string with high depth. Which is to say, whether or not it is compressible in space, it is compressible in time: you get it by starting with something incompressible and shallow and letting it run in time. Then there are going to be some strings x such that D(x/w) + D(w) ~ D(x). There will also be a lot of strings x such that D(x/w) ~ D(x) because D(w) is finite and there tons of deep things the universe can compute that are deeper. So for a given x, D(x) > D(x/w) > D(x) - D(w) , roughly speaking.
I’m saying the h, the humanity data, is logically deep, like w, not incompressible and shallow, like y or the ongoing tricky procession.
Hmm, it looks like I messed up the formula yet again.
What I’m trying to figure out is to select for universes u such that h is responsible for a maximal amount of the total depth. Maybe that’s a matter of minimizing D(u/h). Only that would lead perhaps to globe-flattening shallowness.
What if we tried to maximize D(u) - D(u/h)? That’s like the opposite of what I originally proposed.
I’m still confused as to what D(u/h) means. It looks like it should refer to the number of logical steps you need to predict the state of the universe—exactly, or up to a certain precision—given only knowledge of human history up to a certain point. But then any event you can’t predict without further information, such as the AI killing everyone using some astronomical phenomenon we didn’t include in the definition of “human history”, would have infinite or undefined D(u/h).
General response: I think you should revise the chances of this working way downwards until you have some sort of toy model where you can actually prove, completely, with no “obvious” assumptions necessary, that this will preserve values or at least the existence of an agent in a world. But I think enough has been said about this already.
Specific response:
“Looks like noise” here means uncompressability, and thus logical shallowness. I’ll try again to explain why I think that relative logical depth turns out to not look like human values at all, and you can tell me what you think.
Consider an example.
Imagine, if you will, logical depth relative to a long string of nearly-random digits, called the Ongoing Tricky Procession. This is the computational work needed to output a string from its simplest description, if our agent already knows the Ongoing Tricky Procession.
On the other hand, boring old logical depth is the computational work needed to output a string from its simplest description period. The logical depth of the Ongoing Tricky Procession is not very big, even though it has a long description length.
Now imagine a contest between two agents, Alice and Bob. Alice knows the Ongoing Tricky Procession, and wants to output a string of high logical depth (to other agents who know the Ongoing Tricky Procession). The caveat is Bob has to think that the string has low logical depth. Is this possible?
The answer is yes. Alice and Bob are spies on opposite sides, and Alice is encrypting her deep message with a One Time Pad. Bob can’t decrypt the message because, as every good spy knows, One Time Pads are super-duper secure, and thus Bob can’t tell that Alice’s message is actually logically deep.
Even if the Ongoing Tricky Procession is not actually that K-complex, Alice can still hide a message in it—she just isn’t allowed to give Bob a simple description that actually decomposes into the OTP and the message.
This is almost the opposite of the Slow Growth Law. Slow Growth is where you have shallow inputs and you want to make a deep output. Alice has this deep message she wants to send to her homeland, but she wants her output to be shallow according to Bob. Fast Decay :P
Re: Generality.
Yes, I agree a toy setup and a proof are needed here. In case it wasn’t clear, my intentions with this post was to suss out if there was other related work out there already done (looks like there isn’t) and then do some intuition pumping in preparation for a deeper formal effort, in which you are instrumental and for which I am grateful. If you would be interested in working with me on this in a more formal way, I’m very open to collaboration.
Regarding your specific case, I think we may both be confused about the math. I think you are right that there’s something seriously wrong with the formulas I’ve proposed.
If the string y is incompressible and shallow, then whatever x is, D(x) ~ D(x/y), because D(x) (at least in the version I’m using for this argument) is the minimum computational time of producing x from an incompressible program. If there is a minimum running time program P that produces x, then appending y as noise at the end isn’t going to change the running time.
I think this case with incompressible y is like your Ongoing Tricky Procession.
On the other hand, say w is a string with high depth. Which is to say, whether or not it is compressible in space, it is compressible in time: you get it by starting with something incompressible and shallow and letting it run in time. Then there are going to be some strings x such that D(x/w) + D(w) ~ D(x). There will also be a lot of strings x such that D(x/w) ~ D(x) because D(w) is finite and there tons of deep things the universe can compute that are deeper. So for a given x, D(x) > D(x/w) > D(x) - D(w) , roughly speaking.
I’m saying the h, the humanity data, is logically deep, like w, not incompressible and shallow, like y or the ongoing tricky procession.
Hmm, it looks like I messed up the formula yet again.
What I’m trying to figure out is to select for universes u such that h is responsible for a maximal amount of the total depth. Maybe that’s a matter of minimizing D(u/h). Only that would lead perhaps to globe-flattening shallowness.
What if we tried to maximize D(u) - D(u/h)? That’s like the opposite of what I originally proposed.
I’m still confused as to what D(u/h) means. It looks like it should refer to the number of logical steps you need to predict the state of the universe—exactly, or up to a certain precision—given only knowledge of human history up to a certain point. But then any event you can’t predict without further information, such as the AI killing everyone using some astronomical phenomenon we didn’t include in the definition of “human history”, would have infinite or undefined D(u/h).