As I see it, there are two separate problems. One is preventing catastrophic destruction of humanity (Problem 1). The other is creating utopia (Problem 2). Objective functions that are satisficing with respect to Problem 1 may not be solutions to Problem 2. While as I read it the Yudkowsky post you linked to argues for prioritizing Problem 2, on the contrary my sense of the thrust of Bostrom’s argument is that it’s critical to solve Problem 1. Maybe you can tell me if I’ve misunderstood.
Without implicating human values, I’m claiming that the function f(u) = D(u/ht) / D(u) satisfies Problem 1 (the existential problem). I’m just going to refer to that function as f now.
You seem have conceded this point. Maybe I’ve misinterpreted you.
As for solving Problem 2, I think we’d agree that any solutions to the utopia problem will also be solutions to the existence problem (Problem 1). The nice thing about f is that its range is (0,1), so it’s easy to compose it with other functions that could weight it more towards a solution to Problem 2.
Re: your second point:
I’m not sure if I entirely follow what you’re say here, so I’m having a hard time understanding exactly the point of disagreement.
Is the point you’re making about the unpredictability of the outcome of optimizing for f? Because the abstract patterns favored by f will look like noise relative to physics?
I think there are a couple elaborations worth making.
First, like Kolmogorov complexity, logical depth depends on a universal computer specification. I gather that you are assuming that the universal computer in question is something that simulated fundamental physics. This need not be the case. Depth is computed as the least running time of incompressible programs on the universal computer.
Suppose we were to try to evolve through a computational process a program that outputs a string that represented the ultimate, flourishing potential of humanity. One way to get that is to run the Earth as a physical process for a period of time and get a description of it at the end, selecting only those timelines in its stochastic unfolding in which life on Earth successfully computes itself indefinitely.
If you stop somewhere along the way, like timestep t, then you are going to get a representation that encodes some of the progress towards that teleological end.
(I think there’s a rough conceptual analogy to continuations in functional programming here, if that helps)
An important property of logical depth is the Slow Growth Law. This is proved by Bennett. It says that deep objects cannot be produced quickly from shallow ones. Incompressible programs being the shallowest strings of all. It’s not exactly that depth stacks additively, but I’m going to pretend it does for the intuitive argument here (which may be wrong):
If you have the depth of human progress D(h) and the depth of the universe at some future time D(u), then always D(u/h) < D(u) assuming h is deep at all and the computational products of humanity exist at all. But...
ah, I think I’ve messed up the formula. Let’s see… let’s have h’ be a human slice taken after the time of h.
D(u) > D(u/h’) > D(u/h) > D(h) assuming humanity’s computational process continues. The more that h’ encodes the total computational progress of u, i.e., the higher D(u/h’) is relative to D(u)...
Ok, I think I need to modify the formula some. Here’s function g:
g(u) = (D(h) + D(u/h)) / D(u)
Does maximizing this function produce better results? Or have I missed your point?
General response: I think you should revise the chances of this working way downwards until you have some sort of toy model where you can actually prove, completely, with no “obvious” assumptions necessary, that this will preserve values or at least the existence of an agent in a world. But I think enough has been said about this already.
Specific response:
Is the point you’re making about the unpredictability of the outcome of optimizing for f? Because the abstract patterns favored by f will look like noise relative to physics?
“Looks like noise” here means uncompressability, and thus logical shallowness. I’ll try again to explain why I think that relative logical depth turns out to not look like human values at all, and you can tell me what you think.
Consider an example.
Imagine, if you will, logical depth relative to a long string of nearly-random digits, called the Ongoing Tricky Procession. This is the computational work needed to output a string from its simplest description, if our agent already knows the Ongoing Tricky Procession.
On the other hand, boring old logical depth is the computational work needed to output a string from its simplest description period. The logical depth of the Ongoing Tricky Procession is not very big, even though it has a long description length.
Now imagine a contest between two agents, Alice and Bob. Alice knows the Ongoing Tricky Procession, and wants to output a string of high logical depth (to other agents who know the Ongoing Tricky Procession). The caveat is Bob has to think that the string has low logical depth. Is this possible?
The answer is yes. Alice and Bob are spies on opposite sides, and Alice is encrypting her deep message with a One Time Pad. Bob can’t decrypt the message because, as every good spy knows, One Time Pads are super-duper secure, and thus Bob can’t tell that Alice’s message is actually logically deep.
Even if the Ongoing Tricky Procession is not actually that K-complex, Alice can still hide a message in it—she just isn’t allowed to give Bob a simple description that actually decomposes into the OTP and the message.
This is almost the opposite of the Slow Growth Law. Slow Growth is where you have shallow inputs and you want to make a deep output. Alice has this deep message she wants to send to her homeland, but she wants her output to be shallow according to Bob. Fast Decay :P
Yes, I agree a toy setup and a proof are needed here. In case it wasn’t clear, my intentions with this post was to suss out if there was other related work out there already done (looks like there isn’t) and then do some intuition pumping in preparation for a deeper formal effort, in which you are instrumental and for which I am grateful. If you would be interested in working with me on this in a more formal way, I’m very open to collaboration.
Regarding your specific case, I think we may both be confused about the math. I think you are right that there’s something seriously wrong with the formulas I’ve proposed.
If the string y is incompressible and shallow, then whatever x is, D(x) ~ D(x/y), because D(x) (at least in the version I’m using for this argument) is the minimum computational time of producing x from an incompressible program. If there is a minimum running time program P that produces x, then appending y as noise at the end isn’t going to change the running time.
I think this case with incompressible y is like your Ongoing Tricky Procession.
On the other hand, say w is a string with high depth. Which is to say, whether or not it is compressible in space, it is compressible in time: you get it by starting with something incompressible and shallow and letting it run in time. Then there are going to be some strings x such that D(x/w) + D(w) ~ D(x). There will also be a lot of strings x such that D(x/w) ~ D(x) because D(w) is finite and there tons of deep things the universe can compute that are deeper. So for a given x, D(x) > D(x/w) > D(x) - D(w) , roughly speaking.
I’m saying the h, the humanity data, is logically deep, like w, not incompressible and shallow, like y or the ongoing tricky procession.
Hmm, it looks like I messed up the formula yet again.
What I’m trying to figure out is to select for universes u such that h is responsible for a maximal amount of the total depth. Maybe that’s a matter of minimizing D(u/h). Only that would lead perhaps to globe-flattening shallowness.
What if we tried to maximize D(u) - D(u/h)? That’s like the opposite of what I originally proposed.
I’m still confused as to what D(u/h) means. It looks like it should refer to the number of logical steps you need to predict the state of the universe—exactly, or up to a certain precision—given only knowledge of human history up to a certain point. But then any event you can’t predict without further information, such as the AI killing everyone using some astronomical phenomenon we didn’t include in the definition of “human history”, would have infinite or undefined D(u/h).
Re: your first point:
As I see it, there are two separate problems. One is preventing catastrophic destruction of humanity (Problem 1). The other is creating utopia (Problem 2). Objective functions that are satisficing with respect to Problem 1 may not be solutions to Problem 2. While as I read it the Yudkowsky post you linked to argues for prioritizing Problem 2, on the contrary my sense of the thrust of Bostrom’s argument is that it’s critical to solve Problem 1. Maybe you can tell me if I’ve misunderstood.
Without implicating human values, I’m claiming that the function f(u) = D(u/ht) / D(u) satisfies Problem 1 (the existential problem). I’m just going to refer to that function as f now.
You seem have conceded this point. Maybe I’ve misinterpreted you.
As for solving Problem 2, I think we’d agree that any solutions to the utopia problem will also be solutions to the existence problem (Problem 1). The nice thing about f is that its range is (0,1), so it’s easy to compose it with other functions that could weight it more towards a solution to Problem 2.
Re: your second point:
I’m not sure if I entirely follow what you’re say here, so I’m having a hard time understanding exactly the point of disagreement.
Is the point you’re making about the unpredictability of the outcome of optimizing for f? Because the abstract patterns favored by f will look like noise relative to physics?
I think there are a couple elaborations worth making.
First, like Kolmogorov complexity, logical depth depends on a universal computer specification. I gather that you are assuming that the universal computer in question is something that simulated fundamental physics. This need not be the case. Depth is computed as the least running time of incompressible programs on the universal computer.
Suppose we were to try to evolve through a computational process a program that outputs a string that represented the ultimate, flourishing potential of humanity. One way to get that is to run the Earth as a physical process for a period of time and get a description of it at the end, selecting only those timelines in its stochastic unfolding in which life on Earth successfully computes itself indefinitely.
If you stop somewhere along the way, like timestep t, then you are going to get a representation that encodes some of the progress towards that teleological end.
(I think there’s a rough conceptual analogy to continuations in functional programming here, if that helps)
An important property of logical depth is the Slow Growth Law. This is proved by Bennett. It says that deep objects cannot be produced quickly from shallow ones. Incompressible programs being the shallowest strings of all. It’s not exactly that depth stacks additively, but I’m going to pretend it does for the intuitive argument here (which may be wrong):
If you have the depth of human progress D(h) and the depth of the universe at some future time D(u), then always D(u/h) < D(u) assuming h is deep at all and the computational products of humanity exist at all. But...
ah, I think I’ve messed up the formula. Let’s see… let’s have h’ be a human slice taken after the time of h.
D(u) > D(u/h’) > D(u/h) > D(h) assuming humanity’s computational process continues. The more that h’ encodes the total computational progress of u, i.e., the higher D(u/h’) is relative to D(u)...
Ok, I think I need to modify the formula some. Here’s function g:
g(u) = (D(h) + D(u/h)) / D(u)
Does maximizing this function produce better results? Or have I missed your point?
General response: I think you should revise the chances of this working way downwards until you have some sort of toy model where you can actually prove, completely, with no “obvious” assumptions necessary, that this will preserve values or at least the existence of an agent in a world. But I think enough has been said about this already.
Specific response:
“Looks like noise” here means uncompressability, and thus logical shallowness. I’ll try again to explain why I think that relative logical depth turns out to not look like human values at all, and you can tell me what you think.
Consider an example.
Imagine, if you will, logical depth relative to a long string of nearly-random digits, called the Ongoing Tricky Procession. This is the computational work needed to output a string from its simplest description, if our agent already knows the Ongoing Tricky Procession.
On the other hand, boring old logical depth is the computational work needed to output a string from its simplest description period. The logical depth of the Ongoing Tricky Procession is not very big, even though it has a long description length.
Now imagine a contest between two agents, Alice and Bob. Alice knows the Ongoing Tricky Procession, and wants to output a string of high logical depth (to other agents who know the Ongoing Tricky Procession). The caveat is Bob has to think that the string has low logical depth. Is this possible?
The answer is yes. Alice and Bob are spies on opposite sides, and Alice is encrypting her deep message with a One Time Pad. Bob can’t decrypt the message because, as every good spy knows, One Time Pads are super-duper secure, and thus Bob can’t tell that Alice’s message is actually logically deep.
Even if the Ongoing Tricky Procession is not actually that K-complex, Alice can still hide a message in it—she just isn’t allowed to give Bob a simple description that actually decomposes into the OTP and the message.
This is almost the opposite of the Slow Growth Law. Slow Growth is where you have shallow inputs and you want to make a deep output. Alice has this deep message she wants to send to her homeland, but she wants her output to be shallow according to Bob. Fast Decay :P
Re: Generality.
Yes, I agree a toy setup and a proof are needed here. In case it wasn’t clear, my intentions with this post was to suss out if there was other related work out there already done (looks like there isn’t) and then do some intuition pumping in preparation for a deeper formal effort, in which you are instrumental and for which I am grateful. If you would be interested in working with me on this in a more formal way, I’m very open to collaboration.
Regarding your specific case, I think we may both be confused about the math. I think you are right that there’s something seriously wrong with the formulas I’ve proposed.
If the string y is incompressible and shallow, then whatever x is, D(x) ~ D(x/y), because D(x) (at least in the version I’m using for this argument) is the minimum computational time of producing x from an incompressible program. If there is a minimum running time program P that produces x, then appending y as noise at the end isn’t going to change the running time.
I think this case with incompressible y is like your Ongoing Tricky Procession.
On the other hand, say w is a string with high depth. Which is to say, whether or not it is compressible in space, it is compressible in time: you get it by starting with something incompressible and shallow and letting it run in time. Then there are going to be some strings x such that D(x/w) + D(w) ~ D(x). There will also be a lot of strings x such that D(x/w) ~ D(x) because D(w) is finite and there tons of deep things the universe can compute that are deeper. So for a given x, D(x) > D(x/w) > D(x) - D(w) , roughly speaking.
I’m saying the h, the humanity data, is logically deep, like w, not incompressible and shallow, like y or the ongoing tricky procession.
Hmm, it looks like I messed up the formula yet again.
What I’m trying to figure out is to select for universes u such that h is responsible for a maximal amount of the total depth. Maybe that’s a matter of minimizing D(u/h). Only that would lead perhaps to globe-flattening shallowness.
What if we tried to maximize D(u) - D(u/h)? That’s like the opposite of what I originally proposed.
I’m still confused as to what D(u/h) means. It looks like it should refer to the number of logical steps you need to predict the state of the universe—exactly, or up to a certain precision—given only knowledge of human history up to a certain point. But then any event you can’t predict without further information, such as the AI killing everyone using some astronomical phenomenon we didn’t include in the definition of “human history”, would have infinite or undefined D(u/h).