Don’t these kinds of considerations apply to any decision theory? Don’t they all suppose that you’re given some kind of carving-up of the world into various things with various manipulable properties? Don’t they all suppose that you have some kind of identity criteria for saying when things are “the same”, and for partitioning up events to assign payoffs to them? Is any decision theory responsible for dictating what your initial carving-up of the world should be?
I think that TDT and UDT assume that the agent, for whatever reason, starts out with a given decomposition of itself into program and data. If it had started with a different decomposition, it would have been a different agent, and so, unsurprisingly, might have made different decisions.
This is the reason why I don’t think decision theory is that fundamental to AI. Suppose you have an AI, how should it decide what decision problem it is facing. E.g. what its options are. In reality the choice is never as stark as just one or two boxing. We are often exhorted to think outside the box, if you’ll pardon the pun.
It is not as if we humans get told what we are supposed to be doing or deciding between in this life. We have to invent it for ourselves.
Systems without the ability to go beyond the mental model their creators have (at a certain point in time), are subject to whatever flaws that mental model possesses. I wouldn’t classify them as full intelligences.
I wouldn’t want a flawed system to be the thing to guide humanity to the future.
Systems without the ability to go beyond the mental model their creators have (at a certain point in time), are subject to whatever flaws that mental model possesses.
Where does the basis for deciding something to be a flaw reside?
In humans? No one knows. My best guess at the moment for the lowest level of model choice is some form of decentralised selectionist system, that is much as decision theoretic construct as real evolution is.
We do of course have higher level model choosing systems that might work on a decision theoretic basis, but they have models implicit in them which can be flawed.
Improving the mental model is right there at the centre of the box. Creating a GAI that doesn’t operate according to some sort of decision theory? That’s, well, out of the box crazy talk.
Are you objecting to the possibility of a General intelligence not based on a decision theory at its foundation, or do you just think one would be unsafe?
Do you think us humans are based on some form of decision theory?
Are you objecting to the possibility of a General intelligence not based on a decision theory at its foundation, or do you just think one would be unsafe?
Unsafe.
Do you think us humans are based on some form of decision theory?
No. And I wouldn’t trust a fellow human with that sort of uncontrolled power.
My understanding is that TDT and UDT are supposed to be used by an agent that we design. In all likelihood, we will have decomposed the agent into program and data in the process of designing it. When the agent starts to use the decision theory, it can take that decomposition as given.
This consideration applies to ourselves, insofar as we have a hand in designing ourselves.
My understanding is that TDT and UDT are supposed to be used by an agent that we design. In all likelihood, we will have decomposed the agent into program and data in the process of designing it.
Reading this statement, it comes across as quite objectionable. I think that this is because dividing something into program and data seems it cannot be done in a non-arbitrary manner—many programming languages don’t distinguish between code and data, and a universal Turing machine must interpret its input as program at some point.
Perhaps one could have as special “how to write a program” decision theory, but that would not be a general decision theory applicable to all other decisions.
Isn’t this like criticizing Bayesianism because it doesn’t tell you how to select your initial prior? For practical purposes, that doesn’t matter because you already have a prior; and once you have a prior, Bayesianism is enough to go on from there.
Similarly, you already decompose at least some part of yourself into program and data (don’t you?). This is enough for that part of yourself to work with these decision theories. And using them, you can proceed to decide how to decompose the rest of yourself, or even to reflect on the original decomposition and choose a new one.
The following is slightly tongue in cheek, but I don’t normally place a stable boundary between program and data on myself, I revise it depending on purpose. The following is one view I find useful sometimes
Nope, I’m all program. What you would call data is just programming in weaker languages than Turing complete ones. I can rewrite my programming, do meta analysis on it.
The information streaming into my eyes is a program that I don’t know what it will make me do, it could make me flinch or it change the conceptual way that I see the world. The visual system is just an interpreter for the programming optical signals.
“Prior” is like a get out of jail card. Whenever the solution to some problem turns out to conveniently depend on an unknown probability distribution, you can investigate further, or you can say “prior” and stop there. For example, the naive Bayesian answer to game theory would be “just optimize based on your prior over the enemy’s actions”, which would block the route to discovering Nash equilibria.
It’s true that it’s worthwhile to investigate where priors ought to come from. My point is only that you can still put Bayesianism to work even before you’ve made such investigations.
Don’t these kinds of considerations apply to any decision theory? Don’t they all suppose that you’re given some kind of carving-up of the world into various things with various manipulable properties? Don’t they all suppose that you have some kind of identity criteria for saying when things are “the same”, and for partitioning up events to assign payoffs to them? Is any decision theory responsible for dictating what your initial carving-up of the world should be?
I think that TDT and UDT assume that the agent, for whatever reason, starts out with a given decomposition of itself into program and data. If it had started with a different decomposition, it would have been a different agent, and so, unsurprisingly, might have made different decisions.
This is the reason why I don’t think decision theory is that fundamental to AI. Suppose you have an AI, how should it decide what decision problem it is facing. E.g. what its options are. In reality the choice is never as stark as just one or two boxing. We are often exhorted to think outside the box, if you’ll pardon the pun.
It is not as if we humans get told what we are supposed to be doing or deciding between in this life. We have to invent it for ourselves.
Please do not ever create an AI capable of recursively self improvement. ‘Thinking outside the box’ is a bug.
Systems without the ability to go beyond the mental model their creators have (at a certain point in time), are subject to whatever flaws that mental model possesses. I wouldn’t classify them as full intelligences.
I wouldn’t want a flawed system to be the thing to guide humanity to the future.
Where does the basis for deciding something to be a flaw reside?
In humans? No one knows. My best guess at the moment for the lowest level of model choice is some form of decentralised selectionist system, that is much as decision theoretic construct as real evolution is.
We do of course have higher level model choosing systems that might work on a decision theoretic basis, but they have models implicit in them which can be flawed.
Improving the mental model is right there at the centre of the box. Creating a GAI that doesn’t operate according to some sort of decision theory? That’s, well, out of the box crazy talk.
We might be having different definitions of thinking outside of the box, here.
Are you objecting to the possibility of a General intelligence not based on a decision theory at its foundation, or do you just think one would be unsafe?
Do you think us humans are based on some form of decision theory?
Unsafe.
No. And I wouldn’t trust a fellow human with that sort of uncontrolled power.
Ordinary Causal Decision Theory does not depend on a carving of agents into programs and data.
My understanding is that TDT and UDT are supposed to be used by an agent that we design. In all likelihood, we will have decomposed the agent into program and data in the process of designing it. When the agent starts to use the decision theory, it can take that decomposition as given.
This consideration applies to ourselves, insofar as we have a hand in designing ourselves.
Reading this statement, it comes across as quite objectionable. I think that this is because dividing something into program and data seems it cannot be done in a non-arbitrary manner—many programming languages don’t distinguish between code and data, and a universal Turing machine must interpret its input as program at some point.
Perhaps one could have as special “how to write a program” decision theory, but that would not be a general decision theory applicable to all other decisions.
Isn’t this like criticizing Bayesianism because it doesn’t tell you how to select your initial prior? For practical purposes, that doesn’t matter because you already have a prior; and once you have a prior, Bayesianism is enough to go on from there.
Similarly, you already decompose at least some part of yourself into program and data (don’t you?). This is enough for that part of yourself to work with these decision theories. And using them, you can proceed to decide how to decompose the rest of yourself, or even to reflect on the original decomposition and choose a new one.
The following is slightly tongue in cheek, but I don’t normally place a stable boundary between program and data on myself, I revise it depending on purpose. The following is one view I find useful sometimes
Nope, I’m all program. What you would call data is just programming in weaker languages than Turing complete ones. I can rewrite my programming, do meta analysis on it.
The information streaming into my eyes is a program that I don’t know what it will make me do, it could make me flinch or it change the conceptual way that I see the world. The visual system is just an interpreter for the programming optical signals.
“Prior” is like a get out of jail card. Whenever the solution to some problem turns out to conveniently depend on an unknown probability distribution, you can investigate further, or you can say “prior” and stop there. For example, the naive Bayesian answer to game theory would be “just optimize based on your prior over the enemy’s actions”, which would block the route to discovering Nash equilibria.
It’s true that it’s worthwhile to investigate where priors ought to come from. My point is only that you can still put Bayesianism to work even before you’ve made such investigations.