Both are embedded in the world, but I meant the optimizer in that sentence. The original agent is even more nebulous than the unconstrained optimizer, since it might be operating under unknown constraints on design. (So it could well be cartesian, without self references. If we are introducing a separate optimizer, and only keeping idealized goals from the original agent, there is no more use for the original agent in the resulting story.)
In any case, a more general embedded decision theoretic optimizer should be defined from a position of awareness of the fact that it’s acting from within its world. What this should say about the optimizer itself is a question for decision theory that motivates its design.
Are you trying to advocate for decision theory? You write that this is “a question for decision theory”. But you also write that decision theory is “unclear for embedded agents”. And this whole conversation exclusively is about embedded agents. What parts are you advocating we use decision theory on and what parts are you advocating we don’t use decision theory on? I’m confused.
You write that this is “a question for decision theory”. But you also write that decision theory is “unclear for embedded agents”.
It’s a question of what decision theory for embedded agents should be, for which there is no clear answer. Without figuring that out, designing an optimizer is an even more murky endeavor, since we don’t have desiderata for it that make sense, which is what decision theory is about. So saying that decision theory for embedded agents is unclear is saying that designing embedded optimizers remains an ill-posed problem.
If clicking on the link doesn’t work, then that’s a bug with LW. I used the right link.
It is something of a bug with LW that results in giving you the wrong link to use (notice the #Wer2Fkueti2EvqmqN part of the link, which is the wrong part). The right link is this. It can be obtained by clicking “See in context” at the top of the page. (The threads remain uncombined, but at least they now have different topics.)
Both are embedded in the world, but I meant the optimizer in that sentence. The original agent is even more nebulous than the unconstrained optimizer, since it might be operating under unknown constraints on design. (So it could well be cartesian, without self references. If we are introducing a separate optimizer, and only keeping idealized goals from the original agent, there is no more use for the original agent in the resulting story.)
In any case, a more general embedded decision theoretic optimizer should be defined from a position of awareness of the fact that it’s acting from within its world. What this should say about the optimizer itself is a question for decision theory that motivates its design.
Are you trying to advocate for decision theory? You write that this is “a question for decision theory”. But you also write that decision theory is “unclear for embedded agents”. And this whole conversation exclusively is about embedded agents. What parts are you advocating we use decision theory on and what parts are you advocating we don’t use decision theory on? I’m confused.
It’s a question of what decision theory for embedded agents should be, for which there is no clear answer. Without figuring that out, designing an optimizer is an even more murky endeavor, since we don’t have desiderata for it that make sense, which is what decision theory is about. So saying that decision theory for embedded agents is unclear is saying that designing embedded optimizers remains an ill-posed problem.
I’m combining our two theads into one. Click here for continuation.
[Note: If clicking on the link doesn’t work, then that’s a bug with LW. I used the right link.][Edit: It was the wrong link.]
It is something of a bug with LW that results in giving you the wrong link to use (notice the #Wer2Fkueti2EvqmqN part of the link, which is the wrong part). The right link is this. It can be obtained by clicking “See in context” at the top of the page. (The threads remain uncombined, but at least they now have different topics.)
Fixed. Thank you.