I think there is a very legitimate sense in which optimizing the steps of a plan to do a thing is a separate skill and/or mental propensity to executing that plan (as in, actually sending those signals outside the computer) or wanting it executed, and in which agency is mostly a measure of the latter.
My main criticism is that, in general, you have to think while you’re executing plans, not just while you’re generating them. The paradigm where you plan every step in advance, and then the “agency” comes in only when executing it, is IMO a very misleading one to think in.
(This seems related to Eliezer’s argument that there’s only a one-line difference difference between an oracle AGI and an agent AGI. Sure, that’s true in the limit. But thinking about the limit will make you very confused about realistic situations!)
I’m not sure what the practical difference is between criticizing a post and criticizing people that upvoted it
It’s something like: “I endorse people following the policy of writing posts like this one, it’s great when people work through their thoughts in this way. I don’t endorse people following the policy of upvoting posts like this one to this extent, because it seems likely that they’re mainly responding to high-level applause lights.”
to the extent that this is a criticism of the post I wish you had been more explicit about what you are objecting to.
I’m sympathetic to you wanting more explicit feedback but the fact that this post is so high-level and ungrounded is what makes it difficult for me to give that. To me it reads more like a story than an argument.
The paradigm where you plan every step in advance, and then the “agency” comes in only when executing it, is IMO a very misleading one to think in.
This isn’t what I’m referring to and it’s not in the example in the story. Actions are generated stepwise on demand. It is the ability to generate stepwise outputs of good quality, of which actions are an instance, that is ‘optimization power’. Being able to think of good next actions conditional on past observations is, at least as I understand the terms, quite different to being an agent enacting those actions.
(This seems related to Eliezer’s argument that there’s only a one-line difference difference between an oracle AGI and an agent AGI. Sure, that’s true in the limit. But thinking about the limit will make you very confused about realistic situations!)
I explicitly tried to make the scenario as un-Oracle like as I could, with the system explicitly only producing outputs onscreen that I could explicitly justify being discoverable in reasonable time given the observations it had available.
I am increasingly feeling like I just failed to communicate what I was trying to say and your criticism doesn’t bear much resemblance to what I had intended to write. I’m happy to take responsibility for not writing as well as I should have, but I’d rather you didn’t cast aspersions at my motivations about it.
I didn’t read the post particularly carefully, it’s totally plausible that I’m misunderstanding the key ideas you were trying to convey. I apologise for phrasing my claims in a way that made it sound like I was skeptical of your motivations; I’m not, and I’m glad you wrote this up.
I think my concerns still apply to the position you stated in the previous comment, but insofar as the main motivation behind my comment was to generically nudge LW in a certain direction, I’ll try to do this more directly, rather than via poking at individual posts in an opportunistic way.
My main criticism is that, in general, you have to think while you’re executing plans, not just while you’re generating them. The paradigm where you plan every step in advance, and then the “agency” comes in only when executing it, is IMO a very misleading one to think in.
(This seems related to Eliezer’s argument that there’s only a one-line difference difference between an oracle AGI and an agent AGI. Sure, that’s true in the limit. But thinking about the limit will make you very confused about realistic situations!)
It’s something like: “I endorse people following the policy of writing posts like this one, it’s great when people work through their thoughts in this way. I don’t endorse people following the policy of upvoting posts like this one to this extent, because it seems likely that they’re mainly responding to high-level applause lights.”
I’m sympathetic to you wanting more explicit feedback but the fact that this post is so high-level and ungrounded is what makes it difficult for me to give that. To me it reads more like a story than an argument.
This isn’t what I’m referring to and it’s not in the example in the story. Actions are generated stepwise on demand. It is the ability to generate stepwise outputs of good quality, of which actions are an instance, that is ‘optimization power’. Being able to think of good next actions conditional on past observations is, at least as I understand the terms, quite different to being an agent enacting those actions.
I explicitly tried to make the scenario as un-Oracle like as I could, with the system explicitly only producing outputs onscreen that I could explicitly justify being discoverable in reasonable time given the observations it had available.
I am increasingly feeling like I just failed to communicate what I was trying to say and your criticism doesn’t bear much resemblance to what I had intended to write. I’m happy to take responsibility for not writing as well as I should have, but I’d rather you didn’t cast aspersions at my motivations about it.
I didn’t read the post particularly carefully, it’s totally plausible that I’m misunderstanding the key ideas you were trying to convey. I apologise for phrasing my claims in a way that made it sound like I was skeptical of your motivations; I’m not, and I’m glad you wrote this up.
I think my concerns still apply to the position you stated in the previous comment, but insofar as the main motivation behind my comment was to generically nudge LW in a certain direction, I’ll try to do this more directly, rather than via poking at individual posts in an opportunistic way.