Saying that “naively maximizing the agency of future agents would involve ensuring that they only have very easily-satisfied preferences” is clearly wrong. You appear to be completely misdefining agency here. Agency is full of the ability to come to decisions and value things on your own, not having it picked for you. It is the ability to try to make the world ‘more’ to your liking, not for the world to just be the way you like.
An agent that does not exist has zero agency. An agent that does exist, but has been fully controlled has zero agency. Only agents that make real choices in the world have agency. The maximally satisfiable being (let’s stop calling it an agent) does nothing, or does things without regard to how the world should be and thus has no agency. Maximizing agency does not equal creating only beings with zero agency.
This glaring error makes the whole ‘cooperation-morality’ segment seem to be shakily reasoned. I’m not sure how it changes things, but having a third of it this way makes the whole post seem unreliable.
It’s weird to think about what “respecting agency” means when the agent in question doesn’t currently exist and you are building it from scratch and you get to build it however you want. You can’t apply normal intuitions here.
For example, brainwashing a human such that they are only motivated to play tic-tac-toe is obviously not “respecting their agency”. We’re all on the same page for that situation.
But what if we build an AGI from scratch such that it is only motivated to play tic-tac-toe, and then we let the AGI do whatever it wants in the world (which happens to be playing tic-tac-toe)? Are we disrespecting its agency? If so, I don’t feel the force of the argument that this is bad. Who exactly are we harming here? Is it worse than not making this AGI in the first place?
Was evolution “disrespecting my agency” when I was born with a hunger drive and sex drive and status drive etc.? If not, why would it be any different to make an AGI with (only) a tic-tac-toe drive? Or if yes, well, we face the problem that we need to put some drives into our AGI or else there’s no “agent” at all, just a body that takes random actions, or doesn’t do anything at all.
I never talked about respect in my points for a reason. This isn’t about respect. It’s about how it is not an agent if it doesn’t do anything in an attempt to make the world more to its liking. If it does nothing, or does things randomly (without regard to making things better), that is hardly agentic. If I don’t care at all about colors, then picking between a red shirt, and an otherwise identical blue shirt is not an agentic choice, merely a necessary choice (and I will not likely have even bothered thinking about it as a choice involving color.). Identically, if I just have to always wear a specific color, I am not being an agent by wearing that color. There are obviously degrees of agency, too, but the article is genuinely assuming that beings that do basically nothing are still agents.
Saying that “naively maximizing the agency of future agents would involve ensuring that they only have very easily-satisfied preferences” is clearly wrong. You appear to be completely misdefining agency here. Agency is full of the ability to come to decisions and value things on your own, not having it picked for you. It is the ability to try to make the world ‘more’ to your liking, not for the world to just be the way you like.
An agent that does not exist has zero agency. An agent that does exist, but has been fully controlled has zero agency. Only agents that make real choices in the world have agency. The maximally satisfiable being (let’s stop calling it an agent) does nothing, or does things without regard to how the world should be and thus has no agency. Maximizing agency does not equal creating only beings with zero agency.
This glaring error makes the whole ‘cooperation-morality’ segment seem to be shakily reasoned. I’m not sure how it changes things, but having a third of it this way makes the whole post seem unreliable.
It’s weird to think about what “respecting agency” means when the agent in question doesn’t currently exist and you are building it from scratch and you get to build it however you want. You can’t apply normal intuitions here.
For example, brainwashing a human such that they are only motivated to play tic-tac-toe is obviously not “respecting their agency”. We’re all on the same page for that situation.
But what if we build an AGI from scratch such that it is only motivated to play tic-tac-toe, and then we let the AGI do whatever it wants in the world (which happens to be playing tic-tac-toe)? Are we disrespecting its agency? If so, I don’t feel the force of the argument that this is bad. Who exactly are we harming here? Is it worse than not making this AGI in the first place?
Was evolution “disrespecting my agency” when I was born with a hunger drive and sex drive and status drive etc.? If not, why would it be any different to make an AGI with (only) a tic-tac-toe drive? Or if yes, well, we face the problem that we need to put some drives into our AGI or else there’s no “agent” at all, just a body that takes random actions, or doesn’t do anything at all.
I never talked about respect in my points for a reason. This isn’t about respect. It’s about how it is not an agent if it doesn’t do anything in an attempt to make the world more to its liking. If it does nothing, or does things randomly (without regard to making things better), that is hardly agentic. If I don’t care at all about colors, then picking between a red shirt, and an otherwise identical blue shirt is not an agentic choice, merely a necessary choice (and I will not likely have even bothered thinking about it as a choice involving color.). Identically, if I just have to always wear a specific color, I am not being an agent by wearing that color. There are obviously degrees of agency, too, but the article is genuinely assuming that beings that do basically nothing are still agents.