My argument doesn’t require that anybody be able to formally define “self” or “maximize paperclips”; it doesn’t require the goal G to be picked among those that are easily defined in English.
An agent capable of reasoning about the world should be able to make an inference like “if all copies of me are destroyed, it makes it much less likely that goal G would be reached”; it may not have exactly that form, but it should be something analogous. It doesn’t matter if I can’t formalize that, the agent may not have a completely formal version either, only one that is sufficient for it’s purposes.
My argument doesn’t require that anybody be able to formally define “self” or “maximize paperclips”; it doesn’t require the goal G to be picked among those that are easily defined in English.
Show 3 examples of goal G. Somewhere I’ve read awesome technique for avoiding the abstraction mistakes—asking to show 3 examples.
What’s the point? Are you going to nitpick that my goals aren’t formal enough, even though I’m not making any claim at all about what kind of goals those could be?
Are you claiming that it’s impossible for an agent to have goals? That the set of goals that it’s even conceivable for an AI to have (without immediately wireheading or something) is much narrower than what most people here assume?
I’m not even sure what this disagreement is about right now, or even if there is a disagreement.
Ya, I think the set of goals is very narrow. The AI here starts of Descartes level genius and proceeds to self preserve, understand the map-territory distinction for non-wireheading, foreseeing the possibility that instrumental goals which look good may destroy the terminal goal, and such.
The AI I imagine starts off stupid and has some really narrowly (edit: or should i say, short-foresighted) self improving non self destructive goal likely having to do with maximization of complexity in some way. Think evolution, don’t think fully grown Descartes waking up after amnesia. It ain’t easy to reinvent the ‘self’. It’s also not easy to look at agent (yourself) and say—wow, this agent works to maximize G—without entering infinite recursion. We humans, if we escaped out of our universe into some super-universe, we might wreck some havoc but we’d sacrifice a bit of utility to preserve anything resembling life. Why? Well, we started stupid, and that’s how we got our goals.
My argument doesn’t require that anybody be able to formally define “self” or “maximize paperclips”; it doesn’t require the goal G to be picked among those that are easily defined in English.
An agent capable of reasoning about the world should be able to make an inference like “if all copies of me are destroyed, it makes it much less likely that goal G would be reached”; it may not have exactly that form, but it should be something analogous. It doesn’t matter if I can’t formalize that, the agent may not have a completely formal version either, only one that is sufficient for it’s purposes.
Show 3 examples of goal G. Somewhere I’ve read awesome technique for avoiding the abstraction mistakes—asking to show 3 examples.
What’s the point? Are you going to nitpick that my goals aren’t formal enough, even though I’m not making any claim at all about what kind of goals those could be?
Are you claiming that it’s impossible for an agent to have goals? That the set of goals that it’s even conceivable for an AI to have (without immediately wireheading or something) is much narrower than what most people here assume?
I’m not even sure what this disagreement is about right now, or even if there is a disagreement.
Ya, I think the set of goals is very narrow. The AI here starts of Descartes level genius and proceeds to self preserve, understand the map-territory distinction for non-wireheading, foreseeing the possibility that instrumental goals which look good may destroy the terminal goal, and such.
The AI I imagine starts off stupid and has some really narrowly (edit: or should i say, short-foresighted) self improving non self destructive goal likely having to do with maximization of complexity in some way. Think evolution, don’t think fully grown Descartes waking up after amnesia. It ain’t easy to reinvent the ‘self’. It’s also not easy to look at agent (yourself) and say—wow, this agent works to maximize G—without entering infinite recursion. We humans, if we escaped out of our universe into some super-universe, we might wreck some havoc but we’d sacrifice a bit of utility to preserve anything resembling life. Why? Well, we started stupid, and that’s how we got our goals.