“Unlimited power” is a non-starter. No matter how powerful the AGI is it will be of finite power. Unlimited power is the stuff of theology not of actually achievable minds. Thus the ditty from Epicurus about “God” does not apply. This is not a trivial point. I have a concern Eliezer may get too caught up in these grand sagas and great dilemnas on precisely such a theological absolutist scale. Arguing as if unlimited power is real takes us well into the current essay.
“Wieldable without side effects or configuration constraints” is more of the same. Imaginary thinking far beyond the constraints of any actually realizable situation. More theological daydreaming. It should go without saying that there is no such thing as operating without any configuration constraints and with perfect foresight of all possibly effects. I grow more concerned.
“It is wielded with unlimited precision”?! Come on, what is the game here? Surely you do not believe this is possible. In real engineering effective effort only needs to be precise enough. Infinite precision would incur infinite costs.
Personally I don’t think that making an extremely powerful intelligence that is not sentient is moral. Actually I don’t think that it is possible. If its goal is to be friendly to humans it will need to model humans to a very deep level. This will of necessity include human recognition of and projection of agency and self toward . It will need to model itself in relation to the environment and its actions. How can an intelligence with vast understanding and modeling of all around it not model the one part of that environment that is itself? How can it not entertain various ways of looking at itself if it models other minds that do so? I think your notion of a transcendent mind without sentience is yet another impossible pipe dream at best. It could be much worse than that.
I don’t believe in unlimited power, certainly not as in the hands of humans, even very bright humans, that create the first AGI. There is no unlimited power and thus the impossible is not in our grasp. It is ridiculous to act as if it is. Either an AGI can be created or it can’t. Either it is a good idea to create it without sentience or it isn’t. Either this is possible or it is not. Either we can predict its future limiting parameters or we cannot. It seems you believe we or you or some very bright people somewhere have unlimited power to do whatever they decide to do.
Hmm, there are a lot of problems here.
“Unlimited power” is a non-starter. No matter how powerful the AGI is it will be of finite power. Unlimited power is the stuff of theology not of actually achievable minds. Thus the ditty from Epicurus about “God” does not apply. This is not a trivial point. I have a concern Eliezer may get too caught up in these grand sagas and great dilemnas on precisely such a theological absolutist scale. Arguing as if unlimited power is real takes us well into the current essay.
“Wieldable without side effects or configuration constraints” is more of the same. Imaginary thinking far beyond the constraints of any actually realizable situation. More theological daydreaming. It should go without saying that there is no such thing as operating without any configuration constraints and with perfect foresight of all possibly effects. I grow more concerned.
“It is wielded with unlimited precision”?! Come on, what is the game here? Surely you do not believe this is possible. In real engineering effective effort only needs to be precise enough. Infinite precision would incur infinite costs.
Personally I don’t think that making an extremely powerful intelligence that is not sentient is moral. Actually I don’t think that it is possible. If its goal is to be friendly to humans it will need to model humans to a very deep level. This will of necessity include human recognition of and projection of agency and self toward . It will need to model itself in relation to the environment and its actions. How can an intelligence with vast understanding and modeling of all around it not model the one part of that environment that is itself? How can it not entertain various ways of looking at itself if it models other minds that do so? I think your notion of a transcendent mind without sentience is yet another impossible pipe dream at best. It could be much worse than that.
I don’t believe in unlimited power, certainly not as in the hands of humans, even very bright humans, that create the first AGI. There is no unlimited power and thus the impossible is not in our grasp. It is ridiculous to act as if it is. Either an AGI can be created or it can’t. Either it is a good idea to create it without sentience or it isn’t. Either this is possible or it is not. Either we can predict its future limiting parameters or we cannot. It seems you believe we or you or some very bright people somewhere have unlimited power to do whatever they decide to do.