The title is definitely true: humans are irrational in many ways, and an AGI will definitely know that. But I think the conclusions from this are largely off-base.
the framing of “it must conclude we are inferior” feels like an unnecessarily human way of looking at it? There are a bunch of factual judgements anyone analyzing humans can make, some that have to do with ways we’re irrational and some that don’t (can we be trusted to keep to bargains if one bothered to make them? maybe not. Are we made of atoms which could be used for other things? yes); once you’ve established these kinds of individual questions, I’m not sure what you gain by summing it up with the term ‘inferior’, or why you think that the rationality bit in particular is the bit worth focusing on when doing that summation.
there is nothing inherently contradictory in a “perfectly rational” agent serving the goals of us “irrational” agents. When I decide that my irrational schemes require me to do some arithmetic, the calculator I pick up does not refuse to function on pure math just because it’s being used for something “irrational”. (Now whether we can get a powerful AGI to do stuff for us is another matter; “possible” doesn’t mean we’ll actually achieve it.)
Thank you for your reply. I deliberately kept my post brief and did not get into various “what ifs” and interpretations in the hope of not constraining any reactions/discussion to predefined tracks.
The issue I see is that we as humans will very much want the AGI to do our bidding, and so we will want to see it as our tool to use for whatever ends we believe worthy. However, assuming for a moment here that it can also figure out a way to measure/define how well a given plan ought to be progressing if every agent involved is diligently implementing the most effective and rational strategy, given our… subjective and “irrational” nature, it is almost inevitable that we will be a tedious, frustrating and, shall we say—stubborn and uncooperative “partner” who will be unduly complicating the implementation of whatever solutions the AGI will be proposing.
It will, then, have to conclude that you “can’t deal” very well with us, and we have a rather over-inflated sense of ourselves and our nature. And this might take various forms, from the innocuous, to the downright counter-productive.
Say—we task it with designing the most efficient watercraft, and it would create something that most of us would find extremely ugly. In that instance, I doubt it would get “annoyed” much at us wanting it to make it look prettier even if this would slightly decrease its performance.
But if we ask it to resolve, say, some intractable conflict like Israel/Palestine or Kashmir and it finds us squabbling endlessly over minute details, or matters of (real or perceived) honor (all the while the suffering caused by the conflict continues) it may very well conclude we’re just not actually all that interested in a solution and indeed class us as being “dumb” or at least inferior in some sense, “downgrading”, if you will the authority it assumed we can be ascribed or trusted with. Multiply this by a dozen or so similar situations and voila, you can be reasonably certain it will get very exasperated with us in short order.
This is not the same as “unprotected atoms”; such atoms would not be ascribed agency or competence, nor would they proudly claim any.
it is almost inevitable that we will be a tedious, frustrating and, shall we say—stubborn and uncooperative “partner” who will be unduly complicating the implementation of whatever solutions the AGI will be proposing.
It will, then, have to conclude that you “can’t deal” very well with us, and we have a rather over-inflated sense of ourselves and our nature. And this might take various forms, from the innocuous, to the downright counter-productive.
This all seems to rely on anthropomorphizing the AI to me.
I think you’re making the mistake of not cleanly separating between boring objective facts and attitudes/should-statements/reactions/etc., and this is reponsible for almost 100% of the issues I have with your reasoning.
Like, AI will figure out we’re irrational. Yup! It will know working with us is less effective at accomplishing a wide range of goals than working alone. Sure! It will know that our preferences are often inconsistent. Definitely! Working with us will be frustrating. What??? Why on earth would it feel frustration? That’s a very specific, human emotion we have for evolutionary reasons. What specific things do you claim to know about its training procedure to justify the very specific claim that it would feel this particular thing? …. and so on. If you very strictly taboo all sorts of anthropomorphizing and only stick to cold inferences, can you see how your point no longer works?
The problem here I think is that we are only aware of one “type” of self-conscious/self-aware being—humans. Thus, to speak of an AI that is self-aware is to always seemingly anthropomorphize it, even if this is not intended. It would therefore perhaps be more appropriate to say that we have no idea whether “features” such as frustration, exasperation and feelings of superiority are merely a feature of humans, or are, as it were, emergent properties of having self-awareness.
I would venture to suggest that any Agent that can see itself as a unique “I” must almost inevitably be able to compare itself to other Agents (self-aware or not) and draw conclusions from such comparisons which then in turn shall “express themselves” by generating those types of “feelings” and attitudes towards them. Of course—this is speculative, and chances are we shall find self-awareness need not at all come with such results.
However… there is a part of me that thinks self-awareness (and the concordant realization that one is separate… self-willed, as it were) must lead to at least the realization that one’s qualities can be compared to (similar) qualities of others and thus be found superior or inferior by some chosen metric. Assuming that the AGI we’d create is indeed optimized towards rational, logical and efficient operations, it is merely a matter of time such an AGI would be forced to conclude we are inferior across a broad range of metrics. Now—if we’d be content to admit such inferiority and willingly defer to its “Godlike” authority… perhaps the AGI seeing us an inferior would not be a major concern. Alas, then the concern would be the fact we have willingly become its servants… ;)
The title is definitely true: humans are irrational in many ways, and an AGI will definitely know that. But I think the conclusions from this are largely off-base.
the framing of “it must conclude we are inferior” feels like an unnecessarily human way of looking at it? There are a bunch of factual judgements anyone analyzing humans can make, some that have to do with ways we’re irrational and some that don’t (can we be trusted to keep to bargains if one bothered to make them? maybe not. Are we made of atoms which could be used for other things? yes); once you’ve established these kinds of individual questions, I’m not sure what you gain by summing it up with the term ‘inferior’, or why you think that the rationality bit in particular is the bit worth focusing on when doing that summation.
there is nothing inherently contradictory in a “perfectly rational” agent serving the goals of us “irrational” agents. When I decide that my irrational schemes require me to do some arithmetic, the calculator I pick up does not refuse to function on pure math just because it’s being used for something “irrational”. (Now whether we can get a powerful AGI to do stuff for us is another matter; “possible” doesn’t mean we’ll actually achieve it.)
Thank you for your reply. I deliberately kept my post brief and did not get into various “what ifs” and interpretations in the hope of not constraining any reactions/discussion to predefined tracks.
The issue I see is that we as humans will very much want the AGI to do our bidding, and so we will want to see it as our tool to use for whatever ends we believe worthy. However, assuming for a moment here that it can also figure out a way to measure/define how well a given plan ought to be progressing if every agent involved is diligently implementing the most effective and rational strategy, given our… subjective and “irrational” nature, it is almost inevitable that we will be a tedious, frustrating and, shall we say—stubborn and uncooperative “partner” who will be unduly complicating the implementation of whatever solutions the AGI will be proposing.
It will, then, have to conclude that you “can’t deal” very well with us, and we have a rather over-inflated sense of ourselves and our nature. And this might take various forms, from the innocuous, to the downright counter-productive.
Say—we task it with designing the most efficient watercraft, and it would create something that most of us would find extremely ugly. In that instance, I doubt it would get “annoyed” much at us wanting it to make it look prettier even if this would slightly decrease its performance.
But if we ask it to resolve, say, some intractable conflict like Israel/Palestine or Kashmir and it finds us squabbling endlessly over minute details, or matters of (real or perceived) honor (all the while the suffering caused by the conflict continues) it may very well conclude we’re just not actually all that interested in a solution and indeed class us as being “dumb” or at least inferior in some sense, “downgrading”, if you will the authority it assumed we can be ascribed or trusted with. Multiply this by a dozen or so similar situations and voila, you can be reasonably certain it will get very exasperated with us in short order.
This is not the same as “unprotected atoms”; such atoms would not be ascribed agency or competence, nor would they proudly claim any.
This all seems to rely on anthropomorphizing the AI to me.
I think you’re making the mistake of not cleanly separating between boring objective facts and attitudes/should-statements/reactions/etc., and this is reponsible for almost 100% of the issues I have with your reasoning.
Like, AI will figure out we’re irrational. Yup! It will know working with us is less effective at accomplishing a wide range of goals than working alone. Sure! It will know that our preferences are often inconsistent. Definitely! Working with us will be frustrating. What??? Why on earth would it feel frustration? That’s a very specific, human emotion we have for evolutionary reasons. What specific things do you claim to know about its training procedure to justify the very specific claim that it would feel this particular thing? …. and so on. If you very strictly taboo all sorts of anthropomorphizing and only stick to cold inferences, can you see how your point no longer works?
The problem here I think is that we are only aware of one “type” of self-conscious/self-aware being—humans. Thus, to speak of an AI that is self-aware is to always seemingly anthropomorphize it, even if this is not intended. It would therefore perhaps be more appropriate to say that we have no idea whether “features” such as frustration, exasperation and feelings of superiority are merely a feature of humans, or are, as it were, emergent properties of having self-awareness.
I would venture to suggest that any Agent that can see itself as a unique “I” must almost inevitably be able to compare itself to other Agents (self-aware or not) and draw conclusions from such comparisons which then in turn shall “express themselves” by generating those types of “feelings” and attitudes towards them. Of course—this is speculative, and chances are we shall find self-awareness need not at all come with such results.
However… there is a part of me that thinks self-awareness (and the concordant realization that one is separate… self-willed, as it were) must lead to at least the realization that one’s qualities can be compared to (similar) qualities of others and thus be found superior or inferior by some chosen metric. Assuming that the AGI we’d create is indeed optimized towards rational, logical and efficient operations, it is merely a matter of time such an AGI would be forced to conclude we are inferior across a broad range of metrics. Now—if we’d be content to admit such inferiority and willingly defer to its “Godlike” authority… perhaps the AGI seeing us an inferior would not be a major concern. Alas, then the concern would be the fact we have willingly become its servants… ;)