I know that an organization isn’t a “superintelligence” (maybe it’s more of a scaled up human intelligence), but I think this kind of thing is a useful metaphor for what happens when a powerful intelligence maximizes a simple utility function. In this case, the utility function that was specified was “do the most good in the world.” There was no (as far as I can tell) malicious intent anywhere in the process. But even such a goal can result in manipulative or secretive behavior if the agent pursuing that goal follows it without regard to any other restrictions on behavior. My fear is that we’ll attribute these problems to corruption or malicious intent whereas I don’t think that’s the case here, and one of the reasons I don’t feel like the beginning of your essay involving Ponzi schemes is super appropriate.
Any computable intelligence can be considered scaled-up human intelligence, because humans are smart enough to follow arbitrary programs, just very slowly. Since many organizations can do a lot more than what a normal person can reasonably accomplish in a lifetime, I think it is appropriate to call them superintelligence.
I know that an organization isn’t a “superintelligence” (maybe it’s more of a scaled up human intelligence), but I think this kind of thing is a useful metaphor for what happens when a powerful intelligence maximizes a simple utility function. In this case, the utility function that was specified was “do the most good in the world.” There was no (as far as I can tell) malicious intent anywhere in the process. But even such a goal can result in manipulative or secretive behavior if the agent pursuing that goal follows it without regard to any other restrictions on behavior. My fear is that we’ll attribute these problems to corruption or malicious intent whereas I don’t think that’s the case here, and one of the reasons I don’t feel like the beginning of your essay involving Ponzi schemes is super appropriate.
Any computable intelligence can be considered scaled-up human intelligence, because humans are smart enough to follow arbitrary programs, just very slowly. Since many organizations can do a lot more than what a normal person can reasonably accomplish in a lifetime, I think it is appropriate to call them superintelligence.
So you do you consider yourself a scaled-up Turing Machine?