No such thing [as an AI society] for many (most?) possible AIs; just a monolithic maximizer.
We might attain universal cooperation—but it probably wouldn’t be terribly “monolithic” in the long term. It would be spread out over different planets and star systems. There would be some adaptation to local circumstances.
Could I become superintelligent under a Sysop?
The CEV document is littered with the term “human”, “humanity” and the “human species”—but without defining what they mean. It seems terribly unlikely that our distant descendants will classify themselves or each other as “humans”—except perhaps as a term of abuse. So: once all the “humans” are gone, what happens then?
Also, if a human can change into a superintelligence—and remain a valued person—why can’t a valued superintelligence be created from scratch? Is it because you were once DNA/protein you get special treatment? IMO, the future dominant organisms would see such views as appalling substrate chauvanism—what you are made of is an implementation detail, not who you really are. Is it because of who your ancestors were? That’s biblical morality—the seventh son of the seventh son, and all that. People will be judged for who they are, not for who they once were, long, long ago.
We might attain universal cooperation—but it probably wouldn’t be terribly “monolithic” in the long term. It would be spread out over different planets and star systems. There would be some adaptation to local circumstances.
The CEV document is littered with the term “human”, “humanity” and the “human species”—but without defining what they mean. It seems terribly unlikely that our distant descendants will classify themselves or each other as “humans”—except perhaps as a term of abuse. So: once all the “humans” are gone, what happens then?
Also, if a human can change into a superintelligence—and remain a valued person—why can’t a valued superintelligence be created from scratch? Is it because you were once DNA/protein you get special treatment? IMO, the future dominant organisms would see such views as appalling substrate chauvanism—what you are made of is an implementation detail, not who you really are. Is it because of who your ancestors were? That’s biblical morality—the seventh son of the seventh son, and all that. People will be judged for who they are, not for who they once were, long, long ago.