So, a number of issues stand out to me, some have been noted by others already, but:
My impression is that there are also less endorsable or less altruistic or more silly motives floating around for this attention allocation.
A lot of this list looks to me like the sort of heuristics where, societies that don’t follow them inevitably crash, burn and become awful. A list of famous questions where the obvious answer is horribly wrong, and there’s a long list of groups who came to the obvious conclusion and became awful, and it’s become accepted wisdom to not do that, except among the perpetually stubborn “It’ll be different this time” crowd, and doomers who insist “well, we just have to make it work this time, there’s no alternative”.
if anyone chooses to build, everything is destroyed
The problem with our current prisoner’s dilemma is that China has already openly declared their intentions. You’re playing against a defect bot. Also, your arguments are totally ineffective against them, because you’re not writing in Chinese. And, the opposition is openly malicious, and if alignment turns out to be easy, this ends with hell on earth, which is much worse than the false worst case of universal annihilation.
On the inevitability of AI: I find current attempts at AI alignment to be spaceships with sliderules silliness and not serious. Longer AI timelines are only useful if you can do something with the extra time. You’re missing necessary preconditions to both AI and alignment, and so long as those aren’t met, neither field is going to make any progress at all.
On qualia: I expect intelligence to be more interesting in general than the opposition expects. There are many ways to maximize paperclips, and even if technically, one path is actually correct, it’s almost impossible to produce sufficient pressure to direct a utility function directly at that. I expect an alien super intelligence that’s a 99.9999% perfect paperclip optimizer, and plays fun games on the side, to play above 99% of the quantity of games that a fun game optimizer would get. I accuse the opposition of bigotry towards aliens, and assert that the range of utility functions that produce positive outcomes is much larger than the opposition believes. Also, excluding all AI that would eliminate humanity, excludes lots of likable AI that would live good lives, but reach the obviously correct conclusion that humans are worse than them and need to go, while failing exclude any malicious AI that values human suffering.
On anthropics: We don’t actually experience the worlds that we fail to make interesting, so there’s no point worrying about them anyway. The only thing that actually matters is the utility ratio. It is granted that, if this worldline looked particularly heaven-oriented, and not hellish, it would be reasonable to maximize the amount of qualia attention by being protective of local reality, but just looking around me, that seems obviously not true.
On Existential Risk: I hold that the opposition massively underestimates current existential risks excluding AI, most of which AI is the solution to. The current environment is already fragile. Any stable evil government anywhere means that anything that sets back civilization threatens stagnation or worse, aka, every serious threat, even those that don’t immediately wipe out all life, most notably nuclear weapons, constitutes an existential risk. Propaganda and related can easily drive society into an irrecoverable position using current techniques. Genetics can easily wipe us out, and worse, in either direction. Become too fit, and we’re the ones maximizing paperclips. Alternatively, there’s the grow giant antlers and die problem where species trap themselves in a dysgenic spiral. Evolution does not have to be slow, and especially if social factors accelerate the divide between losers and winners, we could easily breed ourselves to oblivion in a few generations. Almost any technology could get us all killed. Super pathogens with a spread phase and a kill phases. Space technology that slightly adjusts the pathing of large objects. Very big explosions. Cheap stealth, guns that fire accurately across massive distances, fast transportation, easy ways to produce various poison gasses. There seems to be this idea that just because it isn’t exotic it won’t kill you.
In sum: I fully expect that this plan reduces the chances of long term survival of life, while also massively increasing the probability of artificial hell.
So, a number of issues stand out to me, some have been noted by others already, but:
A lot of this list looks to me like the sort of heuristics where, societies that don’t follow them inevitably crash, burn and become awful. A list of famous questions where the obvious answer is horribly wrong, and there’s a long list of groups who came to the obvious conclusion and became awful, and it’s become accepted wisdom to not do that, except among the perpetually stubborn “It’ll be different this time” crowd, and doomers who insist “well, we just have to make it work this time, there’s no alternative”.
The problem with our current prisoner’s dilemma is that China has already openly declared their intentions. You’re playing against a defect bot. Also, your arguments are totally ineffective against them, because you’re not writing in Chinese. And, the opposition is openly malicious, and if alignment turns out to be easy, this ends with hell on earth, which is much worse than the false worst case of universal annihilation.
On the inevitability of AI: I find current attempts at AI alignment to be spaceships with sliderules silliness and not serious. Longer AI timelines are only useful if you can do something with the extra time. You’re missing necessary preconditions to both AI and alignment, and so long as those aren’t met, neither field is going to make any progress at all.
On qualia: I expect intelligence to be more interesting in general than the opposition expects. There are many ways to maximize paperclips, and even if technically, one path is actually correct, it’s almost impossible to produce sufficient pressure to direct a utility function directly at that. I expect an alien super intelligence that’s a 99.9999% perfect paperclip optimizer, and plays fun games on the side, to play above 99% of the quantity of games that a fun game optimizer would get. I accuse the opposition of bigotry towards aliens, and assert that the range of utility functions that produce positive outcomes is much larger than the opposition believes. Also, excluding all AI that would eliminate humanity, excludes lots of likable AI that would live good lives, but reach the obviously correct conclusion that humans are worse than them and need to go, while failing exclude any malicious AI that values human suffering.
On anthropics: We don’t actually experience the worlds that we fail to make interesting, so there’s no point worrying about them anyway. The only thing that actually matters is the utility ratio. It is granted that, if this worldline looked particularly heaven-oriented, and not hellish, it would be reasonable to maximize the amount of qualia attention by being protective of local reality, but just looking around me, that seems obviously not true.
On Existential Risk: I hold that the opposition massively underestimates current existential risks excluding AI, most of which AI is the solution to. The current environment is already fragile. Any stable evil government anywhere means that anything that sets back civilization threatens stagnation or worse, aka, every serious threat, even those that don’t immediately wipe out all life, most notably nuclear weapons, constitutes an existential risk. Propaganda and related can easily drive society into an irrecoverable position using current techniques. Genetics can easily wipe us out, and worse, in either direction. Become too fit, and we’re the ones maximizing paperclips. Alternatively, there’s the grow giant antlers and die problem where species trap themselves in a dysgenic spiral. Evolution does not have to be slow, and especially if social factors accelerate the divide between losers and winners, we could easily breed ourselves to oblivion in a few generations. Almost any technology could get us all killed. Super pathogens with a spread phase and a kill phases. Space technology that slightly adjusts the pathing of large objects. Very big explosions. Cheap stealth, guns that fire accurately across massive distances, fast transportation, easy ways to produce various poison gasses. There seems to be this idea that just because it isn’t exotic it won’t kill you.
In sum: I fully expect that this plan reduces the chances of long term survival of life, while also massively increasing the probability of artificial hell.