Just because a rational person would believe something, doesn’t mean a rational person would say that thing.
If telling people the fate of the world depends on you is going to make them less likely to listen, you probably shouldn’t tell them that. Especially if it’s true (because that just makes it more important that they listen)
FOOM is central to the argument that we need to solve Friendliness up front, rather than build it incrementally as patches to a slowly growing AGI. If you leave it out to get past weirdness censors, you can no longer support the same conclusions.
But it’s much easier to get people to look at an interesting problem so you can then persuade them that it’s serious, than it is to convince them that they are about to die in order to make them look at your problem.
Notice that Nihil didn’t propose never mentioning the urgency you believe exists, just not using it as your rallying cry.
I got fascinated by friendliness theorem despite never believing in the singularity (and in fact, not knowing much about the idea other than that it was being argued on the basis of extrapolating moore’s law, which explains why I didn’t buy it.)
Other people could be drawn in by the interesting philosophical and practical challenges of Friendliness Theorem without the FOOM threat.
It is more important to convince the AGI researchers who see themselves as practical people trying to achieve good results in the real world than people who like an interesting theoretical problem.
No. Because it is better for the people who would otherwise be working on dangerous AGI to realize they should not do that, than to have people who would not have been working on AI at all to comment that the dangerous AGI researchers shouldn’t do that.
Statements like this make posters look like they confuse rationality with the rejection of non-intuitive ideas.
Just because a rational person would believe something, doesn’t mean a rational person would say that thing.
If telling people the fate of the world depends on you is going to make them less likely to listen, you probably shouldn’t tell them that. Especially if it’s true (because that just makes it more important that they listen)
FOOM is central to the argument that we need to solve Friendliness up front, rather than build it incrementally as patches to a slowly growing AGI. If you leave it out to get past weirdness censors, you can no longer support the same conclusions.
NihilCredo said:
Notice that Nihil didn’t propose never mentioning the urgency you believe exists, just not using it as your rallying cry.
I got fascinated by friendliness theorem despite never believing in the singularity (and in fact, not knowing much about the idea other than that it was being argued on the basis of extrapolating moore’s law, which explains why I didn’t buy it.)
Other people could be drawn in by the interesting philosophical and practical challenges of Friendliness Theorem without the FOOM threat.
It is more important to convince the AGI researchers who see themselves as practical people trying to achieve good results in the real world than people who like an interesting theoretical problem.
Because people who like theoretical problems are less effective than people trying for good results? I don’t buy it.
No. Because it is better for the people who would otherwise be working on dangerous AGI to realize they should not do that, than to have people who would not have been working on AI at all to comment that the dangerous AGI researchers shouldn’t do that.