It’s not just his fiction. Recently he went on what he thought was a low-stakes crypto podcast and was surprised that the hosts wanted to actually hear him out when he said we were all going to die soon:
I don’t think we can take this as evidence that Yudkowsky or the average rationalist “underestimates more average people”. In the Bankless podcast, Eliezer was not trying to do anything like trying to explore the beliefs of the podcast hosts, just explaining his views. And there have been attempts at outreach before. If Bankless was evidence towards “the world at large is interested in Eliezer’s ideas and takes them seriously”, The Alignment Problem and Human Compatible and rejection of FDT from academic decision theory journals is stronger evidence against. It seems to me that the lesson we should gather is that alignment’s time in the public consciousness has come sometime in the last ~6 months.
I’m also not sure the techniques are asymmetric.
Have people with false beliefs tried e.g. Street Epistemology and found it to fail?
I think few of us in the alignment community are actually in a position to change our minds about whether alignment is worth working on. With a p(doom) of ~35% I think it’s unlikely that arguments alone push me below the ~5% threshold where working on AI misuse, biosecurity, etc. become competitive with alignment. And there are people with p(doom) of >85%.
That said it seems likely that rationalists should be incredibly embarrassed for not realizing the potential asymmetric weapons in things like Street Epistemology. I’d make a Manifold market for it, but I can’t think of a good operationalization.
I think few of us in the alignment community are actually in a position to change our minds about whether alignment is worth working on. With a p(doom) of ~35% I think it’s unlikely that arguments alone push me below the ~5% threshold where working on AI misuse, biosecurity, etc. become competitive with alignment. And there are people with p(doom) of >85%.
I have changed my mind and now think some of the core arguments for x-risk don’t go through, so it’s plausible that I go below 5% if there is continued success in alignment-related ML fields and could substantially change my mind from a single conversation.
I think few of us in the alignment community are actually in a position to change our minds about whether alignment is worth working on. With a p(doom) of ~35% I think it’s unlikely that arguments alone push me below the ~5% threshold where working on AI misuse, biosecurity, etc. become competitive with alignment. And there are people with p(doom) of >85%.
This makes little sense to me, since “what should I do” isn’t a function of p(doom). It’s a function of both p(doom) and your inclinations, opportunities, and comparative advantages. There should be many people for whom, rationally speaking, a difference between 35% and 34% should change their ideal behavior.
Thanks, I agree. I would still make the weaker claim that more than half the people in alignment are very unlikely to change their career prioritization from Street Epistemology-style conversations, and that in general the person with more information / prior exposure to the arguments will be less likely to change their mind.
I don’t think we can take this as evidence that Yudkowsky or the average rationalist “underestimates more average people”. In the Bankless podcast, Eliezer was not trying to do anything like trying to explore the beliefs of the podcast hosts, just explaining his views. And there have been attempts at outreach before. If Bankless was evidence towards “the world at large is interested in Eliezer’s ideas and takes them seriously”, The Alignment Problem and Human Compatible and rejection of FDT from academic decision theory journals is stronger evidence against. It seems to me that the lesson we should gather is that alignment’s time in the public consciousness has come sometime in the last ~6 months.
I’m also not sure the techniques are asymmetric.
Have people with false beliefs tried e.g. Street Epistemology and found it to fail?
I think few of us in the alignment community are actually in a position to change our minds about whether alignment is worth working on. With a p(doom) of ~35% I think it’s unlikely that arguments alone push me below the ~5% threshold where working on AI misuse, biosecurity, etc. become competitive with alignment. And there are people with p(doom) of >85%.
That said it seems likely that rationalists should be incredibly embarrassed for not realizing the potential asymmetric weapons in things like Street Epistemology. I’d make a Manifold market for it, but I can’t think of a good operationalization.
I have changed my mind and now think some of the core arguments for x-risk don’t go through, so it’s plausible that I go below 5% if there is continued success in alignment-related ML fields and could substantially change my mind from a single conversation.
This makes little sense to me, since “what should I do” isn’t a function of p(doom). It’s a function of both p(doom) and your inclinations, opportunities, and comparative advantages. There should be many people for whom, rationally speaking, a difference between 35% and 34% should change their ideal behavior.
Thanks, I agree. I would still make the weaker claim that more than half the people in alignment are very unlikely to change their career prioritization from Street Epistemology-style conversations, and that in general the person with more information / prior exposure to the arguments will be less likely to change their mind.