The core problem we face is ‘not enough information is transmitting fast enough, because people feel nervous about whether their private thoughts are in the Overton window’.
This applies to the AI x-safety community as well.
I have noticed a lot of difficulties myself in trying to convey new arguments to others, and those arguments are clearly outside the community’s Overton window.
In my case, I am not hesitant to reach out to people and have conversations. Like, if you are up for seriously listening, I am glad to have a call for an hour.
Two key bottlenecks seem to have been:
Busy and non-self-motivated: Most people are busily pursuing their own research or field-building projects.
They lack the bandwidth, patience or willingness to be disoriented about new arguments for a while, even if information value is high on a tiny prior probability of those arguments being true. Ie. when I say that these are arguments for why any (superficial) implementation of AGI-alignment would be impossible (to maintain over the long run).
Easily misinterpreted: In my one-on-one calls so far, researchers almost immediately clicked the terms and arguments I mentioned into their existing conceptual frameworks of alignment. Which I get, because those researchers incline toward and trained themselves to think like theoretical physicists, mathematicians or software engineers – not like linguistic anthropologists.
But the inferential gap is large, and I cannot convey how large without first conveying the arguments. People like to hold on to their own opinions and intuitions, or onto their general-not-specifically-informed skepticism, particularly when they are feeling disoriented and uncomfortable.
So I need to take >5x more time preempting and clearing up people’s common misinterpretations and the unsound premises of their quick-snap-response questions than if my conversation partners just tried listening from a beginner’s mind.
@Rob, I think you are right about us needing be straight about why we think certain arguments are important. However, in my case that is not a bottleneck to sharing information. The bottlenecks are around people spending little time to listen to and consider mentioned arguments step-by-step, in a way where they leave open the possibility that they are not yet interpreting the terms and arguments as meant.
An example, to be frank, was your recent response to my new proposal for a foreseeable alignment difficulty. I think you assumed a philosophical definition of ‘intentionality’ there and then interpreted the conclusion of my claims to be that AI ‘lackstrue intentionality’. That’s an inconsistent and maybe hasty reading of my comment, as the comment’s conclusion was that the intentionality in AI (ie. as having) will diverge from human intentionality.
I spent the last three weeks writing up a GDoc that I hope will not be misinterpreted in basic ways by you or other researchers (occasionally, interpretations are even literally incorrect – people literally say the opposite of part of a statement they heard or read from me a moment before).
It can be helpful if an interlocutor is straight-up about responding with their actual thoughts. I found out this way that some of my descriptive phrasing was way off – confusing and misdirecting the mental story people build up about the arguments.
But honestly, if the point is to have faster information transfers, can people not be a bit more careful about checking whether their interpretations are right? The difference for me is having had a chat with Rob three weeks ago and still taking another week right now asking for feedback from researchers I personally know to refine the writing even further.
Relatedly, a sense I got from reading posts and comments on LessWrong:
A common reason why a researcher ends up Strawmanning another researcher’s arguments or publicly claiming that another researcher is Strawmanning another researcher is that they never spend the time and attention (because they’re busy/overwhelmed?) to carefully read through the sentences written by the other researcher and consider how those sentences feel ambiguous in meaning and could be open to multiple interpretations.
When your interpretation of what (implicit) claim the author is arguing against seems particularly silly (even if the author plainly put a lot of effort into their writing), I personally think it’s often more productive (for yourself, other readers, and the author) to first paraphrase your interpretation back to the author.
That is, check in with the author in to clarify whether they meant y (your interpretation), before going on a long trail of explanations of why y does not match your interpretation of what the other AI x-safety researcher(s) meant when they wrote z.
This applies to the AI x-safety community as well.
I have noticed a lot of difficulties myself in trying to convey new arguments to others, and those arguments are clearly outside the community’s Overton window.
In my case, I am not hesitant to reach out to people and have conversations. Like, if you are up for seriously listening, I am glad to have a call for an hour.
Two key bottlenecks seem to have been:
Busy and non-self-motivated:
Most people are busily pursuing their own research or field-building projects.
They lack the bandwidth, patience or willingness to be disoriented about new arguments for a while, even if information value is high on a tiny prior probability of those arguments being true. Ie. when I say that these are arguments for why any (superficial) implementation of AGI-alignment would be impossible (to maintain over the long run).
Easily misinterpreted:
In my one-on-one calls so far, researchers almost immediately clicked the terms and arguments I mentioned into their existing conceptual frameworks of alignment. Which I get, because those researchers incline toward and trained themselves to think like theoretical physicists, mathematicians or software engineers – not like linguistic anthropologists.
But the inferential gap is large, and I cannot convey how large without first conveying the arguments. People like to hold on to their own opinions and intuitions, or onto their general-not-specifically-informed skepticism, particularly when they are feeling disoriented and uncomfortable.
So I need to take >5x more time preempting and clearing up people’s common misinterpretations and the unsound premises of their quick-snap-response questions than if my conversation partners just tried listening from a beginner’s mind.
@Rob, I think you are right about us needing be straight about why we think certain arguments are important. However, in my case that is not a bottleneck to sharing information. The bottlenecks are around people spending little time to listen to and consider mentioned arguments step-by-step, in a way where they leave open the possibility that they are not yet interpreting the terms and arguments as meant.
An example, to be frank, was your recent response to my new proposal for a foreseeable alignment difficulty. I think you assumed a philosophical definition of ‘intentionality’ there and then interpreted the conclusion of my claims to be that AI ‘lacks true intentionality’. That’s an inconsistent and maybe hasty reading of my comment, as the comment’s conclusion was that the intentionality in AI (ie. as having) will diverge from human intentionality.
I spent the last three weeks writing up a GDoc that I hope will not be misinterpreted in basic ways by you or other researchers (occasionally, interpretations are even literally incorrect – people literally say the opposite of part of a statement they heard or read from me a moment before).
It can be helpful if an interlocutor is straight-up about responding with their actual thoughts. I found out this way that some of my descriptive phrasing was way off – confusing and misdirecting the mental story people build up about the arguments.
But honestly, if the point is to have faster information transfers, can people not be a bit more careful about checking whether their interpretations are right? The difference for me is having had a chat with Rob three weeks ago and still taking another week right now asking for feedback from researchers I personally know to refine the writing even further.
Relatedly, a sense I got from reading posts and comments on LessWrong:
A common reason why a researcher ends up Strawmanning another researcher’s arguments or publicly claiming that another researcher is Strawmanning another researcher is that they never spend the time and attention (because they’re busy/overwhelmed?) to carefully read through the sentences written by the other researcher and consider how those sentences feel ambiguous in meaning and could be open to multiple interpretations.
When your interpretation of what (implicit) claim the author is arguing against seems particularly silly (even if the author plainly put a lot of effort into their writing), I personally think it’s often more productive (for yourself, other readers, and the author) to first paraphrase your interpretation back to the author.
That is, check in with the author in to clarify whether they meant y (your interpretation), before going on a long trail of explanations of why y does not match your interpretation of what the other AI x-safety researcher(s) meant when they wrote z.