Another point about “defection” is which action is a defection with respect to whom.
Sam Altman is the leader of an organization with a real chance of bringing about the literal end of the world, and I find any and all information about his thoughts and his organization to be of the highest interest for the rest of humanity.
Not disclosing whatever such information ones comes into contact with, except in case of speeding up potentially even-less-alignment-focused competitors, is a defection against the rest of us.
If this were an off-the-record meeting with a head of state discussing plans for expanding and/or deploying nuclear weapons capabilities, nobody would dare suggest taking it down, inaccuracy and incompleteness notwithstanding.
Now, Sam Altman appeals emotionally to a lot of us (me included) as much more relatable, being an apparently prosocial nerdy tech guy, but in my opinion he’s a head state in control of WMDs and should be (and expect to be) treated as such.
Update: honestly curious about reasons for downvotes if anyone is willing to share. I have no intention to troll or harm the discussion and am willing to adapt writing style. Thank you.
The point of the defection/cooperation thing isn’t just that cooperation is a kindness to Sam Altman personally, which can be overridden by the greater good. The point is that generally cooperative behavior, and generally high amounts of trust, can make everyone better off. If it was true as you said that:
he’s a head state in control of WMDs and should be (and expect to be) treated as such
and as a consequence, he e.g. expected someone to record him during the Q&A, then he would presumably not have done the Q&A in the first place, or would have shared much less information. This would have lead to humanity learning less about OpenAI’s plans.
And this is definitely not a single-shot interaction. This was Sam’s second Q&A at an ACX meetup, and there was no reason to expect it to be the last. Moreover, there’s been a lot of interactions between the alignment community (including some who write on LW) and OpenAI in the past. And given that OpenAI’s decisions about alignment-related things matter a lot (as you say) it seems important to keep up good relations and high degrees of trust.
honestly curious about reasons for downvotes if anyone is willing to share
I initially downvoted, have since retracted it. Since trust/cooperation can be quite fragile and dependent on expectations about how people will behave, I became worried when I read you as basically announcing that you and other people should and will defect in the future. And I wanted a quick way to mark disagreement with that, to communicate that this isn’t a generally accepted point of view. But your point was phrased in a perfectly civil manner, so I should really have just taken the time to write a response, sorry.
The point about it being an iterated prisoner’s dilemma is a good one, and I would rather there be more such ACX instances where he shares even more of his thinking due to our cooperative/trustworthy behavior, than this to be the last one or have the next ones be filtered PR-speak.
A small number of people in the alignment community repeatedly getting access to better information and being able to act on it beats the value of this one single post staying open to the world. And even in the case of “the cat being out of the bag,” hiding/removing the post would probably do good as a gesture of cooperation.
Another point about “defection” is which action is a defection with respect to whom.
Sam Altman is the leader of an organization with a real chance of bringing about the literal end of the world, and I find any and all information about his thoughts and his organization to be of the highest interest for the rest of humanity.
Not disclosing whatever such information ones comes into contact with, except in case of speeding up potentially even-less-alignment-focused competitors, is a defection against the rest of us.
If this were an off-the-record meeting with a head of state discussing plans for expanding and/or deploying nuclear weapons capabilities, nobody would dare suggest taking it down, inaccuracy and incompleteness notwithstanding.
Now, Sam Altman appeals emotionally to a lot of us (me included) as much more relatable, being an apparently prosocial nerdy tech guy, but in my opinion he’s a head state in control of WMDs and should be (and expect to be) treated as such.
Update: honestly curious about reasons for downvotes if anyone is willing to share. I have no intention to troll or harm the discussion and am willing to adapt writing style. Thank you.
The point of the defection/cooperation thing isn’t just that cooperation is a kindness to Sam Altman personally, which can be overridden by the greater good. The point is that generally cooperative behavior, and generally high amounts of trust, can make everyone better off. If it was true as you said that:
and as a consequence, he e.g. expected someone to record him during the Q&A, then he would presumably not have done the Q&A in the first place, or would have shared much less information. This would have lead to humanity learning less about OpenAI’s plans.
And this is definitely not a single-shot interaction. This was Sam’s second Q&A at an ACX meetup, and there was no reason to expect it to be the last. Moreover, there’s been a lot of interactions between the alignment community (including some who write on LW) and OpenAI in the past. And given that OpenAI’s decisions about alignment-related things matter a lot (as you say) it seems important to keep up good relations and high degrees of trust.
I initially downvoted, have since retracted it. Since trust/cooperation can be quite fragile and dependent on expectations about how people will behave, I became worried when I read you as basically announcing that you and other people should and will defect in the future. And I wanted a quick way to mark disagreement with that, to communicate that this isn’t a generally accepted point of view. But your point was phrased in a perfectly civil manner, so I should really have just taken the time to write a response, sorry.
I appreciate the response and stand corrected.
The point about it being an iterated prisoner’s dilemma is a good one, and I would rather there be more such ACX instances where he shares even more of his thinking due to our cooperative/trustworthy behavior, than this to be the last one or have the next ones be filtered PR-speak.
A small number of people in the alignment community repeatedly getting access to better information and being able to act on it beats the value of this one single post staying open to the world. And even in the case of “the cat being out of the bag,” hiding/removing the post would probably do good as a gesture of cooperation.