I quite appreciate that you’re up front about the distinction between “we shipped things we like” and “we made progress on alignment”. Some of the stuff I’ve read (janus’s posts, infohazard policy) has been pretty thoughtful (even while I disagreed with some of it) and I was definitely worried that the work would be much worse. I recall my main recommendation to you when you visited the Bay (and my offices) was “publish your ideas so I and others can evaluate them”, and I’m definitely glad you’ve done a bunch of that. And I agree that you can do even more :)
You publishing products sounds potentially like the beginning of the end, I’m not sure. Everyone should not really be contributing to AI hype or proving how profitable ML can be. I think maybe there’s ways to do it ethically if your product line isn’t at all branded as ML and doesn’t have the possibility of revolutionizing a whole field (e.g. kickstarting dozens of competitors over the next 2 decades and creating 10s of billions of dollars), just a small market that you can own and gets you the funding you need and no more. Also, seems plausible it will take like 3-5 years of focused effort to become clearly profitable, killing your organizational focus on alignment for that period. I am not actually sure what I would do here, if rich folks like OpenPhil aren’t into funding you for whatever reason (I don’t know).
I’ll mention here that from what little I saw, your coordination efforts seemed a bit misguided and harmful. I am left with a sense that you wanted everyone to be friends and not feel threatened by interacting with others, to feel like “come in, the water is fine, don’t worry, you won’t end up with people criticizing you for maybe ending civilization or self-deceiving along the way or call you unethical”. While I am pretty open to fairly respectful coordination and am a strong fan of finding positive-sum trade, I care more about being frank and honest in my interactions, and a route must be found where communicating such things (insofar as that’s what someone believes) isn’t going to destroy or end the coordination/trade agreement. If you suspect you are the only people in a position to speak the truth about what is going on, you must not trade that away or else nobody will be able to say what is going on, and any remaining hope of successful reacting to what is actually happening is likely lost.
In order to signal that I am not trying to just be critical, I will end on a true positive note, which is that overall, from my current vantage point, your team has been far more competent and focused on alignment than I expected. It’s like my… 88th percentile outcome, in that I assigned 87% to you being less competent on the alignment problem. I am positively surprised :)
I’ll mention here that from what little I saw, your coordination efforts seemed a bit misguided and harmful. I am left with a sense that you wanted everyone to be friends and not feel threatened by interacting with others, to feel like “come in, the water is fine, don’t worry, you won’t end up with people criticizing you for maybe ending civilization or self-deceiving along the way or call you unethical”. While I am pretty open to fairly respectful coordination and am a strong fan of finding positive-sum trade, I care more about being frank and honest in my interactions, and a route must be found where communicating such things (insofar as that’s what someone believes) isn’t going to destroy or end the coordination/trade agreement. Speaking the truth is not something to be traded away, however costly it may be.
I can’t comment on Conjecture specifically’s coordination efforts, but I fairly strongly disagree with this as a philosophy of coordination. There exist a lot of people in the world who have massive empirical or ethical disagreements with me that lead to them taking actions I think are misguided to actively harmful to extremely dangerous. But I think that this often is either logical or understandable from their perspective. I think that being able to communicate productively with these people. see things from their point of view, and work towards common ground is a valuable skill, and an important part of the spirit of cooperation. For example, I think that Leah Garces’s work cooperating with chicken farmers to reduce factory farming is admirable and worthwhile, and I imagine she isn’t always frank and honest with people.
In particular, I think that being frank and honest in this context can basically kill possible cooperation. And good cooperation can lead to things being better by everyone’s lights, so this is a large and important cost not worth taking lightly. Not everyone has to strive for cooperation, but I think it’s very important that at least some people do! I do think that being so cooperative that you lose track of what you personally believe can be misguided and corrosive, but that there’s a big difference between having clear internal beliefs and needing to express all of those beliefs.
Agreed, the releases I’ve seen from Conjecture has made me incredibly hopeful of what they can achieve and interested in their approach to the problem more generally.
When it comes to the coordination efforts, I’m generally of the opinion that we can speak the truth while being inviting and ease people into the arguments. From my chats with Conjecture, it seems like their method is not “come in, the water is fine” but “come in, here are the reasons why the water is boiling and might turn into lava”.
If we start by speaking the truth “however costly it may be”and this leads to them being turned off by alignment, we have not actually introduced them to truth but have achieved the opposite. I’m left with a sense that their coordination efforts are quite promising and follow this line of thinking, though this might be a knowledge gap from my side (I know of 3-5 of their coordination efforts). I’m looking forward to seeing more posts about this work, though.
As per the product side, the goal is not to be profitable fast, it is to be attractive to non-alignment investors (e.g. AirBnB is not profitable). I agree with the risks, of course. I can see something like Loom branching out (haha) as a valuable writing tool with medium risk. Conjecture’s corporate structure and core team alignment seems to be quite protective of a product branch being positive, though money always have unintended consequences.
I have been a big fan of the “new agendas” agenda and I look forward to read about their unified research agenda! The candidness of their contribution to the alignment problem has also updated me positively on Conjecture and the organization-building startup costs seems invaluable and inevitable. Godspeed.
Speaking the truth is not something to be traded away, however costly it may be.
Stated without clarification, this is not something I’d say categorically and I suspect neither would you. As the classic example goes, you would probably lie to the Space Gestapo to prevent them from killing your mother.
Yeah, I spoke too broadly there. I’ve replaced it with a different and more specific claim for now:
If you suspect you are the only people in a position to speak the truth about what is going on, you must not trade that away or else nobody will be able to say what is going on, and any remaining hope of successful reacting to what is actually happening is likely lost.
Like, in this situation, the idea is trading away the ability for AI safety folks to speak the truth in order to be allowed just to openly talk to some people in power. Such a weak reward for such a great loss.
I quite appreciate that you’re up front about the distinction between “we shipped things we like” and “we made progress on alignment”. Some of the stuff I’ve read (janus’s posts, infohazard policy) has been pretty thoughtful (even while I disagreed with some of it) and I was definitely worried that the work would be much worse. I recall my main recommendation to you when you visited the Bay (and my offices) was “publish your ideas so I and others can evaluate them”, and I’m definitely glad you’ve done a bunch of that. And I agree that you can do even more :)
You publishing products sounds potentially like the beginning of the end, I’m not sure. Everyone should not really be contributing to AI hype or proving how profitable ML can be. I think maybe there’s ways to do it ethically if your product line isn’t at all branded as ML and doesn’t have the possibility of revolutionizing a whole field (e.g. kickstarting dozens of competitors over the next 2 decades and creating 10s of billions of dollars), just a small market that you can own and gets you the funding you need and no more. Also, seems plausible it will take like 3-5 years of focused effort to become clearly profitable, killing your organizational focus on alignment for that period. I am not actually sure what I would do here, if rich folks like OpenPhil aren’t into funding you for whatever reason (I don’t know).
I’ll mention here that from what little I saw, your coordination efforts seemed a bit misguided and harmful. I am left with a sense that you wanted everyone to be friends and not feel threatened by interacting with others, to feel like “come in, the water is fine, don’t worry, you won’t end up with people criticizing you for maybe ending civilization or self-deceiving along the way or call you unethical”. While I am pretty open to fairly respectful coordination and am a strong fan of finding positive-sum trade, I care more about being frank and honest in my interactions, and a route must be found where communicating such things (insofar as that’s what someone believes) isn’t going to destroy or end the coordination/trade agreement. If you suspect you are the only people in a position to speak the truth about what is going on, you must not trade that away or else nobody will be able to say what is going on, and any remaining hope of successful reacting to what is actually happening is likely lost.
In order to signal that I am not trying to just be critical, I will end on a true positive note, which is that overall, from my current vantage point, your team has been far more competent and focused on alignment than I expected. It’s like my… 88th percentile outcome, in that I assigned 87% to you being less competent on the alignment problem. I am positively surprised :)
I can’t comment on Conjecture specifically’s coordination efforts, but I fairly strongly disagree with this as a philosophy of coordination. There exist a lot of people in the world who have massive empirical or ethical disagreements with me that lead to them taking actions I think are misguided to actively harmful to extremely dangerous. But I think that this often is either logical or understandable from their perspective. I think that being able to communicate productively with these people. see things from their point of view, and work towards common ground is a valuable skill, and an important part of the spirit of cooperation. For example, I think that Leah Garces’s work cooperating with chicken farmers to reduce factory farming is admirable and worthwhile, and I imagine she isn’t always frank and honest with people.
In particular, I think that being frank and honest in this context can basically kill possible cooperation. And good cooperation can lead to things being better by everyone’s lights, so this is a large and important cost not worth taking lightly. Not everyone has to strive for cooperation, but I think it’s very important that at least some people do! I do think that being so cooperative that you lose track of what you personally believe can be misguided and corrosive, but that there’s a big difference between having clear internal beliefs and needing to express all of those beliefs.
Thanks for the link, I’ll aim to give that podcast a listen, it’s relevant to a bunch of my current thinking.
Agreed, the releases I’ve seen from Conjecture has made me incredibly hopeful of what they can achieve and interested in their approach to the problem more generally.
When it comes to the coordination efforts, I’m generally of the opinion that we can speak the truth while being inviting and ease people into the arguments. From my chats with Conjecture, it seems like their method is not “come in, the water is fine” but “come in, here are the reasons why the water is boiling and might turn into lava”.
If we start by speaking the truth “however costly it may be” and this leads to them being turned off by alignment, we have not actually introduced them to truth but have achieved the opposite. I’m left with a sense that their coordination efforts are quite promising and follow this line of thinking, though this might be a knowledge gap from my side (I know of 3-5 of their coordination efforts). I’m looking forward to seeing more posts about this work, though.
As per the product side, the goal is not to be profitable fast, it is to be attractive to non-alignment investors (e.g. AirBnB is not profitable). I agree with the risks, of course. I can see something like Loom branching out (haha) as a valuable writing tool with medium risk. Conjecture’s corporate structure and core team alignment seems to be quite protective of a product branch being positive, though money always have unintended consequences.
I have been a big fan of the “new agendas” agenda and I look forward to read about their unified research agenda! The candidness of their contribution to the alignment problem has also updated me positively on Conjecture and the organization-building startup costs seems invaluable and inevitable. Godspeed.
Stated without clarification, this is not something I’d say categorically and I suspect neither would you. As the classic example goes, you would probably lie to the Space Gestapo to prevent them from killing your mother.
Yeah, I spoke too broadly there. I’ve replaced it with a different and more specific claim for now:
Like, in this situation, the idea is trading away the ability for AI safety folks to speak the truth in order to be allowed just to openly talk to some people in power. Such a weak reward for such a great loss.