@Alexander Gietelink Oldenziel, you put a soldier mindset react on this (and also my earlier, similar, comment this week).
What makes you think so?
Definitely this model posits that adversariality, but I don’t think that I’m invested in “my side” of the argument winning here, FWTIW. This currently seems like the most plausible high level summary of the situation, given my level of context.
Is there a version of this comment that would regard as better?
Yes sorry Eli, I meant to write out a more fully fleshed out response but unfortunately it got stuck in drafts.
The tl;dr is that I feel this perspective is singling out Sam Altman as some uniquely machiavellian actor in a way I find naive /misleading and ultimately maybe unhelpful.
I think in general im skeptical of the intense focus on individuals & individual tech companies that LW/EA has develloped recently. Frankly, it feels more rooted in savannah-brained tribalism & human interest than a evenkeeled analysis of what factors are actually important, neglected and tractable.
Frankly, it feels more rooted in savannah-brained tribalism & human interest than a evenkeeled analysis of what factors are actually important, neglected and tractable.
Um, I’m not attempting to do cause prioritization or action-planning in the above comment. More like sense-making. Before I move on to the question of what should we do, I want to have an accurate model of the social dynamics in the space.
(That said, it doesn’t seem a foregone conclusion that there are actionable things to do, that will come out of this analysis. If the above story is true, I should make some kind of update about the strategies that EAs adopted with regards to OpenAI in the late 2010s. Insofar as they were mistakes, I don’t want to repeat them.)
It might turn out to be right that the above story is “naive /misleading and ultimately maybe unhelpful”. I’m sure not an expert at understanding these dynamics. But just saying that it’s naive or that it seems rooted in tribalism doesn’t help me or others get a better model.
If it’s misleading, how is it misleading? (And is misleading different than “false”? Are you like “yeah this is technically correct, but it neglects key details”?)
Admittedly, you did label it as a tl;dr, and I did prompt you to elaborate on a react. So maybe it’s unfair of me to request even further elaboration.
I haven’t perceived the degree of focus as intense, and if I had I might be tempted to level similar criticism. But I think current people/companies do clearly matter some, so warrant some focus. For example:
I think it’s plausible that governments will be inclined to regulate AI companies more like “tech startups” than “private citizens building WMDs,” the more those companies strike them as “responsible,” earnestly trying their best, etc. In which case, it seems plausibly helpful to propagate information about how hard they are in fact trying, and how good their best is.
So far, I think many researchers who care non-trivially about alignment—and who might have been capable of helping, in nearby worlds—have for similar reasons been persuaded to join whatever AI company currently has the most safetywashed brand instead. This used to be OpenAI, is now Anthropic, and may be some other company in the future, but it seems useful to me to discuss the details of current examples regardless, in the hope that e.g. alignment discourse becomes better calibrated about how much to expect such hopes will yield.
There may exist some worlds where it’s possible to get alignment right, yet also possible not to, depending on the choices of the people involved. For example, you might imagine that good enough solutions—with low enough alignment taxes—do eventually exist, but that not all AI companies would even take the time to implement those.
Alternatively, you might imagine that some people who come to control powerful AI truly don’t care whether humanity survives, or are even explicitly trying to destroy it. I think such people are fairly common—both in the general population (relevant if e.g. powerful AI is open sourced), and also among folks currently involved with AI (e.g. Sutton, Page, Schmidhuber). Which seems useful to discuss, since e.g. one constraint on our survival is that those who actively wish to kill everyone somehow remain unable to do so.
I definitely understand the skepticism of intense focus on individuals/individual tech companies, but also, these are the groups trying to build the most consequential technology in all of history, so it’s natural that tech companies get the focus here.
@Alexander Gietelink Oldenziel, you put a soldier mindset react on this (and also my earlier, similar, comment this week).
What makes you think so?
Definitely this model posits that adversariality, but I don’t think that I’m invested in “my side” of the argument winning here, FWTIW. This currently seems like the most plausible high level summary of the situation, given my level of context.
Is there a version of this comment that would regard as better?
Yes sorry Eli, I meant to write out a more fully fleshed out response but unfortunately it got stuck in drafts.
The tl;dr is that I feel this perspective is singling out Sam Altman as some uniquely machiavellian actor in a way I find naive /misleading and ultimately maybe unhelpful.
I think in general im skeptical of the intense focus on individuals & individual tech companies that LW/EA has develloped recently. Frankly, it feels more rooted in savannah-brained tribalism & human interest than a evenkeeled analysis of what factors are actually important, neglected and tractable.
Um, I’m not attempting to do cause prioritization or action-planning in the above comment. More like sense-making. Before I move on to the question of what should we do, I want to have an accurate model of the social dynamics in the space.
(That said, it doesn’t seem a foregone conclusion that there are actionable things to do, that will come out of this analysis. If the above story is true, I should make some kind of update about the strategies that EAs adopted with regards to OpenAI in the late 2010s. Insofar as they were mistakes, I don’t want to repeat them.)
It might turn out to be right that the above story is “naive /misleading and ultimately maybe unhelpful”. I’m sure not an expert at understanding these dynamics. But just saying that it’s naive or that it seems rooted in tribalism doesn’t help me or others get a better model.
If it’s misleading, how is it misleading? (And is misleading different than “false”? Are you like “yeah this is technically correct, but it neglects key details”?)
Admittedly, you did label it as a tl;dr, and I did prompt you to elaborate on a react. So maybe it’s unfair of me to request even further elaboration.
yeahh i’m afraid I have too many other obligations right now to give a elaboration that does it justice.
otoh i’m in the Bay and we should definitely catch up sometime!
Fair enough!
Sounds good.
I haven’t perceived the degree of focus as intense, and if I had I might be tempted to level similar criticism. But I think current people/companies do clearly matter some, so warrant some focus. For example:
I think it’s plausible that governments will be inclined to regulate AI companies more like “tech startups” than “private citizens building WMDs,” the more those companies strike them as “responsible,” earnestly trying their best, etc. In which case, it seems plausibly helpful to propagate information about how hard they are in fact trying, and how good their best is.
So far, I think many researchers who care non-trivially about alignment—and who might have been capable of helping, in nearby worlds—have for similar reasons been persuaded to join whatever AI company currently has the most safetywashed brand instead. This used to be OpenAI, is now Anthropic, and may be some other company in the future, but it seems useful to me to discuss the details of current examples regardless, in the hope that e.g. alignment discourse becomes better calibrated about how much to expect such hopes will yield.
There may exist some worlds where it’s possible to get alignment right, yet also possible not to, depending on the choices of the people involved. For example, you might imagine that good enough solutions—with low enough alignment taxes—do eventually exist, but that not all AI companies would even take the time to implement those.
Alternatively, you might imagine that some people who come to control powerful AI truly don’t care whether humanity survives, or are even explicitly trying to destroy it. I think such people are fairly common—both in the general population (relevant if e.g. powerful AI is open sourced), and also among folks currently involved with AI (e.g. Sutton, Page, Schmidhuber). Which seems useful to discuss, since e.g. one constraint on our survival is that those who actively wish to kill everyone somehow remain unable to do so.
I definitely understand the skepticism of intense focus on individuals/individual tech companies, but also, these are the groups trying to build the most consequential technology in all of history, so it’s natural that tech companies get the focus here.