I will push back on democratic in the sense I think Linch is pushing the term being actually all that good a property for cosmically important orgs. See Bryan Caplan’s The Myth of the Rational Voter, and the literature around [Social-desirability bias](Social-desirability bias) for reasons why, which I’m sure Linch is familiar with, but I notice is not mentioned.
I also claim that most catastrophes through both recent and long-ago history have been caused by governments, not just in the trivial sense, but also if we normalize by amount of stuff done. A good example everyone right now should be familiar with is qualified immunity, and the effects it has on irresponsible policing. The fact is we usually hold our companies to much higher standards than our governments (or do we just have more control over the incentives of our companies than our governments). It is also strange that the example Linch gives for a bad company is Blackwater, which while bad, is… about par-for-the-course when it comes to CIA projects.
I note too the America-centric bias with all of these examples & comparisons. Maybe the American government is just too incompetent compared to others, and we should instead embed the project within France or Norway.
There’s a general narrative that basic research is best done in government/academia, but is this true? The academia end seems possibly true in the 20th century, most discoveries were made by academics. But there was also a significant contribution by folks at research labs started by monopolies of the period (most notably Bell Laboratories). Though this seems like the kind of thing which could turn out to be false going forward, as our universities become more bloated, and we kill off our monopolies. But in either case, I don’t know why Linch thinks quality basic research will be done by the government? People like bringing up the Apollo program & Manhattan project, but both of those were quality projects due to their applied research, not their basic research which was all laid down ahead of time. I’m not saying it doesn’t happen, but does anyone have good case studies? CERN comes to mind, but of course for projects that just require governments to throw massive amounts of money at a problem, government does well. AGI is plausibly like this, but alignment is not (though more money would be nice).
Government also tends to go slow, which I think is the strongest argument in favor of doing AGI inside a government. But also, man I don’t trust government to implement an alignment solution if such a solution is invented during the intervening time. I’m imagining trying to convince a stick-in-the-ass bureaucrat fancying himself a scientist philosopher, whose only contribution to the project was politicking at a few important senators to thereby get himself enough authority to stand in the way of anyone changing anything about the project, who thinks he knows the solution to alignment that he is in fact wrong, and he should use so-and-so proven strategy, or such-and-such ensemble approach instead. Maybe a cynical picture, but one I’d bet resonates with those working to improve government processes.
I’d be interested to hear how Austin has updated regarding Sam’s trustworthiness over the past few days.
Hello! Sorry for missing this comment the first time around :)
I will push back on democratic in the sense I think Linch is pushing the term being actually all that good a property for cosmically important orgs. See Bryan Caplan’s The Myth of the Rational Voter, and the literature around [Social-desirability bias](Social-desirability bias) for reasons why, which I’m sure Linch is familiar with, but I notice is not mentioned.
I definitely think this is a reasonable criticism. I think my overall response is the fairly trite Churchill quote “Democracy is the worst form of government, except all the others that have been tried.” I think broadly
a) monopoly on force is good and a historical advancement in many ways
b) (liberal, democratic) governments have done an okay job with the enormous responsibility that we have handed them.
c) I don’t think corporate actors should be given this much power
d) I think I want to separate out considerations of individuals’ moral goodness from what the incentives and institutions point someone towards.
di) I do think the typical OpenAI employees’ values are closer to mine, and they’re more competent than typical Americans, or typical gov’t bureaucrats
dii) OTOH I think the US gov’t has many checks and balances that private companies do not have (I think Leopold made a similar point in his most recent podcast).
relatedly:
The fact is we usually hold our companies to much higher standards than our governments
To the extent this is true, I think it’s because companies have many external checks on them (eg customers, competition, the government). I don’t think I’d be comfortable with corporate actors’ internal checks and balances (employees, their boards, etc) being nearly as strong as gov’ts’ internal checks.
e) I agree with you that democratic governments are heavily flawed. I just think it’s hard (far from impossible!) to do better, and I’m very skeptical that cosmically important organizations ought to be at what I facetiously refer to as the “forefront of corporate governance innovation.” While experiments in policy/governance innovation is very useful and necessary, I think we want to minimize the number of variables that could go wrong on our first few critical tries at doing something both cosmically important and very difficult. Governments in general, and the USG in particular, have been much more battle-tested re: handling important life and death situations, in a way that AI companies very much have not been. ---
I note too the America-centric bias with all of these examples & comparisons. Maybe the American government is just too incompetent compared to others, and we should instead embed the project within France or Norway.
I think my own preferred option is an intergovernmental operation like CERN, ruled by the UN Security Council or NATO or something. I have relatively little hope that the USG will let this happen however. And I have even less hope—vanishingly little—that the USG will be okay with a non-US governmental project in a more “competent” country like Norway or Singapore.
But if we wave aside the impracticality concerns, I’d also be worried about whether it’s strategically wise to locate an AGI project in a smaller/more “competent” government that’s less battle-tested than the US. On the object-level, I’d be very worried about information security concerns, where most of the smaller/more peacetime-competent governments might just not be robust to targeted hacks and cooption attempts (social and otherwise). On the meta-level, the lack of past experience with extreme outside pressure means we should be wary of them repeating their peacetime success “when shit hits the ceiling”, even if we can’t trace an exact causal mechanism for why.
most catastrophes through both recent and long-ago history have been caused by governments
Interesting lens! Though I’m not sure if this is fair—the largest things that are done tend to get done through governments, whether those things are good or bad. If you blame catastrophes like Mao’s famine or Hitler’s genocide on governments, you should also credit things like slavery abolition and vaccination and general decline of violence in civilized society to governments too.
I’d be interested to hear how Austin has updated regarding Sam’s trustworthiness over the past few days.
Hm I feel like a bunch of people have updated majorly negatively, but I haven’t—only small amounts. I think he eg gets credit for the ScarJo thing. I am mostly withholding judgement, though; now that the NDAs have been dropped, curious to see what comes to light (if nothing does, that would be more positive credit towards Sam, and some validation to my point that NDAs were not really concealing much).
Wait, to be clear, are you saying that you think it would be to Sam’s credit to learn that he forced employees to sign NDAs by straightforwardly lying to them about their legal obligations, using extremely adversarial time pressure tactics and making very intense but vague threats?
This behavior seems really obviously indefensible.
I don’t have a strong take on the ScarJo thing. I don’t really see how it would be to his credit, my guess is he straightforwardly lied about his intention to make the voice sound like ScarJo, but that’s of course very hard to verify, and it wouldn’t be a big deal either way IMO.
Sure, but the evidence is about the filtering that has occurred and how the filtering was conducted, not about what the filters were hiding. Threatening someone with violence to not insult you is bad independently of whether they had anything to insult you about.
What does “the evidence is about” mean? I don’t think there’s one piece of evidence, and I think evidence is normally relevant to multiple latent variables.
I agree that the fact there was filtering and how it was conducted is bad evidence. On the other hand, “now that the NDAs have been dropped, curious to see what comes to light (if nothing does, that would be more positive credit towards Sam [...])” seems to be talking about how the lack of something to insult Sam about is positive evidence about Sam. I don’t think it’s very strong evidence, fwiw, but noting that it is positive evidence seems pretty clearly distinct from saying “it would be to Sam’s credit to learn that he forced employees to sign NDAs by [bad stuff]”
Sure, but Austin answered the fully general question of “how [have you] updated regarding Sam’s trustworthiness over the past few days[?]” with “I haven’t updated majorly negatively”, in a generic tone.
When I say “the evidence is about the filtering” I am saying “the thing that seems like the obvious update would be about would be the filtering, not what the filtering was hiding”.
I agree that one can keep a separate ledger, but to not make a major negative update on Sam in the aggregate based on the information that was released requires IMO either that one already knew about such things and had the information integrated (which would presumably result in a low opinion of Sam’s conduct) or a distinct lack of moral compass (or third, a positive update that happened to mostly cancel out the negative update, though I think it would be confusing to communicate that via saying “I [updated] only small amounts”).
I have some feeling that this back-and-forth is bad or a waste of something, but I just don’t see how
[Austin is saying that] it would be to Sam’s credit to learn that he forced employees to sign NDAs by straightforwardly lying to them about their legal obligations, using extremely adversarial time pressure tactics and making very intense but vague threats?
is at all a plausible interpretation, or anything like a necessary implication, of what Austin wrote.
Ah, sure. I didn’t meant to say much about implying a large positive update here, and mostly intended to say “are you saying it’s not to any kind of substantial discredit here?”.
Interesting lens! Though I’m not sure if this is fair—the largest things that are done tend to get done through governments, whether those things are good or bad. If you blame catastrophes like Mao’s famine or Hitler’s genocide on governments, you should also credit things like slavery abolition and vaccination and general decline of violence in civilized society to governments too.
I do mostly[1] credit such things to governments, but the argument is about whether companies or governments are more liable to take on very large tail risks. Not about whether governments are generally good or bad. It may be that governments just like starting larger projects than corporations. But in that case, I think the claim that a greater percentage of those end in catastrophe than similarly large projects started by corporations still looks good.
I definitely don’t credit slavery abolition to governments, at least in America, since that industry was largely made possible in the first place by governments subsidizing the cost of chasing down runaway slaves. I’d guess general decline of violence is more attributable to generally increasing affluence, which has a range of factors associated with it, than government intervention so directly. But I’m largely ignorant on that particular subject. The “mostly” here means “I acknowledge governments do some good things”.
I will push back on democratic in the sense I think Linch is pushing the term being actually all that good a property for cosmically important orgs. See Bryan Caplan’s The Myth of the Rational Voter, and the literature around [Social-desirability bias](Social-desirability bias) for reasons why, which I’m sure Linch is familiar with, but I notice is not mentioned.
I also claim that most catastrophes through both recent and long-ago history have been caused by governments, not just in the trivial sense, but also if we normalize by amount of stuff done. A good example everyone right now should be familiar with is qualified immunity, and the effects it has on irresponsible policing. The fact is we usually hold our companies to much higher standards than our governments (or do we just have more control over the incentives of our companies than our governments). It is also strange that the example Linch gives for a bad company is Blackwater, which while bad, is… about par-for-the-course when it comes to CIA projects.
I note too the America-centric bias with all of these examples & comparisons. Maybe the American government is just too incompetent compared to others, and we should instead embed the project within France or Norway.
There’s a general narrative that basic research is best done in government/academia, but is this true? The academia end seems possibly true in the 20th century, most discoveries were made by academics. But there was also a significant contribution by folks at research labs started by monopolies of the period (most notably Bell Laboratories). Though this seems like the kind of thing which could turn out to be false going forward, as our universities become more bloated, and we kill off our monopolies. But in either case, I don’t know why Linch thinks quality basic research will be done by the government? People like bringing up the Apollo program & Manhattan project, but both of those were quality projects due to their applied research, not their basic research which was all laid down ahead of time. I’m not saying it doesn’t happen, but does anyone have good case studies? CERN comes to mind, but of course for projects that just require governments to throw massive amounts of money at a problem, government does well. AGI is plausibly like this, but alignment is not (though more money would be nice).
Government also tends to go slow, which I think is the strongest argument in favor of doing AGI inside a government. But also, man I don’t trust government to implement an alignment solution if such a solution is invented during the intervening time. I’m imagining trying to convince a stick-in-the-ass bureaucrat fancying himself a scientist philosopher, whose only contribution to the project was politicking at a few important senators to thereby get himself enough authority to stand in the way of anyone changing anything about the project, who thinks he knows the solution to alignment that he is in fact wrong, and he should use so-and-so proven strategy, or such-and-such ensemble approach instead. Maybe a cynical picture, but one I’d bet resonates with those working to improve government processes.
I’d be interested to hear how Austin has updated regarding Sam’s trustworthiness over the past few days.
Hello! Sorry for missing this comment the first time around :)
I definitely think this is a reasonable criticism. I think my overall response is the fairly trite Churchill quote “Democracy is the worst form of government, except all the others that have been tried.” I think broadly
a) monopoly on force is good and a historical advancement in many ways
b) (liberal, democratic) governments have done an okay job with the enormous responsibility that we have handed them.
c) I don’t think corporate actors should be given this much power
d) I think I want to separate out considerations of individuals’ moral goodness from what the incentives and institutions point someone towards.
di) I do think the typical OpenAI employees’ values are closer to mine, and they’re more competent than typical Americans, or typical gov’t bureaucrats
dii) OTOH I think the US gov’t has many checks and balances that private companies do not have (I think Leopold made a similar point in his most recent podcast).
relatedly:
To the extent this is true, I think it’s because companies have many external checks on them (eg customers, competition, the government). I don’t think I’d be comfortable with corporate actors’ internal checks and balances (employees, their boards, etc) being nearly as strong as gov’ts’ internal checks.
e) I agree with you that democratic governments are heavily flawed. I just think it’s hard (far from impossible!) to do better, and I’m very skeptical that cosmically important organizations ought to be at what I facetiously refer to as the “forefront of corporate governance innovation.” While experiments in policy/governance innovation is very useful and necessary, I think we want to minimize the number of variables that could go wrong on our first few critical tries at doing something both cosmically important and very difficult. Governments in general, and the USG in particular, have been much more battle-tested re: handling important life and death situations, in a way that AI companies very much have not been.
---
I think my own preferred option is an intergovernmental operation like CERN, ruled by the UN Security Council or NATO or something. I have relatively little hope that the USG will let this happen however. And I have even less hope—vanishingly little—that the USG will be okay with a non-US governmental project in a more “competent” country like Norway or Singapore.
But if we wave aside the impracticality concerns, I’d also be worried about whether it’s strategically wise to locate an AGI project in a smaller/more “competent” government that’s less battle-tested than the US. On the object-level, I’d be very worried about information security concerns, where most of the smaller/more peacetime-competent governments might just not be robust to targeted hacks and cooption attempts (social and otherwise). On the meta-level, the lack of past experience with extreme outside pressure means we should be wary of them repeating their peacetime success “when shit hits the ceiling”, even if we can’t trace an exact causal mechanism for why.
Interesting lens! Though I’m not sure if this is fair—the largest things that are done tend to get done through governments, whether those things are good or bad. If you blame catastrophes like Mao’s famine or Hitler’s genocide on governments, you should also credit things like slavery abolition and vaccination and general decline of violence in civilized society to governments too.
Hm I feel like a bunch of people have updated majorly negatively, but I haven’t—only small amounts. I think he eg gets credit for the ScarJo thing. I am mostly withholding judgement, though; now that the NDAs have been dropped, curious to see what comes to light (if nothing does, that would be more positive credit towards Sam, and some validation to my point that NDAs were not really concealing much).
Wait, to be clear, are you saying that you think it would be to Sam’s credit to learn that he forced employees to sign NDAs by straightforwardly lying to them about their legal obligations, using extremely adversarial time pressure tactics and making very intense but vague threats?
This behavior seems really obviously indefensible.
I don’t have a strong take on the ScarJo thing. I don’t really see how it would be to his credit, my guess is he straightforwardly lied about his intention to make the voice sound like ScarJo, but that’s of course very hard to verify, and it wouldn’t be a big deal either way IMO.
Austin is saying absence of evidence is evidence of absence (in the absence of a preempting filter)
Sure, but the evidence is about the filtering that has occurred and how the filtering was conducted, not about what the filters were hiding. Threatening someone with violence to not insult you is bad independently of whether they had anything to insult you about.
What does “the evidence is about” mean? I don’t think there’s one piece of evidence, and I think evidence is normally relevant to multiple latent variables.
I agree that the fact there was filtering and how it was conducted is bad evidence. On the other hand, “now that the NDAs have been dropped, curious to see what comes to light (if nothing does, that would be more positive credit towards Sam [...])” seems to be talking about how the lack of something to insult Sam about is positive evidence about Sam. I don’t think it’s very strong evidence, fwiw, but noting that it is positive evidence seems pretty clearly distinct from saying “it would be to Sam’s credit to learn that he forced employees to sign NDAs by [bad stuff]”
Sure, but Austin answered the fully general question of “how [have you] updated regarding Sam’s trustworthiness over the past few days[?]” with “I haven’t updated majorly negatively”, in a generic tone.
When I say “the evidence is about the filtering” I am saying “the thing that seems like the obvious update would be about would be the filtering, not what the filtering was hiding”.
I agree that one can keep a separate ledger, but to not make a major negative update on Sam in the aggregate based on the information that was released requires IMO either that one already knew about such things and had the information integrated (which would presumably result in a low opinion of Sam’s conduct) or a distinct lack of moral compass (or third, a positive update that happened to mostly cancel out the negative update, though I think it would be confusing to communicate that via saying “I [updated] only small amounts”).
I have some feeling that this back-and-forth is bad or a waste of something, but I just don’t see how
is at all a plausible interpretation, or anything like a necessary implication, of what Austin wrote.
OK, perhaps you are saying what I would phrase as “are you saying it’s not greatly to Sam’s discredit if he forced employees to sign …?”.
Ah, sure. I didn’t meant to say much about implying a large positive update here, and mostly intended to say “are you saying it’s not to any kind of substantial discredit here?”.
I do mostly[1] credit such things to governments, but the argument is about whether companies or governments are more liable to take on very large tail risks. Not about whether governments are generally good or bad. It may be that governments just like starting larger projects than corporations. But in that case, I think the claim that a greater percentage of those end in catastrophe than similarly large projects started by corporations still looks good.
I definitely don’t credit slavery abolition to governments, at least in America, since that industry was largely made possible in the first place by governments subsidizing the cost of chasing down runaway slaves. I’d guess general decline of violence is more attributable to generally increasing affluence, which has a range of factors associated with it, than government intervention so directly. But I’m largely ignorant on that particular subject. The “mostly” here means “I acknowledge governments do some good things”.
Whoa! Source?
I don’t know the exact article that convinced me, but I bet this summary of the history of economic thought on the subject is a good place to start, which I have skimmed, and seems to cover the main points with citations.