I tend to dismiss scenarios where it’s obvious, because I expect the demonstration of strong misaligned systems to inspire a strong multi-government response. Why do you expect this not to happen?
It occurs to me that the earliest demonstrations will be ambiguous to external parties, it will be one research org saying that something that doesn’t quite look strong enough to take over would do something if it were put in a position it’s not obvious it could get to, and then the message will spread incompletely, some will believe it, others wont, a moratorium wont be instated, and a condition of continuing to race in sin could take root.
But I doubt that ambiguous incidents like this would be reported to government at all? Private research orgs generally have a really good idea of what they can or can’t communicate to the outside world. Why cry wolf when you still can’t show them the wolf? People in positions of leadership in any sector are generally very good at knowing when to speak or not.
The most central scenario from my perspective is that there is massive competitive pressure and some amount of motivated denial.
It also might be relatively easily to paper over scheming which makes the motivated denial easier. Minimally, just ongoingly training against the examples of misbehaviour you’ve found might remove obvious misalignment.
(Obvious to whom might be an important question here.)
I tend to dismiss scenarios where it’s obvious, because I expect the demonstration of strong misaligned systems to inspire a strong multi-government respons
I think covid was clear-cut, and it did inspire some kind of government response, but not a particularly competent one.
Afaik there were not Generals saying “Covid could kill every one of us if we don’t control the situation” and controlling the situation would have required doing politically unpopular things rather than politically popular things.
Change either of those factors and it’s a completely different kind of situation.
I tend to dismiss scenarios where it’s obvious, because I expect the demonstration of strong misaligned systems to inspire a strong multi-government response. Why do you expect this not to happen?
It occurs to me that the earliest demonstrations will be ambiguous to external parties, it will be one research org saying that something that doesn’t quite look strong enough to take over would do something if it were put in a position it’s not obvious it could get to, and then the message will spread incompletely, some will believe it, others wont, a moratorium wont be instated, and a condition of continuing to race in sin could take root.
But I doubt that ambiguous incidents like this would be reported to government at all? Private research orgs generally have a really good idea of what they can or can’t communicate to the outside world. Why cry wolf when you still can’t show them the wolf? People in positions of leadership in any sector are generally very good at knowing when to speak or not.
The most central scenario from my perspective is that there is massive competitive pressure and some amount of motivated denial.
It also might be relatively easily to paper over scheming which makes the motivated denial easier. Minimally, just ongoingly training against the examples of misbehaviour you’ve found might remove obvious misalignment.
(Obvious to whom might be an important question here.)
I think covid was clear-cut, and it did inspire some kind of government response, but not a particularly competent one.
Afaik there were not Generals saying “Covid could kill every one of us if we don’t control the situation” and controlling the situation would have required doing politically unpopular things rather than politically popular things.
Change either of those factors and it’s a completely different kind of situation.