First, if there is a real danger that the first AGI is going to go foom and control everything, then having fewer people look at the code makes it more likely that something will go wrong.
That is indeed a problem, but it is nowhere near as bad as a public AGI getting forked by people who don’t know what they are doing.
The polished cryptographic functions that benifeted from all those eyeballs were not the first in their code history to be executed. For cryptographic systems that don’t ruin the universe if slightly wrong, that is ok, but for AGI that is very bad.
First, if there is a real danger that the first AGI is going to go foom and control everything, then having fewer people look at the code makes it more likely that something will go wrong.
That is indeed a problem, but it is nowhere near as bad as a public AGI getting forked by people who don’t know what they are doing.
It seems to me that forking is common practice in the open source arena. It rarely causes major problems—and is sometimes very healthy if a dominant project starts going in a direction that people don’t like.
A bad fork rarely destroys the world in the open source arena.
Right—and that is very unlikely in the future too, I reckon. You typically need marketing, support infrastructure, social contacts, etc. to get ahead. Most forks don’t have that. “Bad” forks are especially unlikely to succeed—and good forks we are OK with.
We don’t try and stop the mafia using powerful IT tools—like EMACS. We realise that is not a practical reason for keeping such power secret.
That is indeed a problem, but it is nowhere near as bad as a public AGI getting forked by people who don’t know what they are doing.
The polished cryptographic functions that benifeted from all those eyeballs were not the first in their code history to be executed. For cryptographic systems that don’t ruin the universe if slightly wrong, that is ok, but for AGI that is very bad.
It seems to me that forking is common practice in the open source arena. It rarely causes major problems—and is sometimes very healthy if a dominant project starts going in a direction that people don’t like.
A bad fork rarely destroys the world in the open source arena.
Right—and that is very unlikely in the future too, I reckon. You typically need marketing, support infrastructure, social contacts, etc. to get ahead. Most forks don’t have that. “Bad” forks are especially unlikely to succeed—and good forks we are OK with.
We don’t try and stop the mafia using powerful IT tools—like EMACS. We realise that is not a practical reason for keeping such power secret.