First, if there is a real danger that the first AGI is going to go foom and control everything, then having fewer people look at the code makes it more likely that something will go wrong.
That is indeed a problem, but it is nowhere near as bad as a public AGI getting forked by people who don’t know what they are doing.
It seems to me that forking is common practice in the open source arena. It rarely causes major problems—and is sometimes very healthy if a dominant project starts going in a direction that people don’t like.
A bad fork rarely destroys the world in the open source arena.
Right—and that is very unlikely in the future too, I reckon. You typically need marketing, support infrastructure, social contacts, etc. to get ahead. Most forks don’t have that. “Bad” forks are especially unlikely to succeed—and good forks we are OK with.
We don’t try and stop the mafia using powerful IT tools—like EMACS. We realise that is not a practical reason for keeping such power secret.
It seems to me that forking is common practice in the open source arena. It rarely causes major problems—and is sometimes very healthy if a dominant project starts going in a direction that people don’t like.
A bad fork rarely destroys the world in the open source arena.
Right—and that is very unlikely in the future too, I reckon. You typically need marketing, support infrastructure, social contacts, etc. to get ahead. Most forks don’t have that. “Bad” forks are especially unlikely to succeed—and good forks we are OK with.
We don’t try and stop the mafia using powerful IT tools—like EMACS. We realise that is not a practical reason for keeping such power secret.