Yes. But when it’s being built by someone else in a not very transparent setting, what probability do you assign to its being generally-human-Friendly instead of particular-human-Friendly towards its creators? And how do you make sure its creators don’t commit fatal mistakes?
Suppose a group (foreign government, startup, etc) is building a singleton. They say it’s for the benefit of all mankind and other generally nice things. They won’t let you inspect it up close and don’t reveal their theories & tech, for fear of competition. Even if you offer to help, they say the value of your help is too small to risk your subverting their goals. You have the choice of:
Stopping them by force—use your biggest bombs, no questions asked.
Conquering them, trying to recover their technology, and using it to try to build your own singleton.
As above, but releasing all information and trying to make your singleton-building process 100% transparent and open to anyone who wants to participate.
Ignoring them.
Defending them against any attackers.
Edit: 6. Ignoring them but starting your own singleton project and trying to finish first. Variations: (6a) secret project, (6b) public but not transparent project, just like theirs, and (6c) transparent project inviting others to assist.
Anyone choosing 6 would have to present a very good reason for not choosing 2 or 3 instead, since competition is an obvious problem, winner takes all.
What do you do? I argue that most people/governments/armies will choose 1 or 2.
We are diverting a bit from ‘any’, ‘any other’ and ‘any kind’ here.
Well, I think any group would tend to consider any other group ‘not transparent and reliable enough’, given the extremely high stakes involved. A development effort that most people felt was open enough would be so open that there would effectively be no separate groups.
So, in any situation where the description of “one powerful group building a singleton and another group watching them” applies, I stand by my prediction.
How about if the figurehead made the effort to signal that he is too silly for a sophisticated body to consider a threat? After all, I’d sooner put plastic bottles of water on my roof than admit that some anime obsessed nerd was a bigger threat than me.
Yes. But when it’s being built by someone else in a not very transparent setting, what probability do you assign to its being generally-human-Friendly instead of particular-human-Friendly towards its creators? And how do you make sure its creators don’t commit fatal mistakes?
Suppose a group (foreign government, startup, etc) is building a singleton. They say it’s for the benefit of all mankind and other generally nice things. They won’t let you inspect it up close and don’t reveal their theories & tech, for fear of competition. Even if you offer to help, they say the value of your help is too small to risk your subverting their goals. You have the choice of:
Stopping them by force—use your biggest bombs, no questions asked.
Conquering them, trying to recover their technology, and using it to try to build your own singleton.
As above, but releasing all information and trying to make your singleton-building process 100% transparent and open to anyone who wants to participate.
Ignoring them.
Defending them against any attackers.
Edit: 6. Ignoring them but starting your own singleton project and trying to finish first. Variations: (6a) secret project, (6b) public but not transparent project, just like theirs, and (6c) transparent project inviting others to assist.
Anyone choosing 6 would have to present a very good reason for not choosing 2 or 3 instead, since competition is an obvious problem, winner takes all.
What do you do? I argue that most people/governments/armies will choose 1 or 2.
… We are diverting a bit from ‘any’, ‘any other’ and ‘any kind’ here.
And yes, the only credible alternative to 1 and 2 would be 4 in those cases where the agent predicts their own singleton to foom first.
Well, I think any group would tend to consider any other group ‘not transparent and reliable enough’, given the extremely high stakes involved. A development effort that most people felt was open enough would be so open that there would effectively be no separate groups.
So, in any situation where the description of “one powerful group building a singleton and another group watching them” applies, I stand by my prediction.
How about if the figurehead made the effort to signal that he is too silly for a sophisticated body to consider a threat? After all, I’d sooner put plastic bottles of water on my roof than admit that some anime obsessed nerd was a bigger threat than me.