Ah, but my idea is different! It’s not just that these two AIs will physically merge. I claim that two AIs that are able to communicate are already indistinguishable from one AI with a different utility function. I reject the entire concept of meaningfully counting AIs.
There is a trivial idea that two humans together form a kind of single agent. This agent is not a human (there are too many conditions for being a human), and it might not be very smart (if the humans’ goals don’t align).
Now consider the same idea for two superintelligent AIs. I claim that the “combined” mind is also superintelligent, and it acts as though its utility function was a combination of the two initial utility functions. There are only complications from the possibly distributed physical architecture of the AI.
To take it even further, I claim that given any two AIs called A and B, if they together would choose strategy S, then there also exists a single AI called M(A,B), that would also choose strategy S. If we take the paperclip and staple maximizers, they might physically merge (or they might just randomly destroy one of them?). Now I claim that there is another single AI, with a slightly funky but reasonable architecture, which would be rewarded both for 60% staples and for 60% paperclips, and that this AI would choose to construct a new AI with a more coherent utility function (or it would choose to self modify to make its own utility coherent).
Also, thank you for digging for the old threads. It’s frustrating that there is so much out there that I would never know to even look for.
Edit: damn, I think the second link basically has the same idea as well.
I think if you carefully read everything in these links and let it stew for a bit, you’ll get something like my approach.
More generally, having ideas is great but don’t stop there! Always take the next step, make things slightly more precise, push a little bit past the point where you have everything figured out. That way you’re almost guaranteed to find new territory soon enough. I have an old post about that.
Ah, but my idea is different! It’s not just that these two AIs will physically merge. I claim that two AIs that are able to communicate are already indistinguishable from one AI with a different utility function. I reject the entire concept of meaningfully counting AIs.
There is a trivial idea that two humans together form a kind of single agent. This agent is not a human (there are too many conditions for being a human), and it might not be very smart (if the humans’ goals don’t align).
Now consider the same idea for two superintelligent AIs. I claim that the “combined” mind is also superintelligent, and it acts as though its utility function was a combination of the two initial utility functions. There are only complications from the possibly distributed physical architecture of the AI.
To take it even further, I claim that given any two AIs called A and B, if they together would choose strategy S, then there also exists a single AI called M(A,B), that would also choose strategy S. If we take the paperclip and staple maximizers, they might physically merge (or they might just randomly destroy one of them?). Now I claim that there is another single AI, with a slightly funky but reasonable architecture, which would be rewarded both for 60% staples and for 60% paperclips, and that this AI would choose to construct a new AI with a more coherent utility function (or it would choose to self modify to make its own utility coherent).
Also, thank you for digging for the old threads. It’s frustrating that there is so much out there that I would never know to even look for.
Edit: damn, I think the second link basically has the same idea as well.
I think if you carefully read everything in these links and let it stew for a bit, you’ll get something like my approach.
More generally, having ideas is great but don’t stop there! Always take the next step, make things slightly more precise, push a little bit past the point where you have everything figured out. That way you’re almost guaranteed to find new territory soon enough. I have an old post about that.