Honestly, this whole tool/agent distinction seems tangential to me.
Consider two systems, S1 and S2.
S1 comprises the following elements:
a) a tool T, which when used by a person to achieve some goal G, can efficiently achieve G b) a person P, who uses T to efficiently achieve G.
S2 comprises a non-person agent A which achieves G efficiently.
I agree that A is an agent and T is not an agent, and I agree that T is a tool, and whether A is a tool seems a question not worth asking. But I don’t quite see why I should prefer S1 to S2.
Surely the important question is whether I endorse G?
Well, I certainly agree that both of those things are true.
And it might be that human-level evolved moral behavior is the best we can do… I don’t know. It would surprise me, but it might be true.
That said… given how unreliable such behavior is, if human-level evolved moral behavior even approximates the best we can do, it seems likely that I would do best to work towards neither T nor A ever achieving the level of optimizing power we’re talking about here.
Honestly, this whole tool/agent distinction seems tangential to me.
Consider two systems, S1 and S2.
S1 comprises the following elements: a) a tool T, which when used by a person to achieve some goal G, can efficiently achieve G
b) a person P, who uses T to efficiently achieve G.
S2 comprises a non-person agent A which achieves G efficiently.
I agree that A is an agent and T is not an agent, and I agree that T is a tool, and whether A is a tool seems a question not worth asking. But I don’t quite see why I should prefer S1 to S2.
Surely the important question is whether I endorse G?
A tool+human differs from a pure AI agent in two important ways:
The human (probably) already has naturally-evolved morality, sparing us the very hard problem of formalizing that.
We can arrange for (almost) everyone to have access to the tool, allowing tooled humans to counterbalance eachother.
Well, I certainly agree that both of those things are true.
And it might be that human-level evolved moral behavior is the best we can do… I don’t know. It would surprise me, but it might be true.
That said… given how unreliable such behavior is, if human-level evolved moral behavior even approximates the best we can do, it seems likely that I would do best to work towards neither T nor A ever achieving the level of optimizing power we’re talking about here.
Humanity isn’t that bad. Remember that the world we live in is pretty much the way humans made it, mostly deliberately.
But my main point was that existing humanity bypasses the very hard did-you-code-what-you-meant-to problem.
I agree with that point.