is there at all a clean border between self-modification and simply learning things? We have “design” and “operation” at two places in our maps, but they can be easily mixed up in reality (is it OK to modify interpreted source code if we leave the interpreter alone? what about following verbal instructions then? inventing them? etc...)
Little consideration has been given to a block on self-modification because it seems that it is impossible. You could do a non-Von Neumann machine, separating data and code, but data can be interpreted as code.
Still, consideration should be given to whether anything can be done, even if only as stopgap.
Given that read-only hardware exists, yes, a clean border can be drawn, with the caveat that nothing is stopping the intelligence from emulating itself as if it were modified.
However—and it’s an important however—emulating your own modified code isn’t the same as modifying yourself. Just because you can imagine what your thought processes might be if you were sociopathic doesn’t make you sociopathic; just because an AI can emulate a process to arrive at a different answer than it would have doesn’t necessarily give it the power to -act- on that answer.
Which is to say, emulation can allow an AI to move past blocks on what it is permitted to think, but doesn’t necessarily permit it to move past blocks on what it is permitted to do.
This is particularly important in the case of something like a goal system; if a bug would result in an AI breaking its own goal system on a self-modification, this bug becomes less significant if the goal system is read-only. It could emulate what it would do with a different goal system, but it would be evaluating solutions from that emulation within its original goal system.
is there at all a clean border between self-modification and simply learning things? We have “design” and “operation” at two places in our maps, but they can be easily mixed up in reality (is it OK to modify interpreted source code if we leave the interpreter alone? what about following verbal instructions then? inventing them? etc...)
Little consideration has been given to a block on self-modification because it seems that it is impossible. You could do a non-Von Neumann machine, separating data and code, but data can be interpreted as code.
Still, consideration should be given to whether anything can be done, even if only as stopgap.
Given that read-only hardware exists, yes, a clean border can be drawn, with the caveat that nothing is stopping the intelligence from emulating itself as if it were modified.
However—and it’s an important however—emulating your own modified code isn’t the same as modifying yourself. Just because you can imagine what your thought processes might be if you were sociopathic doesn’t make you sociopathic; just because an AI can emulate a process to arrive at a different answer than it would have doesn’t necessarily give it the power to -act- on that answer.
Which is to say, emulation can allow an AI to move past blocks on what it is permitted to think, but doesn’t necessarily permit it to move past blocks on what it is permitted to do.
This is particularly important in the case of something like a goal system; if a bug would result in an AI breaking its own goal system on a self-modification, this bug becomes less significant if the goal system is read-only. It could emulate what it would do with a different goal system, but it would be evaluating solutions from that emulation within its original goal system.