Mercifully, you didn’t have the ability to make very deep changes. There are advantages to not being software.
The ability to change all aspects of oneself is not a property of software. Software can easily be made completely unable, partially able, or completely able to modify itself.
Fair enough, though evolved beings (which could include software) are probably less likely to be able to break themselves than designed beings capable of useful self-modification.
You know, you could say that software often has two parts: a crystalline part and a fluid part. Programs usually consist mostly of crystalline aspects: if I took a mathematical proof verifier and tweaked its axioms, even only a tiny bit, it would probably break completely. However, they often contain fluid aspects as well, such as the frequency at which the garbage collector should run, or eagerness to try a particular strategy over its alternative. If you change a fluid aspect of a program by a small amount, the program’s behavior might get a bit worse, but it definitely won’t end up being clobbered.
I’ve always thought that we should design Friendly AI like this. Only give it control over the fluid parts of itself, the parts of itself it can modify all it wants without damaging its (self-)honesty. Make the fluid parts powerful enough that if an insight occurs, the insight can be incorporated into the AI’s behavior somehow.
I’m sure that an AI will have more than two levels of internal stability. Some parts will be very stable (presumably, the use of logic and (we hope) Friendliness). Some parts will be very fluid (updating the immediate environment). There would be a big intermediate range of viscous-to-firm (general principles for dealing with people, how to improve its intelligence).
The ability to change all aspects of oneself is not a property of software. Software can easily be made completely unable, partially able, or completely able to modify itself.
Fair enough, though evolved beings (which could include software) are probably less likely to be able to break themselves than designed beings capable of useful self-modification.
You know, you could say that software often has two parts: a crystalline part and a fluid part. Programs usually consist mostly of crystalline aspects: if I took a mathematical proof verifier and tweaked its axioms, even only a tiny bit, it would probably break completely. However, they often contain fluid aspects as well, such as the frequency at which the garbage collector should run, or eagerness to try a particular strategy over its alternative. If you change a fluid aspect of a program by a small amount, the program’s behavior might get a bit worse, but it definitely won’t end up being clobbered.
I’ve always thought that we should design Friendly AI like this. Only give it control over the fluid parts of itself, the parts of itself it can modify all it wants without damaging its (self-)honesty. Make the fluid parts powerful enough that if an insight occurs, the insight can be incorporated into the AI’s behavior somehow.
I’m sure that an AI will have more than two levels of internal stability. Some parts will be very stable (presumably, the use of logic and (we hope) Friendliness). Some parts will be very fluid (updating the immediate environment). There would be a big intermediate range of viscous-to-firm (general principles for dealing with people, how to improve its intelligence).