A wrinkle in the foom argument, re: source code readability
There is a sense in which a programme can easily read its own source code. The whole point of a compiler is to scan and process source code. A C compiler can compile its own source code, providing it is written in C.
The wrinkle is the ability of a programme to knowingly read its own source code. Any running process can be put inside a sandbox or simulated environment, such that there is no surely technical way if circumventing it. A running process accesses its environment using system calls, for instance get_time() or open_file(), and it has to take their results on faith. The get_time() function doesn’t have to return the real system time, and in a visualized process, attempts to access the file system do not access the real file system. There is no is_this_real() call, or at least, no unfakeable one. (Homoiconicity is no magic fix—even if a LISP programme can easily process it’s own code once it has obtained it, it still has to trust the subroutine that advertises itself as returning it)
Therefore, a programme can easily be prevented from accessing and modifying its own code.
It could be argued that an intelligent agent could use social engineering to cajole a human operator into getting the source code out of a locked cabinet, or whatever. This is a variation on the standard MIRI claim that an AI could talk it’s way out of a box. However,in this case the AI needs to talk it’s way out before it starts to recursively self improve, because it needs it’s source code. This suggests that an AI that is below a certain level of intelligence can be maintained there.
A wrinkle in the foom argument, re: source code readability
There is a sense in which a programme can easily read its own source code. The whole point of a compiler is to scan and process source code. A C compiler can compile its own source code, providing it is written in C.
The wrinkle is the ability of a programme to knowingly read its own source code. Any running process can be put inside a sandbox or simulated environment, such that there is no surely technical way if circumventing it. A running process accesses its environment using system calls, for instance get_time() or open_file(), and it has to take their results on faith. The get_time() function doesn’t have to return the real system time, and in a visualized process, attempts to access the file system do not access the real file system. There is no is_this_real() call, or at least, no unfakeable one. (Homoiconicity is no magic fix—even if a LISP programme can easily process it’s own code once it has obtained it, it still has to trust the subroutine that advertises itself as returning it)
Therefore, a programme can easily be prevented from accessing and modifying its own code.
It could be argued that an intelligent agent could use social engineering to cajole a human operator into getting the source code out of a locked cabinet, or whatever. This is a variation on the standard MIRI claim that an AI could talk it’s way out of a box. However,in this case the AI needs to talk it’s way out before it starts to recursively self improve, because it needs it’s source code. This suggests that an AI that is below a certain level of intelligence can be maintained there.
A good point, I must spend some time looking into the FOOM debate.