I find the Thompson hack very fascinating from an agent foundations perspective. It’s basically a small version of reflective stability in the context of operating systems.
I used to find compilers written in their own language kind of—…distasteful, in some way? Some of that is still present, because in reality it’s just that the bootstrapping chains become very long and difficult to follow. But I think a small part of that distaste was the worry that Thompson hack-style errors occur accidentally at some point, and are just propagated through the bootstrapping chain. After thinking about this for a few seconds this was of course patently ridiculous.
But under this lens reflective stability becomes really difficult, because every replicating/successor-generating subsystem needs to be adapted to have the property of reflective stability.
E.g. corrigibility is really hard if one imagines it as a type of Thompson hack, especially under relative robustness to scale. You don’t just get a basin of Thompson-hackness when writing compilers and making mistakes.
I find the Thompson hack very fascinating from an agent foundations perspective. It’s basically a small version of reflective stability in the context of operating systems.
I used to find compilers written in their own language kind of—…distasteful, in some way? Some of that is still present, because in reality it’s just that the bootstrapping chains become very long and difficult to follow. But I think a small part of that distaste was the worry that Thompson hack-style errors occur accidentally at some point, and are just propagated through the bootstrapping chain. After thinking about this for a few seconds this was of course patently ridiculous.
But under this lens reflective stability becomes really difficult, because every replicating/successor-generating subsystem needs to be adapted to have the property of reflective stability.
E.g. corrigibility is really hard if one imagines it as a type of Thompson hack, especially under relative robustness to scale. You don’t just get a basin of Thompson-hackness when writing compilers and making mistakes.
Apparently a Thompson-hack-like bug occurred in LLVM (haven’t read the post in detail yet). Interesting.