I’ve been watching the discussion on Realism About Rationality with some interest and surprise. I had thought of ‘something like realism about rationality’ as more cruxy for alignment work, because the inspectability of the AI matters a lot more than the inspectability of your own mind – mostly because you’re going to scale up the AI a lot more than your own mind is likely to scale up. The amount of disagreement that’s come out more recently about that has been interesting.
Some of the people who seem most invested in the Coherent Agency thing are specifically trying to operate on cosmic scales (i.e. part of their goal is to capture value in other universes and simulations, and to be the sort of person you could safely upload).
Upon reflection though, I guess it’s not surprising that people don’t consider realism “cruxy” for alignment, and also not “cruxy” for personal agency (i.e. upon reflection, I think it’s more like an aesthetic input, than a crux. It’s not necessary for agency to be mathematically simple or formalized, for incremental legibility and coherence to be useful for avoiding wasted motion)
Relatedly:
I’ve been watching the discussion on Realism About Rationality with some interest and surprise. I had thought of ‘something like realism about rationality’ as more cruxy for alignment work, because the inspectability of the AI matters a lot more than the inspectability of your own mind – mostly because you’re going to scale up the AI a lot more than your own mind is likely to scale up. The amount of disagreement that’s come out more recently about that has been interesting.
Some of the people who seem most invested in the Coherent Agency thing are specifically trying to operate on cosmic scales (i.e. part of their goal is to capture value in other universes and simulations, and to be the sort of person you could safely upload).
Upon reflection though, I guess it’s not surprising that people don’t consider realism “cruxy” for alignment, and also not “cruxy” for personal agency (i.e. upon reflection, I think it’s more like an aesthetic input, than a crux. It’s not necessary for agency to be mathematically simple or formalized, for incremental legibility and coherence to be useful for avoiding wasted motion)