Daniel Kokotajlo comments on A shot at the diamond-alignment problem

[Daniel Kokotajlo]

1 is evidentially supported by the only known examples of general intelligences, but also AI will not have the same inductive biases. So the picture might be more complicated. I’d guess shard theory is still appropriate, but that’s ultimately a question for empirical work (with interpretability).^[12]

Shard theory seems more evidentially supported than bag-o-heuristics theory and rational agent theory, but that’s a pretty low bar! I expect a new theory to come along which is as much of an improvement over shard theory as shard theory is over those.

Re the 5 open questions: Yeah 4 and 5 seem like the hard ones to me.

Anyhow, in conclusion, nice work & I look forward to reading future developments. (Now I’ll go read the other comments)