I think the second-setup still allows for powerful AGI that’s more explainable than current AI, in the same way that humans can kind of explain decisions to each other, but not very well at the level of neuroscience.
If something like natural abstractions are real, then this would get easier. I have a hard time not believing a weak version of this (e.g. human and AGI neuron structures could be totally different, but they’d both end up with some basic things like “the concept of 1”).
I think the second-setup still allows for powerful AGI that’s more explainable than current AI, in the same way that humans can kind of explain decisions to each other, but not very well at the level of neuroscience.
If something like natural abstractions are real, then this would get easier. I have a hard time not believing a weak version of this (e.g. human and AGI neuron structures could be totally different, but they’d both end up with some basic things like “the concept of 1”).