^ I think this might be helped by an example of the sort of ontological update you’d expect might be pretty challenging; I’m not sure that I have the same things in mind as you here
(I imagine one broad example is “What if AI discovers some new law of physics that we’re unaware of”, but it isn’t super clear to me how that specifically collides with value-alignment-y things?)
The existing ontology that we have around consciousness is pretty unclear. A better understanding the nature of consciousness and thus what’s valuable will likely come with new ontology.
When it comes to reasoning around statistics, robustness of judgements, causality, what it means not to Goodhart it’s likely that getting better at reasoning also means to come up with new ontology.
Very useful post! Thanks for writing it.
^ I think this might be helped by an example of the sort of ontological update you’d expect might be pretty challenging; I’m not sure that I have the same things in mind as you here
(I imagine one broad example is “What if AI discovers some new law of physics that we’re unaware of”, but it isn’t super clear to me how that specifically collides with value-alignment-y things?)
The existing ontology that we have around consciousness is pretty unclear. A better understanding the nature of consciousness and thus what’s valuable will likely come with new ontology.
When it comes to reasoning around statistics, robustness of judgements, causality, what it means not to Goodhart it’s likely that getting better at reasoning also means to come up with new ontology.