This should result in a massive update of the priors towards “the future will have lots of things that humans like”.
Actually, I think this sentence needs a bit of refinement. After all, the people at the beginning of the graph, and the middle, and indeed most of the graph, are all dead. The things that exist in the future are ‘lots of things that the most capable descendants of the previous generations liked.’ And, what is a descendant if not ‘an independent, intelligent entity with its own values and goals’. I think any AGI we create counts as a descendant for the purposes of the extrapolation of this graph. I am pretty confident that the future will have a lot of value according to our most capable and intelligent descendants by the definition which includes our creations. That doesn’t cheer me up much.
Actually, I think this sentence needs a bit of refinement. After all, the people at the beginning of the graph, and the middle, and indeed most of the graph, are all dead. The things that exist in the future are ‘lots of things that the most capable descendants of the previous generations liked.’ And, what is a descendant if not ‘an independent, intelligent entity with its own values and goals’. I think any AGI we create counts as a descendant for the purposes of the extrapolation of this graph. I am pretty confident that the future will have a lot of value according to our most capable and intelligent descendants by the definition which includes our creations. That doesn’t cheer me up much.