If I might take a crack at summarizing, it seems that you’ve realized the full scope of both the inner alignment and outer alignment problems. It’s pretty scary indeed. I think your insight that we don’t even need full AGI for these kinds of things to become large social problems is spot on and as an industry insider I’m super glad you’re seeing that now!
One other thing that I think is worth contemplating is how much computational irreducibility would actually come into play with respect to still having real-world power and impact. I don’t think you would need to get anywhere near perfect simulation in order to begin to have extremely good predictive power over the world. We’re already seeing this in graphics and physics modeling. We’re starting to be able to do virtual wind tunnel simulations that yield faster and better results than physical simulations,[1] and I think we’ll continue to find this to be the case. So presumably an advanced AGI would be able to create even better simulations that still work just-as-good in the real world and would forego much of the need for doing actual physical experimentation. Though I’m curious about what others think here!
Thanks for the comment, I’ll read some more on the distinction of inner and outer alignment, that sounds interesting.
I don’t think you would need to get anywhere near perfect simulation in order to begin to have extremely good predictive power over the world. We’re already seeing this in graphics and physics modeling.
I think this is a good point, although these are cases where lots of data is available. So I guess any case in which you don’t have the data ready would still have more difficulties. Off the top of my head I don’t know how limiting this would be in practice, but it should be in lots of cases.
If I might take a crack at summarizing, it seems that you’ve realized the full scope of both the inner alignment and outer alignment problems. It’s pretty scary indeed. I think your insight that we don’t even need full AGI for these kinds of things to become large social problems is spot on and as an industry insider I’m super glad you’re seeing that now!
One other thing that I think is worth contemplating is how much computational irreducibility would actually come into play with respect to still having real-world power and impact. I don’t think you would need to get anywhere near perfect simulation in order to begin to have extremely good predictive power over the world. We’re already seeing this in graphics and physics modeling. We’re starting to be able to do virtual wind tunnel simulations that yield faster and better results than physical simulations,[1] and I think we’ll continue to find this to be the case. So presumably an advanced AGI would be able to create even better simulations that still work just-as-good in the real world and would forego much of the need for doing actual physical experimentation. Though I’m curious about what others think here!
See here. Link to paper here.
Thanks for the comment, I’ll read some more on the distinction of inner and outer alignment, that sounds interesting.
I think this is a good point, although these are cases where lots of data is available. So I guess any case in which you don’t have the data ready would still have more difficulties. Off the top of my head I don’t know how limiting this would be in practice, but it should be in lots of cases.
I’m not so sure...another interesting/alarming thing is noting how these models are “grokking” concepts in a way that lets them generalize.