A competitive system can use a very large number of human hours in the future, as long as it uses relatively few human hours today.
By “lack of philosophical understanding isn’t a big risk” I meant: “getting object-level philosophy questions wrong in the immediate future, like how to trade off speed vs. safety or how to compromise amongst different values, doesn’t seem to destroy too much value in expectation.” We may or may not need to solve philosophical problems to build aligned AGI. (I think Wei Dai believes that object-level philosophical errors destroy a lot of value in expectation.)
I think autopoietic is a useful category and captures half of what is interesting about “recursively self-improving AGI.” There is a slightly different economic concept, of automation that can be scaled up using fixed human inputs, without strongly diminishing returns. This would be relevant because it changes the character and pace of economic growth. It’s not clear whether this is equivalent to autopoiesis. For example, Elon Musk seems to hope for technology which is non-autopoeitic but has nearly the same transformative economic impact. (Your view in this post is similar to my best guess at Elon Musk’s view, though more clearly articulated / philosophically crisp.)
OK, it seems like I misinterpreted your comment on philosophy. But in this post you seem to be saying that we might not need to solve philosophical problems related to epistemology and agency?
That concept also seems useful and different from autopoiesis as I understand it (since it requires continual human cognitive work to run, though not very much).
I think that we can avoid coming up with a good decision theory or priors or so on—there are particular reasons that we might have had to solve philosophical problems, which I think we can dodge. But I agree that we need or want to solve some philosophical problems to align AGI (e.g. defining corrigibility precisely is a philosophical problem).
A competitive system can use a very large number of human hours in the future, as long as it uses relatively few human hours today.
By “lack of philosophical understanding isn’t a big risk” I meant: “getting object-level philosophy questions wrong in the immediate future, like how to trade off speed vs. safety or how to compromise amongst different values, doesn’t seem to destroy too much value in expectation.” We may or may not need to solve philosophical problems to build aligned AGI. (I think Wei Dai believes that object-level philosophical errors destroy a lot of value in expectation.)
I think autopoietic is a useful category and captures half of what is interesting about “recursively self-improving AGI.” There is a slightly different economic concept, of automation that can be scaled up using fixed human inputs, without strongly diminishing returns. This would be relevant because it changes the character and pace of economic growth. It’s not clear whether this is equivalent to autopoiesis. For example, Elon Musk seems to hope for technology which is non-autopoeitic but has nearly the same transformative economic impact. (Your view in this post is similar to my best guess at Elon Musk’s view, though more clearly articulated / philosophically crisp.)
That makes sense.
OK, it seems like I misinterpreted your comment on philosophy. But in this post you seem to be saying that we might not need to solve philosophical problems related to epistemology and agency?
That concept also seems useful and different from autopoiesis as I understand it (since it requires continual human cognitive work to run, though not very much).
I think that we can avoid coming up with a good decision theory or priors or so on—there are particular reasons that we might have had to solve philosophical problems, which I think we can dodge. But I agree that we need or want to solve some philosophical problems to align AGI (e.g. defining corrigibility precisely is a philosophical problem).