I don’t think the connotations of “silly” are quite right here. You could still use this program to do quite a lot of useful inference and optimization across a variety of domains, without killing everyone. Sort of like how frequentist statistics can be very accurate in some cases despite being suboptimal by Bayesian standards. Bostrom mostly only talks about agent-like AIs, and while I think that this is mostly the right approach, he should have been more explicit about that. As I said before, we don’t currently know how to build agent-like AGIs at the moment because we haven’t solved the ontology mapping problem, but we do know how to build non-agentlike cross-domain optimizers given enough computation power.
Okay. We seem to be disputing definitions here. By your definition, it is totally possible to build a very good cross-domain optimizer without it being an agent (so it doesn’t optimize a utility function over the universe). It seems like we mostly agree on matters of fact.
I don’t think the connotations of “silly” are quite right here. You could still use this program to do quite a lot of useful inference and optimization across a variety of domains, without killing everyone. Sort of like how frequentist statistics can be very accurate in some cases despite being suboptimal by Bayesian standards. Bostrom mostly only talks about agent-like AIs, and while I think that this is mostly the right approach, he should have been more explicit about that. As I said before, we don’t currently know how to build agent-like AGIs at the moment because we haven’t solved the ontology mapping problem, but we do know how to build non-agentlike cross-domain optimizers given enough computation power.
I don’t see how being able to using a non-agent program to do useful things means it’s not silly to say it has a utility function. It’s not an agent.
Okay. We seem to be disputing definitions here. By your definition, it is totally possible to build a very good cross-domain optimizer without it being an agent (so it doesn’t optimize a utility function over the universe). It seems like we mostly agree on matters of fact.