A program designed to answer a question necessarily wants to answer that question. A superintelligent program trying to answer that particular question runs the risk of acting as a paperclip maximizer.
What does that mean? It’s necessarily satisfying a utility function? It isn’t as Lumifer’s calculator shows.
Suppose you build a superintelligent program that is designed to make precise predictions, by being more creative and better at predictions than any human would. Why are you confident that one of the creative things this program does to make itself better at predictions isn’t turning the matter of the Earth into computronium as step 1?
I can be confident that nonagents wont’t do agentive things.
Why are you so confident your program is a nonagent? Do you have some formula for nonagent-ness? Do you have a program that you can feed some source code to and it will output whether that source code forms an agent or not?
Have you ever heard of someone designing a nonagentive programme that unexpectedly turned out to be agentive? Because to me that sounds like into the workshop to build a skateboard abd coming with a F1 car.
I’ve known plenty of cases where people’s programs were more agentive than they expected. And we don’t have a good track record on predicting which parts of what people do are hard for computers—we thought chess would be harder than computer vision, but the opposite turned out to be true.
I’ve known plenty of cases where people’s programs were more agentive than they expected.
“Doing something other than what the programmer expects” != “agentive”. An optimizer picking a solution that you did not consider is not being agentive.
What does that mean? It’s necessarily satisfying a utility function? It isn’t as Lumifer’s calculator shows.
I can be confident that nonagents wont’t do agentive things.
Why are you so confident your program is a nonagent? Do you have some formula for nonagent-ness? Do you have a program that you can feed some source code to and it will output whether that source code forms an agent or not?
It’s all standard software engineering.
I’m a professional software engineer, feel free to get technical.
Have you ever heard of someone designing a nonagentive programme that unexpectedly turned out to be agentive? Because to me that sounds like into the workshop to build a skateboard abd coming with a F1 car.
I’ve known plenty of cases where people’s programs were more agentive than they expected. And we don’t have a good track record on predicting which parts of what people do are hard for computers—we thought chess would be harder than computer vision, but the opposite turned out to be true.
I haven’t: have you any specific examples?
“Doing something other than what the programmer expects” != “agentive”. An optimizer picking a solution that you did not consider is not being agentive.