Suppose the reward at each timestep is the number of paperclips the agent has.
At each timestep the agent has three “object-level” actions, and two shutdown-related actions:
Object-level:
use current resources to buy the paperclips available on the market
invest its resources in paperclip factories that will gradually make more paperclips at future timesteps
invest its resources in taking over the world to acquire more resources in future timesteps (with some risk that humans will notice and try to shut you down)
Shutdown-related:
Use resources to prevent a human shutdown attempt
Just shut yourself down, no human needed
For interesting behavior, suppose you’ve tuned the environment’s parameters so that there are different optimal strategies for different episode lengths (just buy paperclips at short timescales, build a paperclip factory at medium times, try to take over the world at long times).
Now you train this agent with DREST. What do you expect it to learn to do?
Suppose the reward at each timestep is the number of paperclips the agent has.
At each timestep the agent has three “object-level” actions, and two shutdown-related actions:
Object-level:
use current resources to buy the paperclips available on the market
invest its resources in paperclip factories that will gradually make more paperclips at future timesteps
invest its resources in taking over the world to acquire more resources in future timesteps (with some risk that humans will notice and try to shut you down)
Shutdown-related:
Use resources to prevent a human shutdown attempt
Just shut yourself down, no human needed
For interesting behavior, suppose you’ve tuned the environment’s parameters so that there are different optimal strategies for different episode lengths (just buy paperclips at short timescales, build a paperclip factory at medium times, try to take over the world at long times).
Now you train this agent with DREST. What do you expect it to learn to do?