Recent work on mesa-optimisation is relevant to the discussion of search IMO: see the paper Risks from Learned Optimization. Basically, the idea is that your search procedure finds a thing that’s an agent, and it’s not obvious that the agent it finds has the same goals as the search process.
Recent work on mesa-optimisation is relevant to the discussion of search IMO: see the paper Risks from Learned Optimization. Basically, the idea is that your search procedure finds a thing that’s an agent, and it’s not obvious that the agent it finds has the same goals as the search process.