Update: The setup described in the OP involves a system that models humans. See this MIRI article for a discussion on some important concerns about such systems.
Update: The setup described in the OP involves a system that models humans. See this MIRI article for a discussion on some important concerns about such systems.