I wonder what the Pebblesorter AI would do if successfully programmed to implement Eliezer’s vision of coherent extrapolated volition:
“In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”
Would the AI pebblesort? Or would it figure that if the Pebblesorters got smarter, they would see that pebblesorting was pointless and arbitrary? Would the AI therefore adopt our own parochial morality, forbidding murder, theft and sexual intercourse among too-young people? Would that be the CEV of Pebblesorters?
I imagine we would all like to think so, but it smacks of parochialism, of objective morality. I can’t help thinking that Pebblesorter CEV would have to include some aspect of sorting pebbles. Doesn’t that suggest that CEV can malfunction pretty badly?
Well, if the PSFAI was the AI the Pebblesorters would have wanted to build, it would generally prevent murder of PS’s, because murder reduces the abilities of PS’s to sort pebbles. It would also sort pebbles into more and larger piles than ever before, because that is the core value that PS’s would want maximized. It would be able to see outside the algorithm that the PS run, and see that it was a primality-test function.
I wonder what the Pebblesorter AI would do if successfully programmed to implement Eliezer’s vision of coherent extrapolated volition:
“In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”
Would the AI pebblesort? Or would it figure that if the Pebblesorters got smarter, they would see that pebblesorting was pointless and arbitrary? Would the AI therefore adopt our own parochial morality, forbidding murder, theft and sexual intercourse among too-young people? Would that be the CEV of Pebblesorters?
I imagine we would all like to think so, but it smacks of parochialism, of objective morality. I can’t help thinking that Pebblesorter CEV would have to include some aspect of sorting pebbles. Doesn’t that suggest that CEV can malfunction pretty badly?
Well, if the PSFAI was the AI the Pebblesorters would have wanted to build, it would generally prevent murder of PS’s, because murder reduces the abilities of PS’s to sort pebbles. It would also sort pebbles into more and larger piles than ever before, because that is the core value that PS’s would want maximized. It would be able to see outside the algorithm that the PS run, and see that it was a primality-test function.