lukeprog’s and Louie Helm’s The Singularity and Machine Ethics draft suggests some useful directions for people wondering about their career choice. Some of them are psychology-related:
On the other hand, value extrapolation approaches to machine ethics face their own
challenges. Which value extrapolation algorithm should be used, and why? (Yudkowsky’s
“grown up farther together” provision looks especially vulnerable.) How can one extract a
coherent set of values from the complex valuation processes of the human brain, such that
this set of values can be extrapolated? Whose values should be extrapolated? How much will
values converge upon extrapolation (Sobel 1999; Döring and Andersen 2009)? Is the
extrapolation process computationally tractable, and can it be run without doing
unacceptable harm? How can extrapolated values be implemented in the goal system of a
machine, and how confident can we be that the machine will retain our values during
self-improvement?
These are difficult questions that demand investigation by experts in many different fields.
Neuroeconomists and other cognitive neuroscientists can continue to uncover how human
values are encoded and modified in the brain. Philosophers and mathematicians can develop
more sophisticated value extrapolation algorithms, building on the literature concerning
reflective equilibrium and “ideal preference” or “full information” theories of value.
Economists, neuroscientists, and AI researchers can extend current results in choice
modelling (Hess and Daly 2010) and preference elicitation (Domshlak et al. 2011) to extract
preferences from human behavior and brain activity. Decision theorists can work to develop a
decision theory that is capable of reasoning about decisions and values subsequent to
self-modification: a “reflective” decision theory.
lukeprog’s and Louie Helm’s The Singularity and Machine Ethics draft suggests some useful directions for people wondering about their career choice. Some of them are psychology-related: