Humans rarely optimize, or even approximate optimization. And, I think we only get as close as we do to optimal because we have had many years of evolution honing our motivations. An AI would be created much more quickly and in a larger mindspace. Well intended criteria can go awry and often do. A moral example of this would be the Repugnant Conclusion, and a logical example would be Newcomb’s Paradox.
In those cases, we look at the anticipated outcomes of certain actions and judge those outcomes to be flawed, and so we can revisit our assumptions and tweak our criteria. But if the UGC is true, this tactic will not work in complex cases where there are many constraints. We have nothing to rely on to avoid subtler but equally suboptimal outcomes except our own intuitions about satisficing criteria, even though these intuitions are very weak. Simply hoping that no such subtle difficulties exist seems like a bad plan to me.
I don’t think the difficulties in adapting our moral intuitions into a self-consistent formal system (e.g. the Repugnant Conclusion) is a problem of insufficient optimisation power per se, its more the case that there are multiple systems within the brain (morality, intuitions, logic) and these are not operating in perfect harmony. This doesn’t mean that each individual system is not working ok in its own way.
Surely designing an engine, or writing a novel, or (… you get the gist) are complex optimising problems with many constraints, and yet it seems that humans can at least approximately solve these problems far faster than brute-force trial and error.
AFAICT the UGC was saying that there exist insoluble problems, not that these problems are actually common. It seems to me like Godel’s incompleteness theorem - there are statements which are true but which cannot be proved, but this doesn’t mean that no statement can be proved, or that mathematics is pointless. At the end of the day regardless of whether or not the fundamental underpinning of mathematics are on shaky ground, or whether there are unprovble theorems and unsolvable problems, the actual mathematics that allows us to build aeroplanes works, and the planes fly.
Humans rarely optimize, or even approximate optimization. And, I think we only get as close as we do to optimal because we have had many years of evolution honing our motivations. An AI would be created much more quickly and in a larger mindspace. Well intended criteria can go awry and often do. A moral example of this would be the Repugnant Conclusion, and a logical example would be Newcomb’s Paradox.
In those cases, we look at the anticipated outcomes of certain actions and judge those outcomes to be flawed, and so we can revisit our assumptions and tweak our criteria. But if the UGC is true, this tactic will not work in complex cases where there are many constraints. We have nothing to rely on to avoid subtler but equally suboptimal outcomes except our own intuitions about satisficing criteria, even though these intuitions are very weak. Simply hoping that no such subtle difficulties exist seems like a bad plan to me.
I don’t think the difficulties in adapting our moral intuitions into a self-consistent formal system (e.g. the Repugnant Conclusion) is a problem of insufficient optimisation power per se, its more the case that there are multiple systems within the brain (morality, intuitions, logic) and these are not operating in perfect harmony. This doesn’t mean that each individual system is not working ok in its own way.
Surely designing an engine, or writing a novel, or (… you get the gist) are complex optimising problems with many constraints, and yet it seems that humans can at least approximately solve these problems far faster than brute-force trial and error.
AFAICT the UGC was saying that there exist insoluble problems, not that these problems are actually common. It seems to me like Godel’s incompleteness theorem - there are statements which are true but which cannot be proved, but this doesn’t mean that no statement can be proved, or that mathematics is pointless. At the end of the day regardless of whether or not the fundamental underpinning of mathematics are on shaky ground, or whether there are unprovble theorems and unsolvable problems, the actual mathematics that allows us to build aeroplanes works, and the planes fly.