Our civilization is not provably Friendly and why this should worry us
As I was thinking about my draft “moral progress” sequence and debating people on the LessWrong chat channel, it occurred to me in a sort of “my current beliefs already imply this but I hadn’t noticed it so clearly before” way that our civilization does not reliably ensure its own survival or produce anything like fixed versions of its values. In other words if we judge current human civilization by the same standards as AI it is nearly certainly unfriendly. FAI is thus insurance policy not just against AI or difficult to defeat existential risks but against ourselves too.
While existential risk study basically already implies we currently don’t reliably optimize for civilization wise survival and this is a common topic of LW and my own (as of yet unpublished but supported by many other commenter’s on LW) sequence of posts on moral progress attacks on a fundamental level “fixed values” part, this hasn’t been addressed in this form. Even “people are crazy the world is mad” attitudes that result in “raising the sanity waterline” efforts are fundamentally not addressing the problem, they assume we have a good structure and all we need is more rational people. Or that maybe the structure is rotten but that if we have enough rational people we can use the killer application of FAI to fix it. I see no guarantee of this at all, very little plausibility even. Especially the former assumption seems like assuming better CPU cores make FAI development more likely, while the latter relies on a very rapid difficult to predict hard take off scenario that isn’t universally endorsed by LW/OBers.
Why are so many willing to admit that our societies truth finding mechanism may indeed be broken and that moral change not progress is the name of the game, yet not put this together on a gut level, like with evolution or living in a universe where really bad things can happen? Or even motivate us to expend some effort to at least ascertain if this is really something of as great urgency as it seems at first look.
Should I make a proper article on this topic to cover further thoughts and more supporting arguments?
Our civilization is not provably Friendly and why this should worry us
As I was thinking about my draft “moral progress” sequence and debating people on the LessWrong chat channel, it occurred to me in a sort of “my current beliefs already imply this but I hadn’t noticed it so clearly before” way that our civilization does not reliably ensure its own survival or produce anything like fixed versions of its values. In other words if we judge current human civilization by the same standards as AI it is nearly certainly unfriendly. FAI is thus insurance policy not just against AI or difficult to defeat existential risks but against ourselves too.
While existential risk study basically already implies we currently don’t reliably optimize for civilization wise survival and this is a common topic of LW and my own (as of yet unpublished but supported by many other commenter’s on LW) sequence of posts on moral progress attacks on a fundamental level “fixed values” part, this hasn’t been addressed in this form. Even “people are crazy the world is mad” attitudes that result in “raising the sanity waterline” efforts are fundamentally not addressing the problem, they assume we have a good structure and all we need is more rational people. Or that maybe the structure is rotten but that if we have enough rational people we can use the killer application of FAI to fix it. I see no guarantee of this at all, very little plausibility even. Especially the former assumption seems like assuming better CPU cores make FAI development more likely, while the latter relies on a very rapid difficult to predict hard take off scenario that isn’t universally endorsed by LW/OBers.
Why are so many willing to admit that our societies truth finding mechanism may indeed be broken and that moral change not progress is the name of the game, yet not put this together on a gut level, like with evolution or living in a universe where really bad things can happen? Or even motivate us to expend some effort to at least ascertain if this is really something of as great urgency as it seems at first look.
Should I make a proper article on this topic to cover further thoughts and more supporting arguments?