Living in a society with as little power as the average human citizen has in a current human society.
Well, it’s a list of four then, not a list of three. It’s still much simpler than “morality is everything humans value”.
The nasty part is that AI agents could pretty easily get way, waaaay out of that power-level. Not just by going FOOM, but simply by, say, making a lot of money and purchasing huge sums of computing resources to run multiple copies of themselves which now have more money-making power and as many votes for Parliament as there are copies, and so on. This is roughly the path taken by power-hungry humans already, and look how that keeps turning out.
You seem to be making the tacit assumption that no one really values morality, and just plays along (in egalitarian societies) because they have to.
Friendly AI with large amounts of scientific and technological resources can start spitting out utopian advancements (pop really good art, pop abundance economy, pop immortality, pop space travel, pop whole nonliving planets converted into fun-theoretic wonderlands) on a regular basis.
You seem to be making the tacit assumption that no one really values morality, and just plays along (in egalitarian societies) because they have to.
Let me clarify. My assumption is that “Western liberal meta-morality” is not the morality most people actually believe in, it’s the code of rules used to keep the peace between people who are expected to disagree on moral matters.
For instance, many people believe, for religious reasons or pure Squick or otherwise, that you shouldn’t eat insects, and shouldn’t have multiple sexual partners. These restrictions are explicitly not encoded in law, because they’re matters of expected moral disagreement.
I expect people to really behave according to their own morality, and I also expect that people are trainable, via culture, to adhere to liberal meta-morality as a way of maintaining moral diversity in a real society, since previous experiments in societies run entirely according to a unitary moral code (for instance, societies governed by religious law) have been very low-utility compared to liberal societies.
In short, humans play along with the liberal-democratic social contract because, for us, doing so has far more benefits than drawbacks, from all but the most fundamentalist standpoints. When the established social contract begins to result in low-utility life-states (for example, during an interminable economic depression in which the elite of society shows that it considers the masses morally deficient for having less wealth), the social contract itself frays and people start reverting to their underlying but more conflicting moral codes (ie: people turn to various radical movements offering to enact a unitary moral code over all of society).
Note that all of this also relies upon the fact that human beings have a biased preference towards productive cooperation when compared with hypothetical rational utility-maximizing agents.
None of this, unfortunately, applies to AIs, because AIs won’t have the same underlying moral codes or the same game-theoretic equilibrium policies or the human bias towards cooperation or the same levels of power and influence as human beings.
When dealing with AI, it’s much safer to program in some kind of meta-moral or meta-ethical code directly at the core, thus ensuring that the AI wants to, at the very least, abide by the rules of human society, and at best, give humans everything we want (up to and including AI Pals Who Are Fun To Be With, thank you Sirius Cybernetics Corporation).
Can’t that be done by Oracle AIs?
I haven’t heard the term. Might I guess that it means an AI in a “glass box”, such that it can see the real world but not actually affect anything outside its box?
Yes, a friendly Oracle AI could spit out blueprints or plans for things that are helpful to humans. However, you’re still dealing with the Friendliness problem there, or possibly with something like NP-completeness. Two cases:
We humans have some method for verifying that anything spit out by the potentially unfriendly Oracle AI is actually safe to use. The laws of computation work out such that we can easily check the safety of its output, but it took such huge amounts of intelligence or computation power to create the output that we humans couldn’t have done it on our own and needed an AI to help. A good example would be having an Oracle AI spit out scientific papers for publication: many scientists can replicate a result they wouldn’t have come up with on their own, and verify the safety of doing a given experiment.
We don’t have any way of verifying the safety of following the Oracle’s advice, and are thus trusting it. Friendliness is then once again the primary concern.
For real-life-right-now, it does look like the first case is relatively common. Non-AGI machine learning algorithms have been used before to generate human-checkable scientific findings.
None of this, unfortunately, applies to AIs, because AIs won’t have the same underlying moral codes or the same game-theoretic equilibrium policies or the human bias towards cooperation or the same levels of power and influence as human beings.
None of that necessarily applies to AIs, but then it depends on the AI. We could, for instance, pluck AIs from
virtualised socieities of AIs that haven’t descended into mass slaughter.
Congratulations: you’ve now developed an entire society of agents who specifically blame humans for acting as the survival-culling force in their miniature world.
Did you watch Attack on Titan and think, “Why don’t the humans love their benevolent Titan overlords?”?
They’re doing it to themselves. We wouldn’t have much motivation to close down a vr that contained survivors.
ETA We could make copies of all involved and put them in solipstic robot heavens.
It requires a population that’s capable cumulatively, it doesn’t require that each member of the population be capable.
It’s like arguing a command economy versus a free economy and saying that if the dictator in the command economy doesn’t know how to run an economy, how can each consumer in a free economy know how to run the economy? They don’t, individually, but as a group, the economy they produce is better than the one with the dictatorship.
Democracy has nothing to do with capable populations. It definitely has nothing to do with the median voter being smarter than the average politician. It’s just about giving the population some degree of threat to hold over politicians.
“Smarter” and “capable” aren’t the same thing. Especially if “more capable” is interpreted to be about practicalities: what we mean by “more capable” of doing X is that the population, given a chance is more likely to do X than politicians are. There are several cases where the population is more capable in this sense. For instance, the population is more capable of coming up with decisions that don’t preferentially benefit politicians.
Furthermore, the median voter being smarter and the voters being cumulatively smarter aren’t the same thing either. It may be that an average individual voter is stupider than an average individual politician, but when accumulating votes the errors cancel out in such a manner that the voters cumulatively come up with decisions that are as good as the decisions that a smarter person would make.
I’m increasingly of the opinion that the “real” point of democracy is something entirely aside from the rhetoric used to support it … but you of all people should know that averaging the estimates of how many beans are in the jar does better than any individual guess.
Systems with humans as components can, under the right conditions, do better than those humans could do alone; several insultingly trivial examples spring to mind as soon as it’s phrased that way.
Could you clarify? Are you saying that for democracy to exist it doesn’t require capable voters, or that for democracy to work well that it doesn’t?
In the classic free-market argument, merchants don’t have to be altruistic to accomplish the general good, because the way to advance their private interest is to sell goods that other people want. But that doesn’t generalize to democracy, since there isn’t trading involved in democratic voting.
However there is the question of what “working well” means, given that humans are not rational and satisfying expressed desires might or might not fall under the “working well” label.
Democracy requires capable voters in the same way capitalism requires altruistic merchants.
The grandparent is wrong, but I don’t think this is quite right either. Democracy roughly tracks the capability (at the very least in the domain of delegation) and preference of the median voter, but in a capitalistic economy you don’t have to buy services from the median firm. You can choose to only purchase from the best firm or no firm at all if none offer favorable terms.
in a capitalistic economy you don’t have to buy services from the median firm
In the equilibrium, the average consumer buys from the average firm. Otherwise it doesn’t stay average for long.
However the core of the issue is that democracy is a mechanism, it’s not guaranteed to produce optimal or even good results. Having “bad” voters will not prevent the mechanism of democracy from functioning, it just might lead to “bad” results.
“Democracy is the theory that the common people know what they want, and deserve to get it good and hard.”—H.L.Mencken.
In the equilibrium, the average consumer buys from the average firm. Otherwise it doesn’t stay average for long.
The median consumer of a good purchases from (somewhere around) the median firm selling a good. That doesn’t necessarily aggregate, and it certainly doesn’t weigh all consumers or firms equally. The consumers who buy the most of a good tend to have different preferences and research opportunities than average consumers, for example.
You could get similar results in a democracy, but most democracies don’t really encourage it : most places emphasize voting regardless of knowledge of a topic, and some jurisdictions mandate it.
You say that like it’s a bad thing. I am not multiplying by N the problem of solving and hardwiring friendliness. I am letting them sort it our for themselves. Like an evolutionary algorithm.
Well, how are you going to force them into a society in the first place? Remember, each individual AI is presumed to be intelligent enough to escape any attempt to sandbox it. This society you intend to create is a sandbox.
(It’s worth mentioning now that I don’t actually believe that UFAI is a serious threat. I do believe you are making very poor arguments against that claim that merit counter-arguments.)
Well, it’s a list of four then, not a list of three. It’s still much simpler than “morality is everything humans value”.
You seem to be making the tacit assumption that no one really values morality, and just plays along (in egalitarian societies) because they have to.
Can’t that be done by Oracle AIs?
Let me clarify. My assumption is that “Western liberal meta-morality” is not the morality most people actually believe in, it’s the code of rules used to keep the peace between people who are expected to disagree on moral matters.
For instance, many people believe, for religious reasons or pure Squick or otherwise, that you shouldn’t eat insects, and shouldn’t have multiple sexual partners. These restrictions are explicitly not encoded in law, because they’re matters of expected moral disagreement.
I expect people to really behave according to their own morality, and I also expect that people are trainable, via culture, to adhere to liberal meta-morality as a way of maintaining moral diversity in a real society, since previous experiments in societies run entirely according to a unitary moral code (for instance, societies governed by religious law) have been very low-utility compared to liberal societies.
In short, humans play along with the liberal-democratic social contract because, for us, doing so has far more benefits than drawbacks, from all but the most fundamentalist standpoints. When the established social contract begins to result in low-utility life-states (for example, during an interminable economic depression in which the elite of society shows that it considers the masses morally deficient for having less wealth), the social contract itself frays and people start reverting to their underlying but more conflicting moral codes (ie: people turn to various radical movements offering to enact a unitary moral code over all of society).
Note that all of this also relies upon the fact that human beings have a biased preference towards productive cooperation when compared with hypothetical rational utility-maximizing agents.
None of this, unfortunately, applies to AIs, because AIs won’t have the same underlying moral codes or the same game-theoretic equilibrium policies or the human bias towards cooperation or the same levels of power and influence as human beings.
When dealing with AI, it’s much safer to program in some kind of meta-moral or meta-ethical code directly at the core, thus ensuring that the AI wants to, at the very least, abide by the rules of human society, and at best, give humans everything we want (up to and including AI Pals Who Are Fun To Be With, thank you Sirius Cybernetics Corporation).
I haven’t heard the term. Might I guess that it means an AI in a “glass box”, such that it can see the real world but not actually affect anything outside its box?
Yes, a friendly Oracle AI could spit out blueprints or plans for things that are helpful to humans. However, you’re still dealing with the Friendliness problem there, or possibly with something like NP-completeness. Two cases:
We humans have some method for verifying that anything spit out by the potentially unfriendly Oracle AI is actually safe to use. The laws of computation work out such that we can easily check the safety of its output, but it took such huge amounts of intelligence or computation power to create the output that we humans couldn’t have done it on our own and needed an AI to help. A good example would be having an Oracle AI spit out scientific papers for publication: many scientists can replicate a result they wouldn’t have come up with on their own, and verify the safety of doing a given experiment.
We don’t have any way of verifying the safety of following the Oracle’s advice, and are thus trusting it. Friendliness is then once again the primary concern.
For real-life-right-now, it does look like the first case is relatively common. Non-AGI machine learning algorithms have been used before to generate human-checkable scientific findings.
Programming in a bias towards conformity (kohlberg level 2) maybe a lot easier than EYes fine grained friendliness.
None of that necessarily applies to AIs, but then it depends on the AI. We could, for instance, pluck AIs from virtualised socieities of AIs that haven’t descended into mass slaughter.
Congratulations: you’ve now developed an entire society of agents who specifically blame humans for acting as the survival-culling force in their miniature world.
Did you watch Attack on Titan and think, “Why don’t the humans love their benevolent Titan overlords?”?
Well now I have both a new series to read/watch and a major spoiler for it.
Don’t worry! I’ve spoiled nothing for you that wasn’t apparent from the lyrics of the theme song.
They’re doing it to themselves. We wouldn’t have much motivation to close down a vr that contained survivors. ETA We could make copies of all involved and put them in solipstic robot heavens.
...And that way you turn the problem of making an AI that won’t kill you into one of making a society of AIs that won’t kill you.
If Despotism failed only for want of a capable benevolent despot, what chance has Democracy, which requires a whole population of capable voters?
It requires a population that’s capable cumulatively, it doesn’t require that each member of the population be capable.
It’s like arguing a command economy versus a free economy and saying that if the dictator in the command economy doesn’t know how to run an economy, how can each consumer in a free economy know how to run the economy? They don’t, individually, but as a group, the economy they produce is better than the one with the dictatorship.
Democracy has nothing to do with capable populations. It definitely has nothing to do with the median voter being smarter than the average politician. It’s just about giving the population some degree of threat to hold over politicians.
“Smarter” and “capable” aren’t the same thing. Especially if “more capable” is interpreted to be about practicalities: what we mean by “more capable” of doing X is that the population, given a chance is more likely to do X than politicians are. There are several cases where the population is more capable in this sense. For instance, the population is more capable of coming up with decisions that don’t preferentially benefit politicians.
Furthermore, the median voter being smarter and the voters being cumulatively smarter aren’t the same thing either. It may be that an average individual voter is stupider than an average individual politician, but when accumulating votes the errors cancel out in such a manner that the voters cumulatively come up with decisions that are as good as the decisions that a smarter person would make.
I’m increasingly of the opinion that the “real” point of democracy is something entirely aside from the rhetoric used to support it … but you of all people should know that averaging the estimates of how many beans are in the jar does better than any individual guess.
Systems with humans as components can, under the right conditions, do better than those humans could do alone; several insultingly trivial examples spring to mind as soon as it’s phrased that way.
Is democracy such a system? Eh.
Democracy requires capable voters in the same way capitalism requires altruistic merchants.
In other words, not at all.
Could you clarify? Are you saying that for democracy to exist it doesn’t require capable voters, or that for democracy to work well that it doesn’t?
In the classic free-market argument, merchants don’t have to be altruistic to accomplish the general good, because the way to advance their private interest is to sell goods that other people want. But that doesn’t generalize to democracy, since there isn’t trading involved in democratic voting.
See here
However there is the question of what “working well” means, given that humans are not rational and satisfying expressed desires might or might not fall under the “working well” label.
Ah, I see. You’re just saying that democracy doesn’t stop happening just because voters have preferences I don’t approve of. :)
Actually, I’m making a stronger claim—voters can screw themselves up in pretty serious fashion and it’s still will be full-blown democracy in action.
The grandparent is wrong, but I don’t think this is quite right either. Democracy roughly tracks the capability (at the very least in the domain of delegation) and preference of the median voter, but in a capitalistic economy you don’t have to buy services from the median firm. You can choose to only purchase from the best firm or no firm at all if none offer favorable terms.
In the equilibrium, the average consumer buys from the average firm. Otherwise it doesn’t stay average for long.
However the core of the issue is that democracy is a mechanism, it’s not guaranteed to produce optimal or even good results. Having “bad” voters will not prevent the mechanism of democracy from functioning, it just might lead to “bad” results.
“Democracy is the theory that the common people know what they want, and deserve to get it good and hard.”—H.L.Mencken.
The median consumer of a good purchases from (somewhere around) the median firm selling a good. That doesn’t necessarily aggregate, and it certainly doesn’t weigh all consumers or firms equally. The consumers who buy the most of a good tend to have different preferences and research opportunities than average consumers, for example.
You could get similar results in a democracy, but most democracies don’t really encourage it : most places emphasize voting regardless of knowledge of a topic, and some jurisdictions mandate it.
You say that like it’s a bad thing. I am not multiplying by N the problem of solving and hardwiring friendliness. I am letting them sort it our for themselves. Like an evolutionary algorithm.
Well, how are you going to force them into a society in the first place? Remember, each individual AI is presumed to be intelligent enough to escape any attempt to sandbox it. This society you intend to create is a sandbox.
(It’s worth mentioning now that I don’t actually believe that UFAI is a serious threat. I do believe you are making very poor arguments against that claim that merit counter-arguments.)
I am assuming they are seeds, not superintelligences