it would seem easier to build (or mutate into) something that keeps going forever than it is to build something that goes for a while then stops.
On reflection, I realize this point might be applied to repetitive drudgery. But I was applying it to the behavior “engage in just so much efficient exploration.” My point is that it may be easier to mutate into something that explores and explores and explores, than it would be to mutate into something that explores for a while then stops.
the vast majority of possible expected utility maximizers, would only engage in just so much efficient exploration, and spend most of its time exploiting the best alternative found so far, over and over and over.
I’m not convinced of that. First, “vast majority” needs to use an appropriate measure, one that is applicable to evolutionary results. If, when two equally probable mutations compete in the same environment, one of those mutations wins, making the other extinct, then the winner needs to be assigned the far greater weight. So, for example, if humans were to compete against a variant of human without the boredom instinct, who would win?
Second, it would seem easier to build (or mutate into) something that keeps going forever than it is to build something that goes for a while then stops. Cancer, for example, just keeps going and going, and it takes a lot of bodily tricks to put a stop to that.
Seconding the expectation of a “useless clusterf—k.”
The hope is that the shared biases will be ones that the site owner considers valuable and useful
The obvious way to do that is for the site owner to make some users more equal than others.
the Hacker News website seems to be doing fine.
Security through obscurity. Last I checked, it confirmed my impression, gathered from Digg and Reddit, that as long as the site remains sufficiently unpopular, it will not deteriorate.
Two separate issues:
1) Is it a good (legitimately persuasive) argument?
2) If not then after all the hairsplitting is done, what sort of bad argument is it?
The more important issue is (a). A few points:
a) Quibbling over the categorization of the fallacy is sometimes used to mask the fact that it’s a bad argument.
b) There are plenty of people who can recognize bad arguments without knowing anything about the names of the fallacies, which leads to
c) We learn the names of the fallacies, not in order to learn to spot bad arguments, but as a convenience so that we don’t have to explain at length to the other guy why the argument is bad.
d) Often perfectly legitimate arguments technically fall into one of the categories of fallacy. Technically being a classical fallacy is no guaranteed that an argument is actually fallacious. Some counterexamples.
In short, the classical fallacies are a convenient timesaver. But you don’t need to have learned them to avoid being an idiot, and learning them will not stop you from being an idiot, and taking them too seriously can make you into an idiot.
Morality is an aspect of custom. Custom requires certain preconditions: it is an adaptation to a certain environment. Great political power breaks a key component of that environment.
More specifically, morality is a spontaneously arising system for resolving conflict among people with approximately equal power, such that adherence to morality is an optimal strategy for a typical person. A person with great power has less need to compromise and so his optimal strategy is probably a mix of compromise and brute force—i.e., corruption.
This does not require very specific human psychology. It is likely to describe any set of agents where the agents satisfy certain general conditions. Design two agents (entities with preferences and abilities) and in certain areas those entities are likely to have conflicting desires and are likely, therefore, to come into conflict and to need a system for resolving conflict (a morality) - regardless of their psychology. But grant one of these entities sufficiently great power, and it can resolve conflict by pushing aside the other agent, thereby dispensing with morality, thereby being corrupted by power.
Someone had just asked a malformed version of an old probability puzzle [...] someone said to me, “Well, what you just gave is the Bayesian answer, but in orthodox statistics the answer is 1⁄3.” [..] That was when I discovered that I was of the type called ‘Bayesian’.
I think a more reasonable conclusion is: yes indeed it is malformed, and the person I am speaking to is evidently not competent enough to notice how this necessarily affects the answer and invalidates the familiar answer, and so they may not be a reliable guide to probability and in particular to what is or is not “orthodox” or “bayesian.” What I think you ought to have discovered was not that you were Bayesian, but that you had not blundered, whereas the person you were speaking to had blundered.
Aaron—yes, I know that. It’s beside the point.
My point was that vampires were by definition not real—or at least, not understandable—because any time we found something real and understandable that met the definition of a vampire, we would change the definition to exclude it.
But the same exchange might have occurred with something entirely real. We are not in the habit of giving fully adequate definitions, so it is often possible to find counterexamples to the definitions we give, which might prompt the other person to add to the definition to exclude the counterexample. For example:
A: What is a dog?
B: A dog is a four-footed animal that is a popular pet.
A: So a cat is a dog.
B: Dogs bark.
A: So if I teach a cat to bark, it will become a dog.
etc.
Time—Philip Johnson is not just a Christian but a creationist. Do you mean, “if there are smart creationists out there...”? I don’t really pay much attention to the religious beliefs of the smartest mathematicians and scientists and I’m not especially keen on looking into it now, but I would be surprised if all top scientists without exception were atheists. This page seems to suggest that many of the best scientists are something other than atheist, many of those Christian.
Whoever is censoring Caledonian: can it be done without adding the content-free nastiness (such as “bizarre objection”, “illogic”, and “gibberish”)?
Any two AIs are likely to have a much vaster difference in effective intelligence than you could ever find between two humans (for one thing, their hardware might be much more different than any two working human brains). This likelihood increases further if (at least) some subset of them is capable of strong self-improvement. With enough difference in power, cooperation becomes a losing strategy for the more powerful party.
I read stuff like this and immediately my mind thinks, “comparative advantage.” The point is that it can be (and probably is) worthwhile for Bob and Bill to trade with each other even if Bob is better at absolutely everything than Bill. And if it is worthwhile for them to trade with each other, then it may well be in the interest of neither of them to (say) eliminate the other, and it may be a waste of resources to (say) coerce the other. It is worthwhile for the state to coerce the population because the state is few and the population are many, so the per-person cost of coercion falls below the benefit of coercion; it is much less worthwhile for an individual to coerce another (slavery generally has the backing of the state—see for example the fugitive slave laws). But this mass production of coercive fear works in part because humans are similar to each other and so can be dealt with more or less the same way. If AIs are all over the place, then this does not necessarily hold. Furthermore if one AI decides to coerce the humans (who are admittedly similar to each other) then the other AIs may oppose him in order that they themselves might retain direct access to humans.
The AIs might agree that they’d all be better off if they took the matter currently in use by humans for themselves, dividing the spoils among each other.
Maybe but maybe not. Dividing the spoils paints a picture of the one-time destruction of the human race, and it may well be to the advantage of the AIs not to kill off the humans. After all, if the humans have something worth treating as spoils, then the humans are productive and so might be even more useful alive.
You definitely don’t want an FAI to unpredictably change its terminal values. Figuring out how to reliably prevent this kind of thing from happening, even in a strongly self-modifying mind (which humans aren’t), is one of the sub-problems of the FAI problem.
The FAI may be an unsolvable problem, if by FAI we mean an AI into which certain limits are baked. This has seemed dubious ever since Asimov. The idea of baking in rules of robotics has long seemed to me to fundamentally misunderstand both the nature of morality and the nature of intelligence. But time will tell.
An AI can indeed have preferences that conflict with human preferences, but if it doesn’t start out with such preferences, it’s unclear how it comes to have them later.
We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted. For example, babies do not start out preferring Bach to Beethoven or Beethoven to Bach, but adults are able to develop that preference, even if it is not clear at this point how they come to do so.
If you could do so easily and with complete impunity, would you organize fights to death for your pleasure?
Voters have the ability to vote for policies and to do so easily and with complete impunity (nobody retaliates against a voter for his vote). And, unsurprisingly, voters regularly vote to take from others to give unto themselves—which is something they would never do in person (unless they were criminals, such as muggers or burglars). Moreover humans have an awe-inspiring capacity to clothe their rapaciousness in fine-sounding rhetoric.
Moreover, humans are often tempted to do things they know they shouldn’t, because they also have selfish desires. AIs don’t if you don’t build it into them.
Conflict does not require selfish desires. Any desire, of whatever sort, could potentially come into conflict with another person’s desire, and when there are many minds each with its own set of desires then conflict is almost inevitable. So the problem does not, in fact, turn on whether the mind is “selfish” or not. Any sort of desire can create the conflict, and conflict as such creates the problem I described. In a nutshell: evil men need not be selfish. A man such as Pol Pot could indeed have wanted nothing for himself and still ended up murdering millions of his countrymen.
A tendency to become corrupt when placed into positions of power is a feature of some minds.
Morality in the human universe is a compromise between conflicting wills. The compromise is useful because the alternative is conflict, and conflict is wasteful. Law is a specific instance of this, so let us look at property rights: property rights is a decision-making procedure for deciding between conflicting desires concerning the owned object. There really is no point in even having property rights except in the context of the potential for conflict. Remove conflict, and you remove the raison d’etre of property rights, and more generally the raison d’etre of law, and more generally the raison d’etre of morality. Give a person power, and he no longer needs to compromise with others, and so for him the raison d’etre of morality vanishes and he acts as he pleases.
The feature of human minds that renders morality necessary is the possibility that humans can have preferences that conflict with the preferences of other humans, thereby requiring a decisionmaking procedure for deciding whose will prevails. Preference is, furthermore, revealed in the actions taken by a mind, so a mind that acts has preferences. So all the above is applicable to an artificial intelligence if the artificial intelligence acts.
What makes you think a human-designed AI would be vulnerable to this kind of corruption?
I am assuming it acts, and therefore makes choices, and therefore has preferences, and therefore can have preferences which conflict with the preferences of other minds (including human minds).
We’ve been told that a General AI will have power beyond any despot known to history.
If that will be then we are doomed. Power corrupts. In theory an AI, not being human, might resist the corruption, but I wouldn’t bet on that. I do not think it is a mere peculiarity of humanity that we are vulnerable to corruption.
We humans are kept in check by each other. We might therefore hope, and attempt to engineer, a proliferation of self-improving AIs, to form a society and to keep each other in check. With luck, cooperative AIs might be more successful at improving themselves—just as honest folk are for the most part more successful than criminals—and thus tend for the most part to out-pace the would-be despots.
As far as how a society of AIs would relate to humans, there are various possibilities. One dystopia imagines that humans will be treated like lower animals, but this is not necessarily what will happen. Animals are not merely dumb, but unable to take part in mutual respect of rights. We humans will always be able to and so might well remain forever recipients of an AI respect for our rights however much they evolve past us. We may of course be excluded from aspects of AI society which we humans are not able to handle, just as we exclude animals from rights. We may never earn hyper-rights, whatever those may be. But we might retain our rights.
I say go ahead and pick a number out of the air,
A somewhat arbitrary starting number is also useful as a seed for a process of iterative approximation to a true value.
how you can talk about probabilities without talking about several possible worlds
But if probability is in the mind, and the mind in question is in this world, why are other worlds needed? Moreover (from wikipedia):
In Bayesian theory, the assessment of probability can be approached in several ways. One is based on betting: the degree of belief in a proposition is reflected in the odds that the assessor is willing to bet on the success of a trial of its truth.
Disposition to bet surely does not require a commitment to possible worlds.
Elephants are not properties of physics any more than probabilities are. The concept of an elephant is subjective—as are all concepts.
If you are indeed agreeing with the parallel I have set up between probability and elephants and if this is not just your own personal view, then perhaps the subjectivist theory of probability should more properly be called the subjectivist theory of pretty much everything that populates our familiar world. Anyway, I think I can agree that probability is as subjective and as psychological and as non-physical and as existing in the mind and not in the world as an elephant or, say, an exploding nuclear bomb—another item that populates our familiar world.
With complete information (and a big computer) an observer would know which way the coin would land—and would find probabilities irrelevant.
But this is true of most everyday observations. We observe events on a level far removed from the subatomic level. With complete information and infinite computing power an observer would would find all or virtually all ordinary human-level observations irrelevant. But irrelevancy to such an observer is not the same thing as non-reality. For example, the existence of elephants would be irrelevant to an observer who has complete information on the subatomic level and sufficient computing power to deal with it. But it does not follow that elephants do not exist. Do you think it follows that elephants do not exist?
The probabilities arise from ignorance and lack of computing power—properties of observers, not properties of the observed.
The concept of an elephant could with equal reason be said to arise from ignorance and lack of computing power. I can certainly understand that a thought such as, “the elephant likes peanuts, therefore it will accept this peanut” is a much easier thought to entertain than a thought that infallibly tracks every subatomic particle in its body and in the environment around it. So, certainly, the concept of an elephant is a wonderful shortcut. But I’m not so sure about getting from this to the conclusion that elephants (like probability) are subjective. Do you think that elephants are subjective?
If such ideas seem unproblematic to you
It is the example that seems on the face of it unproblematic. I am open to either (a) a demonstration that it is compatible with subjectivism[*], or (b) a demonstration that it is problematic. I am open to either one. Or to something else. In any case, I don’t adhere to frequentism.
[*] (I made no firm claim that it is not compatible with subjectivism—you are the one who rejected the compatibility—my own purpose was only to raise the question since it seems on the face of it hard to square with subjectivism, not to answer the question definitively.)
I’m getting two things out of this.
1) Evolutionary cynicism produces different predictions from cognitive cynicism, e.g. because the current environment is not the ancestral environment.
2) Cognitive cynicism glooms up Eliezer’s day but evolutionary cynicism does not.
(1) is worth keeping in mind. I’m not sure what significance (2) has.
However, we may want to develop a cynical account of a certain behavior while suspending judgment about whether the behavior was learned or evolved. Call such cynicism “agnostic cynicism”, maybe. So we have three types of cynicism: evolutionary cynicism, cognitive cynicism, and agnostic cynicism.
A careful thinker will want to avoid jumping to conclusions, and because of this, he may lean toward agnostic cynicism.