It is not just their chances of success. For the donations to matter, you need SI to succeed where without SI there is failure. You need to get a basket of eggs, and have all the good looking eggs be rotten inside but one fairly rotten looking egg be fresh. Even if a rotten looking egg is nonetheless more likely to be fresh inside than one would believe, it is highly unlikely situation.
JaneQ
Karma: 71
It has to also be probable that their work averts those risks, which seem incredibly improbable by any reasonable estimate. If the alternative Earth was to adopt a strategy of ignoring prophetic groups of ‘idea guys’ similar to SI and ignore their pleads for donations so that they can hire competent researchers to pursue their ideas, I do not think that such decision would have increased the risk by more than a miniscule amount.
But are you sure that you are not now falling for typical mind fallacy?
The very premise of your original post is that it is not all signaling; that there is a substantial number of the honest-and-naive folk who not only don’t steal but assume that others don’t.
I do not see why are you even interested in asking that sort of question if you have such a view—surely under such view you will steal if you are sure you will get away with it, just as you would e.g. try to manipulate and lie. edit: I.e. the premise seems incoherent to me. You need honesty to exist for your method to be of any value; and you need honesty not to exist for your method to be harmless and neutral. If the language is all signaling, the most your method will do is weed you out at the selection process—you have yourself already sent the wrong signal that you believe language to be all deception and signaling.
It too closely approximates the way the herein proposed unfriendly AI would reason—get itself a goal (predict stealing on job for example) and then proceed to solving it, oblivious to the notion of fairness. I’ve seen several other posts over the time that rub in exactly same wrong way, but I do not remember exact titles. By the way, what if I am to use the same sort of reasoning as in this post on the people who have a rather odd mental model of artificial minds (or mind uploaded fellow humans) ?
Good point. More intricate questions like this, with ‘nobody could resist’ wording, are also much more fair. The question as of what the person believes is the natural human state are more dubious.
I’m not quite sure what is the essence of your disagreement, or what relation the honest people already being harmed has with the argument I made.
I’m not sure what you think my disagreement should have focused on—the technique outlined in the article can be effective and is used in practice, and there is no point to be made that it is bad for the persons employing it; I can not make an argument convincing an antisocial person not to use this technique. However, this technique depletes the common good that is the normal human communication; it is an example of tragedy of the commons—as long as most of us refrain from using this sort of approach, it can work for the few that do not understand the common good or do not care. Hence the Dutch dike example. This method is a social wrong, not necessarily a personal wrong—it may work well in general circumstances. Furthermore it is, in certain sense that is normally quite obvious, unfair. (edit: It is nonetheless widely used, of course—even with all the social mechanisms in human psyche, the human brain is a predictive system that will use any correlations it encounters—and people do often signal their pretend non-understanding -I suspect this silly game significantly discriminates against Aspergers spectrum disorders).
edit: For a more conspicuous example of how predictive methods are not in general socially acceptable, consider that if I am to train a predictor of criminality on the profile data complete with a photograph, the skin albedo estimate from the photograph will be a significant part of the predictor assuming that the data processed originates in north America. Matter of fact I have some experience with predictive software of the kind that processes interview answers. Let me assure you, my best guess from briefly reading your profile is that you are not in the category of people who benefit from this software that just uses all correlations it finds—pretty much all people that have non standard interests and do not answer questions in the standard ways are penalized, and I do not think it would be easy to fake the answers beneficially without access to the model being used.
However, imagine if the typical mind fallacy was correct. The employers could instead ask “what do you think the percentage of employees who have stolen from their job is?”
To be honest, this is a perfect example of what is so off-putting about this community. This method is simply socially wrong—it works against both the people who stole, and people who had something stolen from, who get penalized for the honest answer and, if such methods are to be more widely employed, are now inclined to second-guess and say “no, I don’t think anyone steals” (and yes, this method is employed to some extent, subconsciously at least). The idea parasitizes on the social contract that is human language, with the honest naivete of the asocial. It’s as if a Dutch town was building a dike and someone was suggesting that anyone who needs materials for repairing the house should just take them from that weird pile in the sea. The only reason such method can work, is because others have been losing a little here and there to maintain some trust necessary for effective communication.
I’m not sure why you think that such writings should convince a rational person that you have the relevant skill. If you were an art critic, even a very good one, that would not convince people you are a good artist.
This is not, in any way shape or form, the same skill as the ability to manage a nonprofit.
Indeed, but you are asking me to assume that the skills you display writing your articles are the same skill as the skills relevant to directing the AI effort.
edit: Furthermore, when it comes to works on rationality as ‘applied math of optimization’, the most obvious way to classify those writings is to look for some great success attributable to your writings—some highly successful businessmen saying how much the article on such and such fallacy helped them succeed, that sort of thing.
I think it is fair to say Earth was doing the “AI math” before the computers. Extending to the today—there is a lot of mathematics to be done for a good, safe AI—but how are we to know that the SI has the actionable effort planning skills required to correctly identify and fund research in such mathematics?
I know that you believe that you have the required skills; but note that in my model such belief results from both the presence of extraordinary effort planning skill, and from absence of effort planning skills. The prior probability of extraordinary effort planning skill is very low. Furthermore as the effort planning is, to some extent, a cross domain skill, the prior inefficacy (which was criticized by Holden) seem to be a fairly strong evidence against extraordinary skills in this area.
Great article, however, there is a third important option which is ‘request proof then, if passed, donate’ (Holden seem to have opted for this in his review of S.I., but it is broadly applicable in general).
For example if there is a charity promising to save 10 millions people using a method X that is not very likely to work, but is very cheap—a Pascal Wager like situation. In this situation, even if this charity is presently the best in terms of expected payoff it may be a better option still to, rather than paying the full sum, pay only enough for a basic test of method X which the method X would be unlikely to pass if it is ineffective; then donating if the test has passed. This decreases the expected cost proportionally to the unlikelihood of X efficacy and the specificity of the test.
It seems to me that 100 years ago (or more) you would have to consider pretty much any philosophy and mathematics to be relevant to AI risk reduction, as well as reduction of other potential risks, and the attempts to select the work particularly conductive to the AI risk reduction would not be able to succeed. Effort planning is the key to success.
On somewhat unrelated: Reading the publications and this thread, there is point of definitions that I do not understand: what exactly does S.I. mean when it speaks of “utility function” in the context of an AI? Is it a computable mathematical function over a model, such that the ‘intelligence’ component computes the action that results in maximum of that function taken over the world state resulting from the action?
It seems to me that the premise of funding SI is that people smarter (or more appropriately specialized) than you will then be able to make discoveries that otherwise would be underfunded or wrongly-purposed.
But then SI has to have dramatically better idea what research has to be funded to protect the mankind, than every other group of people capable of either performing such research or employing people to perform such research.
Muehlhauser has stated that SI should be compared to alternatives in form of the organizations working on the AI risk mitigation, but that seems like an overly narrow choice reliant on presumption that it is not an alternative to not work on AI risk mitigation now.
For example, 100 years ago it would seem to have been too early to fund work on AI risk mitigation; that may still be the case; as the time gone on one could naturally expect that the opinions will form a distribution and the first organizations offering AI risk mitigation will pop up earlier than the time at which such work is effective. When we look into the past through the goggles of notoriety, we don’t see all the failed early starts.
All the other people and organizations that are no less capable of identifying the preventable risks (if those exist) and addressing them, have to be unable to prevent destruction of mankind without SI. Just like in the Pascal’s original wager, the Thor and other deities are to be ignored by omission.
On how the SI does not look good, well, it does not look good to Holden Karnofsky, or me for that matter. Resistance to feedback loops is an extremely strong point of his.
On the rationality movement, here’s a quote from Holden.