Wow, when I read “should not be treated differently from those issues”, I assumed the intention was likely to be “child acting, indoctrination, etc., should be considered abuse and not tolerated by society”, a position I would tentatively support (tentatively due to lack of expertise).
Incidentally, I found many of the other claims to be at least plausible and discussion-worthy, if not probably true (and certainly not things that people should be afraid to say).
Yes, and isn’t it interesting to note that Robin Hanson sought his own higher degrees for the express purpose of giving his smart contrarian ideas (and way of thinking) more credibility?
One issue is that the same writing sends different signals to different people. I remember thinking about free will early in life (my parents thought they’d tease me with the age-old philosophical question) and, a little later in life, thinking that I had basically solved it—that people were simply thinking about it the wrong way. People around me often didn’t accept my solution, but I was never convinced that they even understood it (not due to stupidity, but failure to adjust their perspective in the right way), so my confidence remained high.
Later I noticed that my solution is a standard kind of “compatibilist” position, which is given equal attention by philosophers as many other positions and sub-positions, fiercely yet politely discussed without the slightest suggestion that it is a solution, or even more valid than other positions except as the one a particular author happens to prefer.
Later I noticed that my solution was also independently reached and exposited by Eliezer Yudkowsky (on Overcoming Bias before LW was created, if I remember correctly). The solution was clearly presented as such—a solution—and one which is easy to find with the right shift in perspective—that is, an answer to a wrong question. I immediately significantly updated the likelihood of the same author having further useful intellectual contributions, to my taste at least, and found the honesty thoroughly refreshing.
What is “intuition” but any set of heuristic approaches to generating conjectures, proofs, etc., and judging their correctness, which isn’t a naive search algorithm through formulas/proofs in some formal logical language? At a low level, all mathematics, including even the judgment of whether a given proof is correct (or “rigorous”), is done by intuition (at least, when it is done by humans). I think in everyday usage we reserve “intuition” for relatively high level heuristics, guesses, hunches, and so on, which we can’t easily break down in terms of simpler thought processes, and this is the sort of “intuition” that Terence Tao is discussing in those quotes. But we should recognize that even regarding the very basics of what it means to accept a proof is correct, we are using the same kinds of thought processes, scaled down.
Few mathematicians want to bother with actual formal logical proofs, whether producing them or reading them.
(And there’s an even subtler issue, that logicians don’t have any one really convincing formal foundation to offer, and Godel’s theorem makes it hard to know which ones are even consistent—if ZFC turned out to be inconsistent, would that mean that most of our math is wrong? Probably not, but since people often cite ZFC as being the formal logical basis for their work, what grounds do we we have for this prediction?)
Imagine you have an oracle that can determine if an arbitrary statement is provable in Peano arithmetic. Then you can try using it as a halting oracle: for an arbitrary Turing machine T, ask “can PA prove that there’s an integer N such that T makes N steps and then halts?”. If the oracle says yes, you know that the statement is true for standard integers because they’re one of the models of PA, therefore N is a standard integer, therefore T halts. And if the oracle says no, you know that there’s no such standard integer N because otherwise the oracle would’ve found a long and boring proof involving the encoding of N as SSS...S0, therefore T doesn’t halt. So your oracle can indeed serve as a halting oracle.
I don’t think this works. We can’t expect PA to decide whether or not any given Turing machine halts. For example, there is a machine which enumerates the theorems proven by PA and halts if it ever encounters a proof of 0=1. By incompleteness, PA will not prove that that this machine halts. (I’m assuming PA is consistent.) This argument works for any stronger consistent theory as well, such as ZFC or even much stronger ones. Note: I basically stole this argument from Scott Aaronson.
Note that this is different from the question of whether or not the halting problem is reducible to the set of theorems of PA (i.e. whether or not the oracle you’ve specified is enough to compute whether or not a given TM halts). It’s just that this particular approach does not give such an algorithm.
ETA: I was in error, see replies. In the OP, PA doesn’t need to prove that a non-halting machine doesn’t halt, it only needs to fail to prove that it halts (and it certainly does, if we believe PA is sound).
Scott Aaronson (a well-known complexity theorist) has written a survey article about exactly this question.
Upvoted for accuracy. My maternal grandmother is the same way and just the resulting politics in my mother’s family for how to deal with her empty shell are unpleasant, let alone the fact that she died so slowly hardly anyone acknowledged it as it was happening.
I think you mean, “When is it irrational to study rationality explicitly?”
For me there is always the lurking suspicion that my biggest reason for reading LessWrong is that it’s a whole lot of fun.
Do you mean what Eliezer calls the Machiavellian intelligence hypothesis? (That is, human intelligence evolved via runaway sexual selection—people who were politically competent were more reproductively successful, and as people got better and better at the game of politics, so too did the game get harder and harder, hence the feedback loop.)
Perhaps a species could evolve intelligence without such a mechanism, if something about their environment is just dramatically more complex in a peculiar way compared with ours, so that intelligence was worthwhile just for non-social purposes. The species’ ancestors may have been large predators on top of the food chain, where members are typically solitary and territorial and hunt over a large region of land, with its ecosystem strangely different from ours in some way that I’m not specifying (but you’d need a pretty clever idea about it to make this whole thing work).
These aliens wouldn’t be inherently social the way humans are, but they wouldn’t be antisocial either—they would have noticed that cooperation allows them to solve even more difficult problems and get even more from their environment. (Still in a pre-technological stage. Remember, something about this environment is such that it provides a nearly smooth, practically unbounded (in difficulty) array of puzzles/problems to solve with increasing rewards.) Eventually, they may build a civilization just to more efficiently organize to obtain these benefits, which will also allow them to advance technologically. (I’m probably drawing too sharp of a distinction between technological advancement and interaction with their strangely complex environment.)
They might lack the following trait that is very central to human nature: affection through familiarity. When we spend a lot of time with a person/thing/activity/idea, we grow fond of it. They might not have this, or they might not have it in such generality (e.g. they might still have it for, say, mates, if they reproduce sexually). They might also be a lot less biased than we are by social considerations, for the obvious reason, but perhaps they have less raw cognitive horsepower (their environment being no substitute for the human pastimes of politics and sex).
Recklessly speculative, obviously, but I gather that’s all we can hope to offer to Solvent.
Ah, now I feel extremely silly. The irony did not occur to me; it was simply a long comment that I agreed with completely, and I wasn’t satisfied merely upvoting it because it didn’t have any (other) upvotes yet at the time. Plus, doubly ironically, I was on a moral crusade to defend the karma system...
Right on.
Imagine a thousand professional philosophers would join lesswrong, or worse, a thousand creationists.
This test seems rather unfair—it’s pretty much a known that people who join LessWrong are likely to be already sympathetic to the LessWrong’s way of thinking. Besides, the only way to avoid a situation where thousands of dissidents joining could wreck the system is to have centralized power, i.e., more traditional moderation, which I think we were hoping to avoid for exactly the types of reasons that are being brought up here (politics, etc.).
The availability of a reputation system also discourages people to actually explain themselves by being able to let off steam or ignore cognitive dissonance by downvoting someone with a single mouse click.
True, but I think you have missed a positive incentive for response that is created by the reputation system in addition to the negative ones—a post/comment with a bad argument or worse creates an opportunity to win karma by writing a clear refutation, and I frequently see such responses being highly upvoted.
The initial population of a community might have been biased about something and the reputation system might provide a positive incentive to keep the bias and a negative incentive for those who disagree.
This is a problem, but based purely on my subjective experience it seems that people are more than willing to upvote posts that try to shatter a conventional LessWrong belief, and do so with good argumentation.
Why all the karma bashing? Yes, absolutely, people will upvote or downvote for political reasons and be heavily influenced by the name behind the post/comment. All the time. But as far as I can tell, politics is a problem with any evaluation system whatsoever, and karma does remarkably well. In my experience, post and comment scores are strongly correlated with how useful I find them, how much they contribute to my experience of the discussion. And the list of top contributors is full of people who have written posts that I have saved forever, that in many cases irreversibly impacted my thinking. The fact that EY is sometimes deservingly downvoted is a case in point. The abuse described in the original post is unfortunate, but overall the LessWrong system does a difficult job incredibly well.
I have seen too many discussions of Friendly AI, here and elsewhere (e.g. in comments at Michael Anissimov’s blog), detached from any concrete idea of how to do it....
At present, it is discussed in conjunction with a whole cornucopia of science fiction notions such as: immortality, conquering the galaxy, omnipresent wish-fulfilling super-AIs, good and bad Jupiter-brains, mind uploads in heaven and hell, and so on. Similarly, we have all these thought-experiments: guessing games with omniscient aliens, decision problems in a branching multiverse, “torture versus dust specks”. Whatever the ultimate relevance of such ideas, it is clearly possible to divorce the notion of Friendly AI from all of them....
SIAI, in discussing the quest for the right goal system, emphasizes the difficulties of this process and the unreliability of human judgment. Their idea of a solution is to use artificial intelligence to neuroscientifically deduce the actual algorithmic structure of human decision-making, and to then employ a presently nonexistent branch of decision theory to construct a goal system embodying ideals implicit in the unknown human cognitive algorithms.
In short, there is a dangerous and almost universal tendency to think about FAI (and AGI generally) primarily in far mode. Yes!
However, I’m less enamored with the rest of your post. The reason is that building AGI is simply an altogether higher-risk activity than traveling to the moon. Using “build a chemical powered rocket” as your starting point for getting to the moon is reasonable in part because the worst that could plausibly happen is that the rocket will blow up and kill a lot of volunteers who knew what they were getting into. In the case of FAI, Eliezer Yudkowsky has taken great pains to show that the slightest, subtlest mistake, one which could easily pass through any number of rounds of committee decision making, coding, and code checking, could lead to existence failure for humanity. He has also taken pains to show that approaches to the problem which entire committees have in the past thought were a really good idea, would also lead to such a disaster. As far as I can tell, the LessWrong consensus agrees with him on the level of risk here, at least implicitly.
There is another approach. My own research pertains to automated theorem proving, and its biggest application, software verification. We would still need to produce a formal account of the invariants we’d want the AGI to preserve, i.e., a formal account of what it means to respect human values. When I say “formal”, I mean it: a set of sentences in a suitable formal symbolic logic, carefully chosen to suit the task at hand. Then we would produce a mathematical proof that our code preserves the invariants, or, more likely we would use techniques for producing the code and the proof at the same time. So we’d more or less have a mathematical proof that the AI is Friendly. I don’t know how the SIAI is trying to think about the problem now, exactly, but I don’t think Eliezer would be satisfied by anything less certain than this sort of approach.
Not that this outline, at this point, is satisfactory. The formalization of human value is a massive problem, and arguably where most of the trouble lies anyway. I don’t think anyone’s ever solved anything even close to this. But I’d argue that this outline does clarify matters a bit, because we have a better idea what a solution to this problem would look like. And it makes it clear how dangerous the loose approach recommended here is: virtually all software has bugs, and a non-verified recursively self-improving AI could magnify a bug in its value system until it no better approximates human values than does paperclip-maximizing. Moreover, the formal proof doesn’t do anyone a bit of good if the invariants were not designed correctly.
I don’t know if there’s a big reason behind it, but because .com is so entrenched as the “default” TLD, I think it’s probably best to be LessWrong.com rather than LessWrong.net or any other choice, simply because ”LessWrong.com″ is more likely to be correctly remembered by people who hear of it briefly, or correctly guessed by people who heard “Less Wrong” and randomly take a stab at their brower’s navigation bar.
I admit this point may be relatively trivial since it’s the first google hit for “less wrong” and that’s probably how a lot of people look for it who’ve only heard of it.
Agreed, would like to see this again in the form of a top-level post. This cuts at the heart of one of the most important sets of lies we are told by society and expected to repeat.
Personally, I’d need see good reasons to expect the charity I’m donating to is going to have a significant positive impact before I consider donating, relative to other charities I might be able to find on my own. Inefficiency, corruption, and poor choice of target are major concerns. (One example of the latter issuemight be donating to help US poor when it’s possible to just as efficiently help people who are far worse off somewhere else). Also the mechanism by which to help maybe poorly thought out. (Do the poor really need education, as opposed to other more concrete things? I’m not giving an answer, just saying I’d need to see one before I donated.)
I think many here are already aware of GiveWell, an organization which evaluates charities on many of these criteria, and is nice enough to publish the details of their analysis. GiveWell finds that overwhelming numbers of charities fare very poorly. Helpfully, they also say very clearly what they think the most effective charity to donate to is, how effective they think it is, and why. (Currently VillageReach, last I checked, which works on very basic medical supply infrastructure in Africa.)
EDIT: Should have paid more attention to what you actually said. Obviously if you are already earning these “reward points” then spending them on donations is no additional cost to you. However, the questions about effectiveness stand, and based on analyses I’ve seen, many charities are so poor that you’d be obviously doing more good spending the same money on yourself. Or using your reward points on some other trivial reward. (Technically, in terms of opportunity cost, spending the reward points is still like spending money, if you can spend them on other things you would spend money on.)
So Alcor runs at a loss and doesn’t actually freeze that many people because it can’t afford to?
This seems extremely misleading. Unless I’m very much mistaken, Alcor cryopreserves every one of its members upon their legal death to the absolute best of its ability, as indeed they are contractually obligated to do. They even now have an arrangement with Suspended Animation so that an SA team can provide SST (standby, stabilization, and transport) in cases where Alcor cannot easily get there in time. (SA is a for-profit company founded to provide exactly this type of service; they also have a working relationship with Cryonics Institute of a different sort.)
To my understanding, Alcor runs at a “loss” (in quotes because donations are just as much a source of revenue as membership and cryopreservation fees) for similar reasons that any small-but-growing business would: because growth is the best way to ensure long-term stability, and keeping the price of cryopreservation as low as possible given the other constraints promotes growth.
Finally, I think it’s worth mentioning Alcor created the Patient Care Trust fund and gave it legal independence specifically to prevent funds from being usurped that are intended to go toward the care and eventual resuscitation of Alcor’s patients, regardless of Alcor’s future financial situation. Even if Alcor collapses financially, these funds are contractually mandated to be used toward protecting Alcor’s patients, and maximizing their continued chances of being successfully revived (for example, by transferring them to another cryonics organization).
I think you may be missing a silent majority of people who passively judge cryonics as unlikely to work, and do not develop strong feelings or opinions about it besides that, because they have no reason to. I think this category, together with “too expensive to think about right now”, forms the bulk of intelligent friends with whom I’ve discussed cryonics.