This jumped out instantly when I looked at the charts: Your prior and evidence can’t possibly both be correct at the same time. Everywhere the prior has non-negligible density has negligible likelihood. Everywhere that has substantial likelihood has negligible prior density. If you try multiplying the two together to get a compromise probability estimate instead of saying “I notice that I am confused”, I would hold this up as a pretty strong example of the real sin that I think this post should be arguing against, namely that of trying to use math too blindly without sanity-checking its meaning.
Quick comment two:
I’m a major fan of Down-To-Earthness as a virtue of rationality, and I have told other SIAI people over and over that I really think they should stop using “small probability of large impact” arguments. I’ve told cryonics people the same. If you can’t argue for a medium probability of a large impact, you shouldn’t bother.
Part of my reason for saying this is, indeed, that trying to multiply a large utility interval by a small probability is an argument-stopper, an attempt to shut down further debate, and someone is justified in having a strong prior, when they see an attempt to shut down further debate, that further argument if explored would result in further negative shifts from the perspective of the side trying to shut down the debate.
With that said, any overall scheme of planetary philanthropic planning that doesn’t spend ten million dollars annually on Friendly AI is just stupid. It doesn’t just fail the Categorical Imperative test of “What if everyone did that?”, it fails the Predictable Retrospective Stupidity test of, “Assuming civilization survives, how incredibly stupid will our descendants predictably think we were to do that?”
Of course, I believe this because I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI, and I don’t think there’s nothing that can be done about that, either.
I would summarize my quick reply by saying,
“I agree that it’s a drastic warning sign when your decision process is spending most of its effort trying to achieve unprecedented outcomes of unquantifiable small probability, and that what I consider to be down-to-earth common sense is a great virtue of a rationalist. That said, down-to-earth common-sense says that AI is a screaming emergency at this point in our civilization’s development, and I don’t consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy.”
I don’t consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy
What about people who do think SIAI’s probability of success is small? Perhaps they have different intuitions about how hard FAI is, or don’t have enough knowledge to make an object-level judgement so they just apply the absurdity heuristic. Being one of those people, I think it’s still an important question whether it’s rational to support SIAI given a small estimate of probability of success, even if SIAI itself doesn’t want to push this line of inquiry too hard for fear of signaling that their own estimate of probability of success is low.
Leaving aside Aumann questions: If people like that think that the Future of Humanity Institute, work on human rationality, or Giving What We Can has a large probability of catalyzing the creation of an effective institution, they should quite plausibly be looking there instead. “I should be doing something I think is at least medium-probably remedying the sheerly stupid situation humanity has gotten itself into with respect to the intelligence explosion” seems like a valuable summary heuristic.
If you can’t think of anything medium-probable, using that as an excuse to do nothing is unacceptable. Figure out which of the people trying to address the problem seem most competent and gamble on something interesting happening if you give them more money. Money is the unit of caring and I can’t begin to tell you how much things change when you add more money to them. Imagine what the global financial sector would look like if it was funded to the tune of $600,000/year. You would probably think it wasn’t worth scaling up Earth’s financial sector.
If you can’t think of anything medium-probable, using that as an excuse to do nothing is unacceptable.
That’s my gut feeling as well, but can we give a theoretical basis for that conclusion, which might also potentially be used to convince people who can’t think of anything medium-probable to “do something”?
My current thoughts are
I assign some non-zero credence to having an unbounded utility function.
Bostrom and Toby’s moral parliament idea seems to be the best that we have about how to handle moral uncertainty.
If Pascal’s wager argument works, and to the extent that I have a faction representing unbounded utility in my moral parliament, I ought to spend a fraction of my resources on Pascal’s wager type “opportunities”
If Pascal’s wager argument works, I should pick the best wager to bet on, which intuitively could well be “push for a positive Singularity”
But it’s not clear that Pascal’s wager argument works or what could be the justification for thinking that “push for a positive Singularity” is the best wager. We also don’t have any theory to handle this kind of philosophical uncertainty.
Given all this, I still have to choose between “do nothing”, “push for positive Singularity”, or “investigate Pascal’s wager”. Is there any way, in this decision problem, to improve upon going with my gut?
Anyway, I understand that you probably have reasons not to engage too deeply with this line of thought, so I’m mostly explaining where I’m currently at, as well as hoping that someone else can offer some ideas.
Imagine what the global financial sector would look like if it was funded to the tune of $600,000/year. You would probably think it wasn’t worth scaling up Earth’s financial sector.
And one might even be right about that.
A better analogy might be if regulation of the global financial sector were funded at 600k/yr.
Money is the unit of caring and it really is impossible to overstate how much things change when you add money to them.
Can you give an example relevant to the context at hand to illustrate what you have in mind? I don’t necessarily disagree, but I presently think that there’s a tenable argument that money is seldom the key limiting factor for philanthropic efforts in the developed world.
BTW, note that I deleted the “impossible to overstate” line on grounds of its being false. It’s actually quite possible to overstate the impact of adding money. E.g., “Adding one dollar to this charity will CHANGE THE LAWS OF PHYSICS.”
What sort of key limiting factors do you have in mind that are untouched by money? Every limiting factor I can think of, whether it’s lack of infrastructure or corruption or lack of political will in the West, is something that you could spend money on doing something about.
If nothing else, historical examples show that huge amounts of money lobbed at a cause can go to waste or do more harm than good (e.g. the Iraq war as a means to improve relations with the middle East).
Eliezer and I were both speaking in vague terms; presumably somebody intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute can levy money toward some positive expected change in a given direction. There remains the question about the conversion factor between money and other goods and how quickly it changes at the margin; it could be negligible in a given instance.
The main limiting factor that I had in mind was human capital: an absence of people who are sufficiently intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute.
I would add that a group of such people would have substantially better than average odds of attracting sufficient funding from some philanthropist; further diminishing the value of donations (on account of fungibility).
I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI
That’s the probability statement in his post. He didn’t mention the probability of SIAI’s success, and hasn’t previously when I’ve emailed him or asked in public forums, nor has he at any point in time that I’ve heard. Shortly after I asked, he posted When (Not) To Use Probabilities.
You might even be justified in refusing to use probabilities at this point. In all honesty, I really don’t know how to estimate the probability of solving an impossible problem that I have gone forth with intent to solve; in a case where I’ve previously solved some impossible problems, but the particular impossible problem is more difficult than anything I’ve yet solved, but I plan to work on it longer, etcetera.
People ask me how likely it is that humankind will survive, or how likely it is that anyone can build a Friendly AI, or how likely it is that I can build one. I really don’t know how to answer. I’m not being evasive; I don’t know how to put a probability estimate on my, or someone else, successfully shutting up and doing the impossible. Is it probability zero because it’s impossible? Obviously not. But how likely is it that this problem, like previous ones, will give up its unyielding blankness when I understand it better? It’s not truly impossible, I can see that much. But humanly impossible? Impossible to me in particular? I don’t know how to guess. I can’t even translate my intuitive feeling into a number, because the only intuitive feeling I have is that the “chance” depends heavily on my choices and unknown unknowns: a wildly unstable probability estimate.
But it’s not clear whether Eliezer means that he can’t even translate his intuitive feeling into a word like “small” or “medium”. I thought the comment I was replying to was saying that SIAI had a “medium” chance of success, given:
If you can’t argue for a medium probability of a large impact, you shouldn’t bother.
and
I don’t consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy
But perhaps I misinterpreted? In any case, there’s still the question of what is rational for those of us who do think SIAI’s chance of success is “small”.
Sufficiently-Friendly AI can be hard for SIAI-now but easy or medium for non-SIAI-now (someone else now, someone else future, SIAI future). I personally believe this, since SIAI-now is fucked up (and SIAI-future very well will be too). (I won’t substantiate that claim here.) Eliezer didn’t talk about SIAI specifically. (He probably thinks SIAI will be at least as likely to succeed as anyone else because he thinks he’s super awesome, but it can’t be assumed he’d assert that with confidence, I think.)
SingInst seems a lot better since I wrote that comment; you and Luke are doing some cool stuff. Around August everything was in a state of disarray and it was unclear if you’d manage to pull through.
Regarding the graphs, I assumed that they were showing artificial examples so that we could viscerally understand at a glance what the adjustment does, not that this is what the prior and evidence should look like in a real case.
This jumped out instantly when I looked at the charts: Your prior and evidence can’t possibly both be correct at the same time. Everywhere the prior has non-negligible density has negligible likelihood. Everywhere that has substantial likelihood has negligible prior density. If you try multiplying the two together to get a compromise probability estimate instead of saying “I notice that I am confused”, I would hold this up as a pretty strong example of the real sin that I think this post should be arguing against, namely that of trying to use math too blindly without sanity-checking its meaning.
(I deleted my response to this following othonormal’s comments; see this one for my revised thought here.)
Of course, I believe this because I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI, and I don’t think there’s nothing that can be done about that, either.
Why do you think that there’s something that can be done about it?
I disagree. It can be rational to shift subjective probabilities by many orders of magnitude in response to very little new information.
What your example looks like is a nearly uniform prior over a very large space- nothing’s wrong when we quickly update to believe that yesterday’s lottery numbers are 04-15-21-31-36.
But the point where you need to halt, melt, and catch fire is if your prior assigns the vast majority of the probability mass to a small compact region, and then the evidence comes along and lands outside that region. That’s the equivalent of starting out 99.99% confident that you know tomorrow’s lottery numbers will begin with 01-02-03, and being proven wrong.
Yes, you’re right, I wasn’t thinking clearly, thanks for catching me. I think there’s something to what I was trying to say, but I need to think about it through more carefully. I find the explanation that you give in your other comment convincing (that the point of the graphs is to clearly illustrate the principle).
10 million dollars buys quite a few programmers. SIAI is presently nowhere near that amount of money, and doesn’t look likely to be any time soon. When does it make sense to start talking to volunteer programmers? Presumably, when the risk of opening up the project is less than the risk of failing to get it done before someone else does it wrong.
When is “soon” for these purposes. It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort. We certainly would hire any such programmers now.
It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort.
(Please don’t upvote this comment till you’ve read it fully; I’m interpreting upvotes in a specific way.) Question for anyone on LW: If I had a viable preliminary Friendly AI research program, aimed largely at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty of Friendly AI for various values of “Friendly”, and wrote clearly and concretely about the necessary steps in pursuing this analysis, and listed and described a small number of people (less than 5, but how many could actually be convinced to focus on doing the analysis would depend on funds) who I know of who could usefully work on such an analysis, and committed to have certain summaries published online at various points (after actually considering concrete possibilities for failure, planning fallacy, etc., like real rationalists should), and associated with a few (roughly 5) high status people (people like Anders Sandberg or Max Tegmark, e.g. by convincing them to be on an advisory board), would this have a decent chance of causing you or someone you know to donate $100 or more to support this research program? (I have a weird rather mixed reputation among the greater LW community, so if that affects you negatively please pretend that someone with a more solid reputation but without super high karma is asking this question, like Steven Kaas.) You can upvote for “yes” and comment about any details, e.g. if you know someone who would possibly donate significantly more than $100. (Please don’t downvote for “no”, ’cuz that’s the default answer and will drown out any “yes” ones.)
I have a weird rather mixed reputation among the greater LW community, so if that affects you negatively please pretend that someone with a more solid reputation but without super high karma is asking this question, like Steven Kaas.
Unless you would be much less involved in this potential program than the comment indicates, this seems like an inappropriate request. If people view you negatively due to your posting history, they should absolutely take that information into account in assessing how likely they would be to provide financial support to such a program (assuming that the negative view is based on relevant considerations such as your apparent communication or reasoning skills as demonstrated in your comments).
Fair enough, but in light of your phrasing in both the original comment (“If I [did the following things]”) and your comment immediately following it (quoted below; emphasis added), it certainly appeared to me that you seemed to be describing a significant role for yourself, even though your proposal was general overall.
(Some people, including me, would really like it if a competent and FAI-focused uber-rationalist non-profit existed. I know people who will soon have enough momentum to make this happen. I am significantly more familiar with the specifics of FAI (and of hardcore SingInst-style rationality) than many of those people and almost anyone else in the world, so it’d be necessary that I put a lot of hours into working with those who are higher status than me and better at getting things done but less familiar with technical Friendliness. But I have many other things I could be doing. Hence the question.)
Sorry, could you say again what exactly you want to do? I mean, what’s the output here that the money is paying for; a Friendly AI, a theory that can be used to construct a Friendly AI, or an analysis that purports to say whether or not Friendly AI is “feasible”, or what?
Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things.
I was trying to quickly gauge vague interest in a vague notion. I think that my original comment was at roughly the most accurate and honest level of vagueness (i.e. “aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]”). Value of information regarding difficulty of Friendly-ish AI is high, but research into that question is naturally tied to Friendly AI theory itself. I’m thinking… Goedel machine stability more than ambient decision theory, history of computation more than any kind of validity semantics. To some extent it depends on who plans to actually work on what stuff from the open problems lists. There are many interesting technical threads that people might start pulling on soon, and it’s unclear to me to what extent they actually will pull on them or to what extent pulling on them will give us a better sense of the problem.
[Stuff it would take too many paragraphs to explain why it’s worth pointing out specifically:] Theory of justification seems to be roughly as developed as theory of computation was before the advent of Leibniz; Leibniz saw a plethora of connections between philosophy, symbolic logic, and engineering and thus developed some correctly thematically centered proto-theory. I’m trying to make a Leibniz, and hopefully SingInst can make a Turing. (Two other roughly analogous historical conceptual advances are natural selection and temperature.)
Well, my probability that you could or would do anything useful, given money, just dropped straight off a cliff. But perhaps you’re just having trouble communicating. That is to say: What the hell are you talking about.
If you’re going to ask for money on LW, plain English response, please: What’s the output here that the money is paying for; (1) a Friendly AI, (2) a theory that can be used to construct a Friendly AI, or (3) an analysis that purports to say whether or not Friendly AI is “feasible”? Please pick one of the pre-written options; I now doubt your ability to write your response ab initio.
Dude, it’s right there: “feasibility and difficulty”, in this sentence which I am now repeating for the second time:
aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]”).
(Bold added for emphasis, annotations in [brackets] were in the original.)
The next sentence:
Value of information regarding difficulty of Friendly-ish AI is high, but research into that question is naturally tied to Friendly AI theory itself.
Or if you really need it spelled out for you again and again, the output would primarily be (3) but secondarily (2) as you need some of (2) to do (3).
Because you clearly need things pointed out multiple times, I’ll remind you that I put my response in the original comment that you originally responded to, without the later clarifications that I’d put in for apparently no one’s benefit:
If I had a viable preliminary Friendly AI research program, aimed largely at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty of Friendly AI for various values of “Friendly” [...]
(Those italics were in the original comment!)
If you’re going to ask for money on LW
I wasn’t asking for money on Less Wrong! As I said, “I was trying to quickly gauge vague interest in a vague notion.” What the hell are you talking about.
I now doubt your ability to write your response ab initio.
I’ve doubted your ability to read for a long time, but this is pretty bad. The sad thing is you’re probably not doing this intentionally.
I think the problem here is that your posting style, to be frank, often obscures your point.
In most cases, posts that consist of a to-the-point answer followed by longer explanations use the initial statement to make a concise case. For instance, in this post, my first sentence sums up what I think about the situation and the rest explains that thought in more detail so as to convey a more nuanced impression.
By contrast, when Eliezer asked “What’s the output here that the money is paying for,” your first sentence was “Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things.” This does not really answer his question, and while you clarify this with your later points, the overall message is garbled.
The fact that your true answer is buried in the middle of a paragraph does not really help things much. Though I can see what you are trying to say, I can’t in good and honest conscience describe it as clear. Had you answered, on the other hand, “Money would pay for the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI...” as your first sentence, I think your post would have been more clear and more likely to be understood.
The sentences I put before the direct answer to Eliezer’s question were meant to correct some of Eliezer’s misapprehensions that were more fundamental than the object of his question. Eliezer’s infamous for uncharitably misinterpreting people and it was clear he’d misinterpreted some key aspects of my original comment, e.g. my purpose in writing it. If I’d immediately directly answered his question that would have been dishonest; it would have contributed further to his having a false view of what I was actually talking about. Less altruistically it would be like I was admitting to his future selves or to external observers that I agreed that his model of my purposes was accurate and that this model could legitimately be used to assert that I was unjustified in any of many possible ways. Thus I briefly (a mere two sentences) attempted to address what seemed likely to be Eliezer’s underlying confusions before addressing his object level question. (Interestingly Eliezer does this quite often, but unfortunately he often assumes people are confused in ways that they are not.)
Given these constraints, what should I have done? In retrospect I should have gone meta, of course, like always. What else?
Given those constraints, I would probably write something like “Money would pay for marginal output in the form of increased collaboration on the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI” for my first sentence and elaborate as strictly necessary. That seems rather more cumbersome than I’d like, but it’s also a lot of information to try and convey in one sentence!
Alternatively, I would consider something along the lines of “Money would pay for the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI, but not directly—since the best Friendliness-cognizant x-rationalists would likely already be working on similar things, the money would go towards setting up better communication, coordination, and collaboration for that group.”
That said, I am unaware of any reputation Eliezer has in the field of interpreting people, and personally haven’t received the impression that he’s consistently unusually bad or uncharitable at it. Then again, I have something of a reputation—at least in person—for being too charitable, so perhaps I’m being too light on Eliezer (or you?) here.
I think the problem here is that your posting style, to be frank, often obscures your point.
I acknowledge this. But it seems to me that the larger problem is that Eliezer simply doesn’t know how to read what people actually say. Less Wrong mostly doesn’t either, and humans in general certainly don’t. This is a very serious problem with LW-style rationality (and with humanity). There are extremely talented rationalists who do not have this problem; it is an artifact of Eliezer’s psychology and not of the art of rationality.
It’s hardly fair to blame the reader when you’ve got “sentences” like this:
I think that my original comment was at roughly the most accurate and honest level of vagueness (i.e. “aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]”).
That was the second version of the sentence, the first one had much clearer syntax and even italicized the answer to Eliezer’s subsequent question. It looks the way it does because Eliezer apparently couldn’t extract meaning out of my original sentence despite it clearly answering his question, so I tried to expand on the relevant points with bracketed concrete examples. Here’s the original:
If I had a viable preliminary Friendly AI research program, aimed largely at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty of Friendly AI for various values of “Friendly” [...]
What you say might be true, but this one example is negligible compared to the mountain of other evidence concerning inability to read much more important things (which are unrelated to me). I won’t give that evidence here.
If that’s indeed the case (I haven’t noticed this flaw myself), I suggest that you write articles (or perhaps commission/petition others to have them written) describing this flaw and how to correct it. Eliminating such a flaw or providing means of averting it would greatly aid LW and the community in general.
Unfortunately that is not currently possible for many reasons, including some large ones I can’t talk about and that I can’t talk about why I can’t talk about. I can’t see any way that it would become possible in the next few years either. I find this stressful; it’s why I make token attempts to communicate in extremely abstract or indirect ways with Less Wrong, despite the apparent fruitlessness. But there’s really nothing for it.
Unrelated public announcement: People who go back and downvote every comment someone’s made, please, stop doing that. It’s a clever way to pull information cascades in your direction but it is clearly an abuse of the content filtering system and highly dishonorable. If you truly must use such tactics, downvoting a few of your enemy’s top level posts is much less evil; your enemy loses the karma and takes the hint without your severely biasing the public perception of your enemy’s standard discourse. Please.
(I just lost 150 karma points in a few minutes and that’ll probably continue for awhile. This happens a lot.)
Unfortunately that is not currently possible for many reasons, including some large ones I can’t talk about and that I can’t talk about why I can’t talk about.
Why can’t you talk about why you can’t talk about them?
I’m not a big fan of the appeal to secret reasons, so I think I’m going to have pull out of this discussion. I will note, however, that you personally seem to be involved in more misunderstandings than the average LW poster, so while it’s certainly possible that your secret reasons are true and valid and Eliezer just sucks at reading or whatever, you may want to clarify certain elements of your own communication as well.
I unfortunately predict that “going more meta” will not be strongly received here.
Unfortunately that is not currently possible for many reasons, including some large ones I can’t talk about and that I can’t talk about why I can’t talk about.
Are we still talking about improving general reading comprehension? What could possibly be dangerous about that?
(Some people, including me, would really like it if a competent and FAI-focused uber-rationalist non-profit existed. I know people who will soon have enough momentum to make this happen. I am significantly more familiar with the specifics of FAI (and of hardcore SingInst-style rationality) than many of those people and almost anyone else in the world, so it’d be necessary that I put a lot of hours into working with those who are higher status than me and better at getting things done but less familiar with technical Friendliness. But I have many other things I could be doing. Hence the question.)
Does “FAI-focused” mean what I called code first? What are your thoughts on that post and its followup? What is this new non-profit planning to do differently from SIAI and why? What are the other things that you could be doing?
Jah. Well, at least determining whether or not “code first” is even reasonable, yeah, which is a difficult question in itself and only partially tied in with making direct progress on FAI.
What are your thoughts on that post and its followup?
You seem to have missed Oracle AI? (Eliezer’s dismissal of it isn’t particularly meaningful.) I agree with your concerns. This is why the main focus would at least initially be determining whether or not “code first” is a plausible approach (difficulty-wise and safety-wise). The value of information on that question is incredibly high and as you’ve pointed out it has not been sufficiently researched.
What is this new non-profit planning to do differently from SIAI and why?
Basically everything. SingInst is focused on funding a large research program and gaining the prestige necessary to influence (academic) culture and academic and political policy. They’re not currently doing any research on Friendly AI, and their political situation is such that I don’t expect them to be able to do so effectively for a while, if ever. I will not clarify this. (Actually their research associates are working on FAI-related things, but SingInst doesn’t pay them to do that.)
What are the other things that you could be doing?
Learning, mostly. Working with an unnamed group of x-risk-cognizant people that LW hasn’t heard of, in a way unrelated to their setting up a non-profit.
They’re not currently doing any research on Friendly AI, and their political situation is such that I don’t expect them to be able to do so effectively for a while, if ever.
My understanding is that SIAI recently tried to set up a new in-house research team to do preliminary research into FAI (i.e., not try to build an FAI yet, but just do whatever research that might be eventually helpful to that project). This effort didn’t get off the ground, but my understanding again is that it was because the researchers they tried to recruit had various reasons for not joining SIAI at this time. I was one of those they tried to recruit, and while I don’t know what the others’ reasons were, mine were mostly personal and not related to politics.
You must also know all this, since you were involved in this effort. So I’m confused why you say SIAI won’t be doing effective research on FAI due to its “political situation”. Did the others not join SIAI because they thought SIAI was in a bad political situation? (This seems unlikely to me.) Or are you referring to the overall lack of qualified, recruitable researchers as a “political situation”? If you are, why do you think this new organization would be able to do better?
(Or did you perhaps not learn the full story, and thought SIAI stopped this effort for political reasons?)
The answer to your question isn’t among your list of possible answers. The recent effort to start an in-house research team was a good attempt and didn’t fail for political reasons. I am speaking of other things. However I want to take a few weeks off from discussion of such topics; I seem to have given off the entirely wrong impression and would prefer to start such discussion anew in a better context, e.g. one that better emphasizes cooperation and tentativeness rather than reactionary competition. My apologies.
I was trying to quickly gauge vague interest in a vague notion.
I won’t give that evidence here.
(I won’t substantiate that claim here.)
I will not clarify this.
The answer to your question isn’t among your list of possible answers.
I find this stressful; it’s why I make token attempts to communicate in extremely abstract or indirect ways with Less Wrong, despite the apparent fruitlessness. But there’s really nothing for it.
It’s a good heuristic, but can be very psychologically difficult. E.g. if you think that not even trying to communicate will be seen as unjustified in retrospect even if people should know that there was no obvious way for you to communicate. This has happened enough to me that the thought of just giving up on communication is highly aversive; my fear of being blamed for not preventing others to take unjustified actions (that will cause me, them, and the universe counterfactually-needless pain) is too great. But whatever, I’m starting to get over it.
Like, I remember a pigeon dying… people dying… a girl who starved herself… a girl who cut herself… a girl who wanted to commit suicide… just, trust me, there are reasons that I’m afraid. I could talk about those reasons but I’d rather not. It’s just, if you don’t even make a token gesture it’s like you don’t even care at all, and it’s easier to be unjustified in a way that can be made to look sorta like caring than in a way that looks like thoughtlessness or carelessness.
(ETA: A lot of the time when people give me or others advice I mentally translate it to “The solution is simple, just shut up and be evil.”.)
I have no particular attachments to SIAI and would love to see a more effective Singularitarian organization formed if that were possible. I’m just having genuine trouble understanding why you think this new proposed organization will be able to do more effective FAI research. Perhaps you could use these few weeks off to ask some trusted advisors how to better communicate this point. (I understand you have sensitive information that you can’t reveal, but I’m guessing that you can do better even within that constraint.)
Perhaps you could use these few weeks off to ask some trusted advisors how to better communicate this point.
This is exactly the strategy I’ve decided to enact, e.g. talking to Anna. Thanks for being… gentle, I suppose? I’ve been getting a lot of flak lately, it’s nice to get some non-insulting advice sometimes. :)
(Somehow I completely failed to communicate the tentativeness of the ideas I was throwing around; in my head I was giving it about a 3% chance that I’d actually work on helping build an organization but I seem to have given off an impression of about 30%. I think this caused everyone’s brains to enter politics mode, which is not a good mode for brains to be in.)
I have no particular attachments to SIAI and would love to see a more effective Singularitarian organization formed if that were possible.
It’s rather strange how the SIAI is secretive. The military projects are secretive, the commercial projects are secretive alas: so few value transparency. An open project would surely do better, through being more obviously trustworthy and accountable, being better able to use talent across the internet, etc. I figure if the SIAI persists in not getting to grips with this issue, some other organisation will.
Well, looking at the post on SIAI finances from a few months back, SIAI’s annual revenue is growing at a rate of roughly 100k/year, and thus would take nearly a century to reach 10 million / year. Of course, all sorts of things could change these numbers. Eliezer has stated that he believes that UFAI will happen somewhat sooner than a century.
Since SIAI does seem to have at least some unused budget for programmers now, I emailed a friend who might be a good fit for a research associate to suggest that he apply.
It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort.
ADBOC. SIAI should never be content with any funding level. It should get the resources to hire, bribe or divert people who may otherwise make breakthroughs on UAI, and set them to doing anything else.
That intuition seems to me to follow from the probably false assumption that if behavior X would, under some circumstances, be utility maximizing, it is also likely to be utility maximizing to fund a non-profit to engage in behavior X. SIAI isn’t a “do what seems to us to maximize expected utility” organization because such vague goals don’t make for a good organizational culture. Organizing and funding research into FAI and research inputs into FAI, plus doing normal non-profit fund-raising and outreach, that’s a feasible non-profit directive.
Quick comment one:
This jumped out instantly when I looked at the charts: Your prior and evidence can’t possibly both be correct at the same time. Everywhere the prior has non-negligible density has negligible likelihood. Everywhere that has substantial likelihood has negligible prior density. If you try multiplying the two together to get a compromise probability estimate instead of saying “I notice that I am confused”, I would hold this up as a pretty strong example of the real sin that I think this post should be arguing against, namely that of trying to use math too blindly without sanity-checking its meaning.
Quick comment two:
I’m a major fan of Down-To-Earthness as a virtue of rationality, and I have told other SIAI people over and over that I really think they should stop using “small probability of large impact” arguments. I’ve told cryonics people the same. If you can’t argue for a medium probability of a large impact, you shouldn’t bother.
Part of my reason for saying this is, indeed, that trying to multiply a large utility interval by a small probability is an argument-stopper, an attempt to shut down further debate, and someone is justified in having a strong prior, when they see an attempt to shut down further debate, that further argument if explored would result in further negative shifts from the perspective of the side trying to shut down the debate.
With that said, any overall scheme of planetary philanthropic planning that doesn’t spend ten million dollars annually on Friendly AI is just stupid. It doesn’t just fail the Categorical Imperative test of “What if everyone did that?”, it fails the Predictable Retrospective Stupidity test of, “Assuming civilization survives, how incredibly stupid will our descendants predictably think we were to do that?”
Of course, I believe this because I think the creation of smarter-than-human intelligence has a (very) large probability of an (extremely) large impact, and that most of the probability mass there is concentrated into AI, and I don’t think there’s nothing that can be done about that, either.
I would summarize my quick reply by saying,
“I agree that it’s a drastic warning sign when your decision process is spending most of its effort trying to achieve unprecedented outcomes of unquantifiable small probability, and that what I consider to be down-to-earth common sense is a great virtue of a rationalist. That said, down-to-earth common-sense says that AI is a screaming emergency at this point in our civilization’s development, and I don’t consider myself to be multiplying small probabilities by large utility intervals at any point in my strategy.”
What about people who do think SIAI’s probability of success is small? Perhaps they have different intuitions about how hard FAI is, or don’t have enough knowledge to make an object-level judgement so they just apply the absurdity heuristic. Being one of those people, I think it’s still an important question whether it’s rational to support SIAI given a small estimate of probability of success, even if SIAI itself doesn’t want to push this line of inquiry too hard for fear of signaling that their own estimate of probability of success is low.
Leaving aside Aumann questions: If people like that think that the Future of Humanity Institute, work on human rationality, or Giving What We Can has a large probability of catalyzing the creation of an effective institution, they should quite plausibly be looking there instead. “I should be doing something I think is at least medium-probably remedying the sheerly stupid situation humanity has gotten itself into with respect to the intelligence explosion” seems like a valuable summary heuristic.
If you can’t think of anything medium-probable, using that as an excuse to do nothing is unacceptable. Figure out which of the people trying to address the problem seem most competent and gamble on something interesting happening if you give them more money. Money is the unit of caring and I can’t begin to tell you how much things change when you add more money to them. Imagine what the global financial sector would look like if it was funded to the tune of $600,000/year. You would probably think it wasn’t worth scaling up Earth’s financial sector.
That’s my gut feeling as well, but can we give a theoretical basis for that conclusion, which might also potentially be used to convince people who can’t think of anything medium-probable to “do something”?
My current thoughts are
I assign some non-zero credence to having an unbounded utility function.
Bostrom and Toby’s moral parliament idea seems to be the best that we have about how to handle moral uncertainty.
If Pascal’s wager argument works, and to the extent that I have a faction representing unbounded utility in my moral parliament, I ought to spend a fraction of my resources on Pascal’s wager type “opportunities”
If Pascal’s wager argument works, I should pick the best wager to bet on, which intuitively could well be “push for a positive Singularity”
But it’s not clear that Pascal’s wager argument works or what could be the justification for thinking that “push for a positive Singularity” is the best wager. We also don’t have any theory to handle this kind of philosophical uncertainty.
Given all this, I still have to choose between “do nothing”, “push for positive Singularity”, or “investigate Pascal’s wager”. Is there any way, in this decision problem, to improve upon going with my gut?
Anyway, I understand that you probably have reasons not to engage too deeply with this line of thought, so I’m mostly explaining where I’m currently at, as well as hoping that someone else can offer some ideas.
And one might even be right about that.
A better analogy might be if regulation of the global financial sector were funded at 600k/yr.
Can you give an example relevant to the context at hand to illustrate what you have in mind? I don’t necessarily disagree, but I presently think that there’s a tenable argument that money is seldom the key limiting factor for philanthropic efforts in the developed world.
BTW, note that I deleted the “impossible to overstate” line on grounds of its being false. It’s actually quite possible to overstate the impact of adding money. E.g., “Adding one dollar to this charity will CHANGE THE LAWS OF PHYSICS.”
What sort of key limiting factors do you have in mind that are untouched by money? Every limiting factor I can think of, whether it’s lack of infrastructure or corruption or lack of political will in the West, is something that you could spend money on doing something about.
If nothing else, historical examples show that huge amounts of money lobbed at a cause can go to waste or do more harm than good (e.g. the Iraq war as a means to improve relations with the middle East).
Eliezer and I were both speaking in vague terms; presumably somebody intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute can levy money toward some positive expected change in a given direction. There remains the question about the conversion factor between money and other goods and how quickly it changes at the margin; it could be negligible in a given instance.
The main limiting factor that I had in mind was human capital: an absence of people who are sufficiently intelligent, knowledgeable, sophisticated, motivated, energetic & socially/politically astute.
I would add that a group of such people would have substantially better than average odds of attracting sufficient funding from some philanthropist; further diminishing the value of donations (on account of fungibility).
That’s the probability statement in his post. He didn’t mention the probability of SIAI’s success, and hasn’t previously when I’ve emailed him or asked in public forums, nor has he at any point in time that I’ve heard. Shortly after I asked, he posted When (Not) To Use Probabilities.
Yes, I had read that, and perhaps even more apropos (from Shut up and do the impossible!):
But it’s not clear whether Eliezer means that he can’t even translate his intuitive feeling into a word like “small” or “medium”. I thought the comment I was replying to was saying that SIAI had a “medium” chance of success, given:
and
But perhaps I misinterpreted? In any case, there’s still the question of what is rational for those of us who do think SIAI’s chance of success is “small”.
I thought he was taking the “don’t bother” approach by not giving a probability estimate or arguing about probabilities.
I propose that the rational act is to investigate approaches to greater than human intelligence which would succeed.
This. I’m flabbergasted this isn’t pursued further.
Sufficiently-Friendly AI can be hard for SIAI-now but easy or medium for non-SIAI-now (someone else now, someone else future, SIAI future). I personally believe this, since SIAI-now is fucked up (and SIAI-future very well will be too). (I won’t substantiate that claim here.) Eliezer didn’t talk about SIAI specifically. (He probably thinks SIAI will be at least as likely to succeed as anyone else because he thinks he’s super awesome, but it can’t be assumed he’d assert that with confidence, I think.)
Will you substantiate it elsewhere?
Second that interest in hearing it substantiated elsewhere.
Your comments are a cruel reminder that I’m in a world where some of the very best people I know are taken from me.
SingInst seems a lot better since I wrote that comment; you and Luke are doing some cool stuff. Around August everything was in a state of disarray and it was unclear if you’d manage to pull through.
Regarding the graphs, I assumed that they were showing artificial examples so that we could viscerally understand at a glance what the adjustment does, not that this is what the prior and evidence should look like in a real case.
Upvoted.
(I deleted my response to this following othonormal’s comments; see this one for my revised thought here.)
Why do you think that there’s something that can be done about it?
What your example looks like is a nearly uniform prior over a very large space- nothing’s wrong when we quickly update to believe that yesterday’s lottery numbers are 04-15-21-31-36.
But the point where you need to halt, melt, and catch fire is if your prior assigns the vast majority of the probability mass to a small compact region, and then the evidence comes along and lands outside that region. That’s the equivalent of starting out 99.99% confident that you know tomorrow’s lottery numbers will begin with 01-02-03, and being proven wrong.
Yes, you’re right, I wasn’t thinking clearly, thanks for catching me. I think there’s something to what I was trying to say, but I need to think about it through more carefully. I find the explanation that you give in your other comment convincing (that the point of the graphs is to clearly illustrate the principle).
10 million dollars buys quite a few programmers. SIAI is presently nowhere near that amount of money, and doesn’t look likely to be any time soon. When does it make sense to start talking to volunteer programmers? Presumably, when the risk of opening up the project is less than the risk of failing to get it done before someone else does it wrong.
When is “soon” for these purposes. It seems to me that with continuing support, SIAI will be able to hire as many of the right programmers as we can find and effectively integrate into a research effort. We certainly would hire any such programmers now.
My main source of uncertainty as to SIAI’s value comes from the fact that as far as I can tell nobody has a viable Friendly AI research program.
(Please don’t upvote this comment till you’ve read it fully; I’m interpreting upvotes in a specific way.) Question for anyone on LW: If I had a viable preliminary Friendly AI research program, aimed largely at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty of Friendly AI for various values of “Friendly”, and wrote clearly and concretely about the necessary steps in pursuing this analysis, and listed and described a small number of people (less than 5, but how many could actually be convinced to focus on doing the analysis would depend on funds) who I know of who could usefully work on such an analysis, and committed to have certain summaries published online at various points (after actually considering concrete possibilities for failure, planning fallacy, etc., like real rationalists should), and associated with a few (roughly 5) high status people (people like Anders Sandberg or Max Tegmark, e.g. by convincing them to be on an advisory board), would this have a decent chance of causing you or someone you know to donate $100 or more to support this research program? (I have a weird rather mixed reputation among the greater LW community, so if that affects you negatively please pretend that someone with a more solid reputation but without super high karma is asking this question, like Steven Kaas.) You can upvote for “yes” and comment about any details, e.g. if you know someone who would possibly donate significantly more than $100. (Please don’t downvote for “no”, ’cuz that’s the default answer and will drown out any “yes” ones.)
Unless you would be much less involved in this potential program than the comment indicates, this seems like an inappropriate request. If people view you negatively due to your posting history, they should absolutely take that information into account in assessing how likely they would be to provide financial support to such a program (assuming that the negative view is based on relevant considerations such as your apparent communication or reasoning skills as demonstrated in your comments).
I was more interested in Less Wrong’s interest in new FAI-focused organizations generally than in anything particularly tied to me.
Fair enough, but in light of your phrasing in both the original comment (“If I [did the following things]”) and your comment immediately following it (quoted below; emphasis added), it certainly appeared to me that you seemed to be describing a significant role for yourself, even though your proposal was general overall.
Sorry, could you say again what exactly you want to do? I mean, what’s the output here that the money is paying for; a Friendly AI, a theory that can be used to construct a Friendly AI, or an analysis that purports to say whether or not Friendly AI is “feasible”, or what?
Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things.
I was trying to quickly gauge vague interest in a vague notion. I think that my original comment was at roughly the most accurate and honest level of vagueness (i.e. “aimed largely [i.e. primarily] at doing the technical analysis necessary to determine as well as possible the feasibility and difficulty [e.g. how many Von Neumanns, Turings, and/or Aristotles would it take?] of Friendly AI for various (logical) probabilities of Friendliness [e.g. is the algorithm meta-reflective enough to fall into (one of) some imagined Friendliness attractor basin(s)?]”). Value of information regarding difficulty of Friendly-ish AI is high, but research into that question is naturally tied to Friendly AI theory itself. I’m thinking… Goedel machine stability more than ambient decision theory, history of computation more than any kind of validity semantics. To some extent it depends on who plans to actually work on what stuff from the open problems lists. There are many interesting technical threads that people might start pulling on soon, and it’s unclear to me to what extent they actually will pull on them or to what extent pulling on them will give us a better sense of the problem.
[Stuff it would take too many paragraphs to explain why it’s worth pointing out specifically:] Theory of justification seems to be roughly as developed as theory of computation was before the advent of Leibniz; Leibniz saw a plethora of connections between philosophy, symbolic logic, and engineering and thus developed some correctly thematically centered proto-theory. I’m trying to make a Leibniz, and hopefully SingInst can make a Turing. (Two other roughly analogous historical conceptual advances are natural selection and temperature.)
Well, my probability that you could or would do anything useful, given money, just dropped straight off a cliff. But perhaps you’re just having trouble communicating. That is to say: What the hell are you talking about.
If you’re going to ask for money on LW, plain English response, please: What’s the output here that the money is paying for; (1) a Friendly AI, (2) a theory that can be used to construct a Friendly AI, or (3) an analysis that purports to say whether or not Friendly AI is “feasible”? Please pick one of the pre-written options; I now doubt your ability to write your response ab initio.
That was amusingly written, but probably too harsh. You want people to like you, even if it’s only so they say nice things about you.
Dude, it’s right there: “feasibility and difficulty”, in this sentence which I am now repeating for the second time:
(Bold added for emphasis, annotations in [brackets] were in the original.)
The next sentence:
Or if you really need it spelled out for you again and again, the output would primarily be (3) but secondarily (2) as you need some of (2) to do (3).
Because you clearly need things pointed out multiple times, I’ll remind you that I put my response in the original comment that you originally responded to, without the later clarifications that I’d put in for apparently no one’s benefit:
(Those italics were in the original comment!)
I wasn’t asking for money on Less Wrong! As I said, “I was trying to quickly gauge vague interest in a vague notion.” What the hell are you talking about.
I’ve doubted your ability to read for a long time, but this is pretty bad. The sad thing is you’re probably not doing this intentionally.
I think the problem here is that your posting style, to be frank, often obscures your point.
In most cases, posts that consist of a to-the-point answer followed by longer explanations use the initial statement to make a concise case. For instance, in this post, my first sentence sums up what I think about the situation and the rest explains that thought in more detail so as to convey a more nuanced impression.
By contrast, when Eliezer asked “What’s the output here that the money is paying for,” your first sentence was “Money would pay for marginal output, e.g. in the form of increased collaboration, I think, since the best Friendliness-cognizant x-rationalists would likely already be working on similar things.” This does not really answer his question, and while you clarify this with your later points, the overall message is garbled.
The fact that your true answer is buried in the middle of a paragraph does not really help things much. Though I can see what you are trying to say, I can’t in good and honest conscience describe it as clear. Had you answered, on the other hand, “Money would pay for the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI...” as your first sentence, I think your post would have been more clear and more likely to be understood.
The sentences I put before the direct answer to Eliezer’s question were meant to correct some of Eliezer’s misapprehensions that were more fundamental than the object of his question. Eliezer’s infamous for uncharitably misinterpreting people and it was clear he’d misinterpreted some key aspects of my original comment, e.g. my purpose in writing it. If I’d immediately directly answered his question that would have been dishonest; it would have contributed further to his having a false view of what I was actually talking about. Less altruistically it would be like I was admitting to his future selves or to external observers that I agreed that his model of my purposes was accurate and that this model could legitimately be used to assert that I was unjustified in any of many possible ways. Thus I briefly (a mere two sentences) attempted to address what seemed likely to be Eliezer’s underlying confusions before addressing his object level question. (Interestingly Eliezer does this quite often, but unfortunately he often assumes people are confused in ways that they are not.)
Given these constraints, what should I have done? In retrospect I should have gone meta, of course, like always. What else?
Thanks much for the critique.
Given those constraints, I would probably write something like “Money would pay for marginal output in the form of increased collaboration on the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI” for my first sentence and elaborate as strictly necessary. That seems rather more cumbersome than I’d like, but it’s also a lot of information to try and convey in one sentence!
Alternatively, I would consider something along the lines of “Money would pay for the technical analysis necessary to determine as well as possible the feasibility and difficulty of FAI, but not directly—since the best Friendliness-cognizant x-rationalists would likely already be working on similar things, the money would go towards setting up better communication, coordination, and collaboration for that group.”
That said, I am unaware of any reputation Eliezer has in the field of interpreting people, and personally haven’t received the impression that he’s consistently unusually bad or uncharitable at it. Then again, I have something of a reputation—at least in person—for being too charitable, so perhaps I’m being too light on Eliezer (or you?) here.
I acknowledge this. But it seems to me that the larger problem is that Eliezer simply doesn’t know how to read what people actually say. Less Wrong mostly doesn’t either, and humans in general certainly don’t. This is a very serious problem with LW-style rationality (and with humanity). There are extremely talented rationalists who do not have this problem; it is an artifact of Eliezer’s psychology and not of the art of rationality.
It’s hardly fair to blame the reader when you’ve got “sentences” like this:
That was the second version of the sentence, the first one had much clearer syntax and even italicized the answer to Eliezer’s subsequent question. It looks the way it does because Eliezer apparently couldn’t extract meaning out of my original sentence despite it clearly answering his question, so I tried to expand on the relevant points with bracketed concrete examples. Here’s the original:
(emphasis in original)
Which starts with the word ‘if’ and fails to have a ‘then’.
If you took out ‘If I had’ and replaced it with ‘I would create’, then maybe it would be more in line with what you’re trying to say?
What you say might be true, but this one example is negligible compared to the mountain of other evidence concerning inability to read much more important things (which are unrelated to me). I won’t give that evidence here.
Certainly true, but that only means that we need to spend more effort on being as clear as possible.
If that’s indeed the case (I haven’t noticed this flaw myself), I suggest that you write articles (or perhaps commission/petition others to have them written) describing this flaw and how to correct it. Eliminating such a flaw or providing means of averting it would greatly aid LW and the community in general.
Unfortunately that is not currently possible for many reasons, including some large ones I can’t talk about and that I can’t talk about why I can’t talk about. I can’t see any way that it would become possible in the next few years either. I find this stressful; it’s why I make token attempts to communicate in extremely abstract or indirect ways with Less Wrong, despite the apparent fruitlessness. But there’s really nothing for it.
Unrelated public announcement: People who go back and downvote every comment someone’s made, please, stop doing that. It’s a clever way to pull information cascades in your direction but it is clearly an abuse of the content filtering system and highly dishonorable. If you truly must use such tactics, downvoting a few of your enemy’s top level posts is much less evil; your enemy loses the karma and takes the hint without your severely biasing the public perception of your enemy’s standard discourse. Please.
(I just lost 150 karma points in a few minutes and that’ll probably continue for awhile. This happens a lot.)
Why can’t you talk about why you can’t talk about them?
I’m not a big fan of the appeal to secret reasons, so I think I’m going to have pull out of this discussion. I will note, however, that you personally seem to be involved in more misunderstandings than the average LW poster, so while it’s certainly possible that your secret reasons are true and valid and Eliezer just sucks at reading or whatever, you may want to clarify certain elements of your own communication as well.
I unfortunately predict that “going more meta” will not be strongly received here.
I’m sorry to hear that you’re up against something so difficult, and I hope you find a way out.
Thank you… I think I just need to be more meta. Meta never fails.
Are we still talking about improving general reading comprehension? What could possibly be dangerous about that?
To save some time and clarify, this was option 3: an analysis that purports to say whether or not Friendly AI is “feasible”.
(Some people, including me, would really like it if a competent and FAI-focused uber-rationalist non-profit existed. I know people who will soon have enough momentum to make this happen. I am significantly more familiar with the specifics of FAI (and of hardcore SingInst-style rationality) than many of those people and almost anyone else in the world, so it’d be necessary that I put a lot of hours into working with those who are higher status than me and better at getting things done but less familiar with technical Friendliness. But I have many other things I could be doing. Hence the question.)
Does “FAI-focused” mean what I called code first? What are your thoughts on that post and its followup? What is this new non-profit planning to do differently from SIAI and why? What are the other things that you could be doing?
Incomplete response:
Jah. Well, at least determining whether or not “code first” is even reasonable, yeah, which is a difficult question in itself and only partially tied in with making direct progress on FAI.
You seem to have missed Oracle AI? (Eliezer’s dismissal of it isn’t particularly meaningful.) I agree with your concerns. This is why the main focus would at least initially be determining whether or not “code first” is a plausible approach (difficulty-wise and safety-wise). The value of information on that question is incredibly high and as you’ve pointed out it has not been sufficiently researched.
Basically everything. SingInst is focused on funding a large research program and gaining the prestige necessary to influence (academic) culture and academic and political policy. They’re not currently doing any research on Friendly AI, and their political situation is such that I don’t expect them to be able to do so effectively for a while, if ever. I will not clarify this. (Actually their research associates are working on FAI-related things, but SingInst doesn’t pay them to do that.)
Learning, mostly. Working with an unnamed group of x-risk-cognizant people that LW hasn’t heard of, in a way unrelated to their setting up a non-profit.
My understanding is that SIAI recently tried to set up a new in-house research team to do preliminary research into FAI (i.e., not try to build an FAI yet, but just do whatever research that might be eventually helpful to that project). This effort didn’t get off the ground, but my understanding again is that it was because the researchers they tried to recruit had various reasons for not joining SIAI at this time. I was one of those they tried to recruit, and while I don’t know what the others’ reasons were, mine were mostly personal and not related to politics.
You must also know all this, since you were involved in this effort. So I’m confused why you say SIAI won’t be doing effective research on FAI due to its “political situation”. Did the others not join SIAI because they thought SIAI was in a bad political situation? (This seems unlikely to me.) Or are you referring to the overall lack of qualified, recruitable researchers as a “political situation”? If you are, why do you think this new organization would be able to do better?
(Or did you perhaps not learn the full story, and thought SIAI stopped this effort for political reasons?)
The answer to your question isn’t among your list of possible answers. The recent effort to start an in-house research team was a good attempt and didn’t fail for political reasons. I am speaking of other things. However I want to take a few weeks off from discussion of such topics; I seem to have given off the entirely wrong impression and would prefer to start such discussion anew in a better context, e.g. one that better emphasizes cooperation and tentativeness rather than reactionary competition. My apologies.
If you can’t say anything, don’t say anything.
It’s a good heuristic, but can be very psychologically difficult. E.g. if you think that not even trying to communicate will be seen as unjustified in retrospect even if people should know that there was no obvious way for you to communicate. This has happened enough to me that the thought of just giving up on communication is highly aversive; my fear of being blamed for not preventing others to take unjustified actions (that will cause me, them, and the universe counterfactually-needless pain) is too great. But whatever, I’m starting to get over it.
Like, I remember a pigeon dying… people dying… a girl who starved herself… a girl who cut herself… a girl who wanted to commit suicide… just, trust me, there are reasons that I’m afraid. I could talk about those reasons but I’d rather not. It’s just, if you don’t even make a token gesture it’s like you don’t even care at all, and it’s easier to be unjustified in a way that can be made to look sorta like caring than in a way that looks like thoughtlessness or carelessness.
(ETA: A lot of the time when people give me or others advice I mentally translate it to “The solution is simple, just shut up and be evil.”.)
I have no particular attachments to SIAI and would love to see a more effective Singularitarian organization formed if that were possible. I’m just having genuine trouble understanding why you think this new proposed organization will be able to do more effective FAI research. Perhaps you could use these few weeks off to ask some trusted advisors how to better communicate this point. (I understand you have sensitive information that you can’t reveal, but I’m guessing that you can do better even within that constraint.)
This is exactly the strategy I’ve decided to enact, e.g. talking to Anna. Thanks for being… gentle, I suppose? I’ve been getting a lot of flak lately, it’s nice to get some non-insulting advice sometimes. :)
(Somehow I completely failed to communicate the tentativeness of the ideas I was throwing around; in my head I was giving it about a 3% chance that I’d actually work on helping build an organization but I seem to have given off an impression of about 30%. I think this caused everyone’s brains to enter politics mode, which is not a good mode for brains to be in.)
It’s rather strange how the SIAI is secretive. The military projects are secretive, the commercial projects are secretive alas: so few value transparency. An open project would surely do better, through being more obviously trustworthy and accountable, being better able to use talent across the internet, etc. I figure if the SIAI persists in not getting to grips with this issue, some other organisation will.
Could you tell us about them?
Well, looking at the post on SIAI finances from a few months back, SIAI’s annual revenue is growing at a rate of roughly 100k/year, and thus would take nearly a century to reach 10 million / year. Of course, all sorts of things could change these numbers. Eliezer has stated that he believes that UFAI will happen somewhat sooner than a century.
Since SIAI does seem to have at least some unused budget for programmers now, I emailed a friend who might be a good fit for a research associate to suggest that he apply.
Even if past values are linear, exponential estimates are probably more clarifying.
(ETA: This isn’t the right place for what I wrote.)
ADBOC. SIAI should never be content with any funding level. It should get the resources to hire, bribe or divert people who may otherwise make breakthroughs on UAI, and set them to doing anything else.
That intuition seems to me to follow from the probably false assumption that if behavior X would, under some circumstances, be utility maximizing, it is also likely to be utility maximizing to fund a non-profit to engage in behavior X. SIAI isn’t a “do what seems to us to maximize expected utility” organization because such vague goals don’t make for a good organizational culture. Organizing and funding research into FAI and research inputs into FAI, plus doing normal non-profit fund-raising and outreach, that’s a feasible non-profit directive.
It also follows from the assumption that the claims in any comment submitted on August 20, 2011 are true. Yet I do not believe this.
I had, to the best of my ability, considered the specific situation when giving my advice.
Any advice can be dismissed by suggesting it came from a too generalized assumption.
If you thought someone was about to foom an unfriendly AI, you would do something about it, and without waiting to properly update your 501(c) forms.
I confirm this.