Do you know the person that wrote that post? Or anyone else supposedly involved in the events it describes? I’m not sure I could adjudicate the claims in that post, for my own judgement, given my remove from everyone supposedly involved.
I’m also still unsure how any of that, assuming it’s true, should be weighed against the ‘official’ work MIRI has done or is doing. Surely AI safety has to be balanced against those (unrelated) claims somehow, as terrible as they are (or might be), and as terrible as it is to think about ‘balancing’ these ‘costs’ and (potential) ‘benefits’.
Some of the claims in that post also aren’t obviously terrible to me, e.g. MIRI reaching a legal settlement with someone that ‘blackmailed’ them.
And if the person “Ziz” mentioned in the post is the same person I’m thinking of, I’m really confused as to what to think about the other claims, given the conflicting info about them I’ve read.
The post quotes something (about some kind of recollection of a conversation) about “a drama thing” and all of this seems very much like “a drama thing” (or several such ‘drama things’) and it’s really hard to think of any way for me, or anyone not involved, or even anyone that is or was involved, to determine with any confidence what’s actually true about whatever it is that (may have) happened.
I know a few people involved, and I trust that they’re not lying, especially given that some of my own experiences overlap. I lived in the Bay for a couple years, and saw how people acted, so I’m fairly confident that the main claims in the open letter are true.
I’ve written myself a bit about why the payout was so bad here, which the author of the open letter appears to reference.
MIRI wrote this paper: https://arxiv.org/abs/1710.05060 The paper is pretty clear that it’s bad decision theory to pay out to extortion. I agree with the paper’s reasoning, and independently came to a similar conclusion, myself. MIRI paying out means MIRI isn’t willing to put their money where their mouth is. Your ability to actually follow through on what you believe is necessary when doing high-stakes work.
Like, a lot of MIRI’s research is built around this claim about decision theory. It’s fundamental to MIRI’s approach. If one buys that FDT is correct, then MIRI’s failure to consistently implement it here undermines one’s trust in them as an institution. They folded like wet cardboard. If one doesn’t buy FDT, or if one generally thinks paying out to extortionists isn’t a big deal, then it wouldn’t appear to be a big deal that they did. But a big part of the draw towards rationalist spaces and MIRI is that they claim to take ideas seriously. This behaviour indicates (to me) that they don’t, not where it counts.
As for Ziz, from what I understand she’s been the victim of a rather vicious defamation campaign chiefly organized by a determined stalker who is angry with her for not sleeping with him. If you reach out to some rationalist discord mods, you should be able to get a hold of sufficient evidence to back the claims in that post.
I’m still not sure what to think as an outsider, but I appreciate the details you shared.
With respect to the “extortion” specifically, I’d (charitably) expect that MIRI is somewhat constrained by their funders and advisors with respect to settling a (potential) lawsuit, i.e. making a “pay out to extortion”.
I still think all of this, even if it’s true (to any significant extent), isn’t an overwhelming reason not to support MIRI (at all), given that they do seem to be doing good technical work.
Is there another organization that you think is doing similarly good work without being involved in the same kind of alleged bad behavior?
From what I understand, some of their funders were convinced MIRI would never pay out, and were quite upset to learn they did. For example, one of the people quoted in that open letter was Paul Crowley, a long time supporter who has donated almost $50k. Several donors were so upset they staged a protest.
I still think all of this, even if it’s true (to any significant extent), isn’t an overwhelming reason not to support MIRI (at all), given that they do seem to be doing good technical work.
MIRI should’ve been an attempt to keep AGI out of the hands of the state
Eliezer several times expressed the view that it’s a mistake to focus too much on whether “good” or “bad” people are in charge of AGI development. Good people with a mistaken methodology can still produce a “bad” AI, and a sufficiently robust methodology (e.g. by aligning with an idealized abstract human rather than a concrete individual) would still produce a “good” AI from otherwise unpromising circumstances.
Can you refer me to a textbook or paper written by the AGI crowd which establishes how we get from GPT-n to an actual AGI? I am very skeptical of AI safety but want to give it a second hearing.
My impression is that there is not a convincing roadmap. I certainly haven’t seen one. However, I recognize that there is a healthy possibility that there is one, and I just haven’t seen it.
Which is why I’m asking for the white paper / textbook chapter that presumably has convinced everyone that we can expect AGI in the coming decades. I would be very grateful for anyone who could provide it.
Obviously, AGI is feasible (more than could be said for things like nanotech or quantum computing). However, it’s feasible in the sense that a rocket ship was feasible in the time of the ancient Greeks. Obviously a lot of knowledge was necessary to get from Greeks to Armstrong.
Right now all we have are DL models that are basically kernel machines which convert noise into text, right? My intuition is that there is no path from that to AGI, and that AGI would need to come from some sort of dynamic system, and that we’re nowhere near creating such. I would like to be proven wrong though!
If you know how to make an AGI, you are only a little bit of coding before making it. We have limited AI’s that can do some things, and aren’t clear what we are missing. Experts are inventing all sorts of algorithms.
There are various approaches like mind uploading, evolutionary algorithms etc that fairly clearly would work if we threw enough effort at them. Current reinforcement learning approaches seem like they might get smart, with enough compute and the right environment.
Unless you personally end up helping make the first AGI, then you personally will probably not be able to see how to do it until after it is done (if at all). The fact that you personally can’t think of any path to AGI does not tell us where we are on the tech path. Someone else might be putting the finishing touches on their AI right now. Once you know how to do it, you’ve done it.
FWIW, I think that mind uploading is much less likely to work than a purely synthetic AI, at least in reasonably near-term scenarios. I have never read any description of how mind uploading is going to work which doesn’t begin by assuming that the hard part (capturing all of the necessary state from an existing mind) is already done.
Of course mind uploading would work hypothetically. The question is, how much of the mind must be uploaded? A directed graph and an update rule? Or an atomic-level simulation of the entire human body? The same principle applies to evolutionary algorithms, reinforcement learning (not the DL sort imo tho, it’s a dead end), etc. I actually don’t think it would be impossible to at least get a decent lower bound on the complexity needed by each of these approaches. Do the AI safety people do anything like this? That would be a paper I’d like to read.
I don’t know whether to respond to the “Once you know how to do it, you’ve done it” bit. Should I claim that this is not the case in other fields? Or will AI be “different”? What is the standard under which this statement could be falsified?
For the goal of getting humans to mars, we can do the calculations and see that we need quite a bit of rocket fuel. You could reasonably be in a situation where you had all the design work done, but you still needed to get atoms into the right places, and that took a while. Big infrastructure projects can be easier to design. For a giant damm, most of the effort is in actually getting all the raw materials in place. This means you can know what it takes to build a damm, and be confident it will take at least 5 years given the current rate of concrete production.
Mathematics is near the other end of the scale. If you know how to prove theorem X, you’ve proved it. This stops us being confident that a theorem won’t be proved soon. Its more like a radioactive decay of an fairly long lived atom more likely to be next week than any other week.
I think AI is fairly close to the maths, most of the effort is figuring out what to do.
Ways my statement could be false.
If we knew the algorithm, and the compute needed, but couldn’t get that compute.
If AI development was an accumulation of many little tricks, and we knew how many tricks were needed.
But at the moment, I think we can rule out confident long termism on AI. We have no way of knowing that we aren’t just one clever idea away from AGI.
We can get a rough idea of this by considering how much physical changes have a mental effect. Psychoactive chemicals, brain damage etc. Look at how much ethanol changes the behaviour of a single neuron in a lab dish. How much it changes human behaviour. And that gives a rough indication of how sensitively dependant human behaviour is on the exact behaviour of its constituent neurons.
I don’t believe supporting MIRI is a good use of money.
Why do you think support MIRI isn’t a good use of money?
My comment is based on this post, which seems to cover the matter thoroughly.
Thanks!
Do you know the person that wrote that post? Or anyone else supposedly involved in the events it describes? I’m not sure I could adjudicate the claims in that post, for my own judgement, given my remove from everyone supposedly involved.
I’m also still unsure how any of that, assuming it’s true, should be weighed against the ‘official’ work MIRI has done or is doing. Surely AI safety has to be balanced against those (unrelated) claims somehow, as terrible as they are (or might be), and as terrible as it is to think about ‘balancing’ these ‘costs’ and (potential) ‘benefits’.
Some of the claims in that post also aren’t obviously terrible to me, e.g. MIRI reaching a legal settlement with someone that ‘blackmailed’ them.
And if the person “Ziz” mentioned in the post is the same person I’m thinking of, I’m really confused as to what to think about the other claims, given the conflicting info about them I’ve read.
The post quotes something (about some kind of recollection of a conversation) about “a drama thing” and all of this seems very much like “a drama thing” (or several such ‘drama things’) and it’s really hard to think of any way for me, or anyone not involved, or even anyone that is or was involved, to determine with any confidence what’s actually true about whatever it is that (may have) happened.
I know a few people involved, and I trust that they’re not lying, especially given that some of my own experiences overlap. I lived in the Bay for a couple years, and saw how people acted, so I’m fairly confident that the main claims in the open letter are true.
I’ve written myself a bit about why the payout was so bad here, which the author of the open letter appears to reference.
MIRI wrote this paper: https://arxiv.org/abs/1710.05060 The paper is pretty clear that it’s bad decision theory to pay out to extortion. I agree with the paper’s reasoning, and independently came to a similar conclusion, myself. MIRI paying out means MIRI isn’t willing to put their money where their mouth is. Your ability to actually follow through on what you believe is necessary when doing high-stakes work.
Like, a lot of MIRI’s research is built around this claim about decision theory. It’s fundamental to MIRI’s approach. If one buys that FDT is correct, then MIRI’s failure to consistently implement it here undermines one’s trust in them as an institution. They folded like wet cardboard. If one doesn’t buy FDT, or if one generally thinks paying out to extortionists isn’t a big deal, then it wouldn’t appear to be a big deal that they did. But a big part of the draw towards rationalist spaces and MIRI is that they claim to take ideas seriously. This behaviour indicates (to me) that they don’t, not where it counts.
As for Ziz, from what I understand she’s been the victim of a rather vicious defamation campaign chiefly organized by a determined stalker who is angry with her for not sleeping with him. If you reach out to some rationalist discord mods, you should be able to get a hold of sufficient evidence to back the claims in that post.
Thanks!
I’m still not sure what to think as an outsider, but I appreciate the details you shared.
With respect to the “extortion” specifically, I’d (charitably) expect that MIRI is somewhat constrained by their funders and advisors with respect to settling a (potential) lawsuit, i.e. making a “pay out to extortion”.
I still think all of this, even if it’s true (to any significant extent), isn’t an overwhelming reason not to support MIRI (at all), given that they do seem to be doing good technical work.
Is there another organization that you think is doing similarly good work without being involved in the same kind of alleged bad behavior?
From what I understand, some of their funders were convinced MIRI would never pay out, and were quite upset to learn they did. For example, one of the people quoted in that open letter was Paul Crowley, a long time supporter who has donated almost $50k. Several donors were so upset they staged a protest.
I disagree. I’ve written a bit about why here.
You write
Eliezer several times expressed the view that it’s a mistake to focus too much on whether “good” or “bad” people are in charge of AGI development. Good people with a mistaken methodology can still produce a “bad” AI, and a sufficiently robust methodology (e.g. by aligning with an idealized abstract human rather than a concrete individual) would still produce a “good” AI from otherwise unpromising circumstances.
Can you link to 3 times?
Unequivocal example from 2015: “You can’t take for granted that good people build good AIs and bad people build bad AIs.”
A position paper from 2004. See the whole section “Avoid creating a motive for modern-day humans to fight over the initial dynamic.”
Tweets from 2020.
That’s an artificially narrow example. You can have...
a good person with good methodology
a good person with bad methodology
a bad person with good methodology
a bad person with bad methodology
A question to ask is, when someone aligns an AGI with some approximation of “good values,” whose approximation are we using?
If what’s written in this post is true it’s a major problem for the long-term goals of MIRI and the bay-area rationalist community in general.
Can you refer me to a textbook or paper written by the AGI crowd which establishes how we get from GPT-n to an actual AGI? I am very skeptical of AI safety but want to give it a second hearing.
Why is this a response to the parent comment?
It sounds like you think that actual AGI won’t happen? And your argument is that we don’t have a convincing roadmap for how to get there?
Well, I didn’t mean to propose an argument.
My impression is that there is not a convincing roadmap. I certainly haven’t seen one. However, I recognize that there is a healthy possibility that there is one, and I just haven’t seen it.
Which is why I’m asking for the white paper / textbook chapter that presumably has convinced everyone that we can expect AGI in the coming decades. I would be very grateful for anyone who could provide it.
Obviously, AGI is feasible (more than could be said for things like nanotech or quantum computing). However, it’s feasible in the sense that a rocket ship was feasible in the time of the ancient Greeks. Obviously a lot of knowledge was necessary to get from Greeks to Armstrong.
Right now all we have are DL models that are basically kernel machines which convert noise into text, right? My intuition is that there is no path from that to AGI, and that AGI would need to come from some sort of dynamic system, and that we’re nowhere near creating such. I would like to be proven wrong though!
I think this post sums up the situation.
https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence
If you know how to make an AGI, you are only a little bit of coding before making it. We have limited AI’s that can do some things, and aren’t clear what we are missing. Experts are inventing all sorts of algorithms.
There are various approaches like mind uploading, evolutionary algorithms etc that fairly clearly would work if we threw enough effort at them. Current reinforcement learning approaches seem like they might get smart, with enough compute and the right environment.
Unless you personally end up helping make the first AGI, then you personally will probably not be able to see how to do it until after it is done (if at all). The fact that you personally can’t think of any path to AGI does not tell us where we are on the tech path. Someone else might be putting the finishing touches on their AI right now. Once you know how to do it, you’ve done it.
FWIW, I think that mind uploading is much less likely to work than a purely synthetic AI, at least in reasonably near-term scenarios. I have never read any description of how mind uploading is going to work which doesn’t begin by assuming that the hard part (capturing all of the necessary state from an existing mind) is already done.
I agree that purely synthetic AI will probably happen sooner.
Of course mind uploading would work hypothetically. The question is, how much of the mind must be uploaded? A directed graph and an update rule? Or an atomic-level simulation of the entire human body? The same principle applies to evolutionary algorithms, reinforcement learning (not the DL sort imo tho, it’s a dead end), etc. I actually don’t think it would be impossible to at least get a decent lower bound on the complexity needed by each of these approaches. Do the AI safety people do anything like this? That would be a paper I’d like to read.
I don’t know whether to respond to the “Once you know how to do it, you’ve done it” bit. Should I claim that this is not the case in other fields? Or will AI be “different”? What is the standard under which this statement could be falsified?
For the goal of getting humans to mars, we can do the calculations and see that we need quite a bit of rocket fuel. You could reasonably be in a situation where you had all the design work done, but you still needed to get atoms into the right places, and that took a while. Big infrastructure projects can be easier to design. For a giant damm, most of the effort is in actually getting all the raw materials in place. This means you can know what it takes to build a damm, and be confident it will take at least 5 years given the current rate of concrete production.
Mathematics is near the other end of the scale. If you know how to prove theorem X, you’ve proved it. This stops us being confident that a theorem won’t be proved soon. Its more like a radioactive decay of an fairly long lived atom more likely to be next week than any other week.
I think AI is fairly close to the maths, most of the effort is figuring out what to do.
Ways my statement could be false.
If we knew the algorithm, and the compute needed, but couldn’t get that compute.
If AI development was an accumulation of many little tricks, and we knew how many tricks were needed.
But at the moment, I think we can rule out confident long termism on AI. We have no way of knowing that we aren’t just one clever idea away from AGI.
The question is not just “how much is needed” but also “what’s a reasonable difference between the new digital mind and the biological sustance”.
We can get a rough idea of this by considering how much physical changes have a mental effect. Psychoactive chemicals, brain damage etc. Look at how much ethanol changes the behaviour of a single neuron in a lab dish. How much it changes human behaviour. And that gives a rough indication of how sensitively dependant human behaviour is on the exact behaviour of its constituent neurons.