“We think MIRI is literally useless” is a decent reason not to fund MIRI at all, and is broadly consistent with Holden’s early thoughts on the matter. But it’s a weird reason to give MIRI $500K but OpenAI $30M. It’s possible that no one has the capacity to do direct work on the long-run AI alignment problem right now. In that case, backwards-chaining to how to build the capacity seems really important.
While I disagree with Holden that MIRI is near-useless, I think his stated reasons for giving MIRI $500k are pretty good reasons that I’d do myself if I had that money and thought MIRI was near-useless.
(Namely, that so far MIRI has had a lot of good impacts regardless of the quality of their research, in terms of general community building, and that this should be rewarded so that other orgs are incentivized to do things like that)
True, but 2012 might be long enough ago that many of the concerns he had then may now be irrelevant. In addition, based on my understanding of MIRI’s current approach and their arguments for that approach, I feel that many of his concerns either represent fundamental misunderstandings or are based on viewpoints that have significantly changed within MIRI since that time. For example, I have a hard time wrapping my head around this objection:
Objection 1: it seems to me that any AGI that was set to maximize a “Friendly” utility function would be extraordinarily dangerous.
This seems to be precisely the same concern expressed by MIRI and one of the fundamental arguments that their Agent Foundations approach is based on, in particular, what they deem the Value Specification problem. And I believe Yudkowsky has used this as a primary argument for AI safety in general for quite a while, very likely before 2012.
There is also the “tool/agent” distinction cited as objection 2 that I think is well addressed in MIRI’s publications as well as Bostrom’s Superintelligence, where it’s made pretty clear that the distinction is not quite that clear cut (and gets even blurrier the more intelligent the “tool AI” gets).
Given that MIRI has had quite some time to refine their views as well as their arguments, as well as having gone through a restructuring and hiring quite a few new researchers since that time, what is the likelihood that Holden holds the same objections that were stated in the 2012 review?
Holden spent a lot of effort stating reasons in http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/
“We think MIRI is literally useless” is a decent reason not to fund MIRI at all, and is broadly consistent with Holden’s early thoughts on the matter. But it’s a weird reason to give MIRI $500K but OpenAI $30M. It’s possible that no one has the capacity to do direct work on the long-run AI alignment problem right now. In that case, backwards-chaining to how to build the capacity seems really important.
While I disagree with Holden that MIRI is near-useless, I think his stated reasons for giving MIRI $500k are pretty good reasons that I’d do myself if I had that money and thought MIRI was near-useless.
(Namely, that so far MIRI has had a lot of good impacts regardless of the quality of their research, in terms of general community building, and that this should be rewarded so that other orgs are incentivized to do things like that)
True, but 2012 might be long enough ago that many of the concerns he had then may now be irrelevant. In addition, based on my understanding of MIRI’s current approach and their arguments for that approach, I feel that many of his concerns either represent fundamental misunderstandings or are based on viewpoints that have significantly changed within MIRI since that time. For example, I have a hard time wrapping my head around this objection:
This seems to be precisely the same concern expressed by MIRI and one of the fundamental arguments that their Agent Foundations approach is based on, in particular, what they deem the Value Specification problem. And I believe Yudkowsky has used this as a primary argument for AI safety in general for quite a while, very likely before 2012.
There is also the “tool/agent” distinction cited as objection 2 that I think is well addressed in MIRI’s publications as well as Bostrom’s Superintelligence, where it’s made pretty clear that the distinction is not quite that clear cut (and gets even blurrier the more intelligent the “tool AI” gets).
Given that MIRI has had quite some time to refine their views as well as their arguments, as well as having gone through a restructuring and hiring quite a few new researchers since that time, what is the likelihood that Holden holds the same objections that were stated in the 2012 review?