Holden seems to assume that GMAGI has access to predictive algorithms for all possible questions—this seems to me to be unlikely (say, 1% chance), compared to the possibility that it has the ability to write novel code for new problems. If it writes novel code and runs it, it must have some algorithm for how that code is written and what resources are used to implement it—limiting that seems like the domain of SIAI research.
Holden explicitly states:
All I know is that you think prediction_function() is where the risk is. I don’t understand this position, because prediction_function() need not be a self-improving function or a “maximizing” function; it can just be an implementation of the human version with fewer distractions and better hardware.
i.e., that he believes that all novel questions will have algorithms already implemented, which seems to me to be clearly his weakest assumption, if he is assuming that GMAGI is non-narrow.
I would bet that development teams would tap into budding AGI in whatever ways they could that were clearly “safe,” such as asking it for ways to improve itself and considering them
I thought the whole danger of a black box scenario is that humans may be unable to successfully screen unsafe improvements?
These seem like serious weaknesses in his PoV to me.
However his points that
a) a narrow philosophy-AI could outdo SIAI in the arena of FAI (in which case identifying the problem of FAI is 99% of SIAI’s value and is already accomplished)
b) FAI research may not be used for whatever reason by whatever teams DO develop AGI
c) Something Weird Happens
Seem like very strong points diminishing the value of SIAI.
I can sympathize with his position that as an advocate of efficient charity he should be focused on promoting actions of charitable actors which he has a high level of certainty will be significantly more efficient, and that maintaining the mindset in himself that highly reliable charities are preferable to highly valuable charities helps him fulfill the social role he is in. That is, he should be very averse to a scenario in which he recommends a charity which turns out to be less efficient than charities he is recommending it over. The value of SIAI does not seem to me to be totally overdetermined.
In conclusion, I have some updating to do, but I don’t know in which direction it is. And I absolutely love reading serious well thought out conversations by intelligent people about important subjects.
Holden seems to assume that GMAGI has access to predictive algorithms for all possible questions—this seems to me to be unlikely (say, 1% chance), compared to the possibility that it has the ability to write novel code for new problems. If it writes novel code and runs it, it must have some algorithm for how that code is written and what resources are used to implement it—limiting that seems like the domain of SIAI research.
Holden explicitly states:
i.e., that he believes that all novel questions will have algorithms already implemented, which seems to me to be clearly his weakest assumption, if he is assuming that GMAGI is non-narrow.
I thought the whole danger of a black box scenario is that humans may be unable to successfully screen unsafe improvements?
These seem like serious weaknesses in his PoV to me.
However his points that
a) a narrow philosophy-AI could outdo SIAI in the arena of FAI (in which case identifying the problem of FAI is 99% of SIAI’s value and is already accomplished) b) FAI research may not be used for whatever reason by whatever teams DO develop AGI c) Something Weird Happens
Seem like very strong points diminishing the value of SIAI.
I can sympathize with his position that as an advocate of efficient charity he should be focused on promoting actions of charitable actors which he has a high level of certainty will be significantly more efficient, and that maintaining the mindset in himself that highly reliable charities are preferable to highly valuable charities helps him fulfill the social role he is in. That is, he should be very averse to a scenario in which he recommends a charity which turns out to be less efficient than charities he is recommending it over. The value of SIAI does not seem to me to be totally overdetermined.
In conclusion, I have some updating to do, but I don’t know in which direction it is. And I absolutely love reading serious well thought out conversations by intelligent people about important subjects.