Could you give me some examples of other people and organizations trying to prevent the risk of an Unfriendy AI? Because for me, it’s not like I believe that SI has a great chance to develop the theory and prevent the danger, but rather like they are the only people who even care about this specific risk (which I believe to be real).
As soon as the message becomes widely known, and smart people and organizations will start rationally discussing the dangers of Unfriendly AI, and how to make a Friendly AI (avoiding some obvious errors, such as “a smart AI simply must develop a human-compatible morality, because it would be too horrible to think otherwise”), then there is a pretty good chance that some of those organization will be more capable than SI to reach that goal: more smart people, better funding, etc. But at this moment, SI seems to be the only one paying attention to this topic.
It’s a crooked game, but it’s the only game in town?
None of that is evidence that SI would be more effective if it had more money. Assign odds to hostile AI becoming extant given low funding for SI, and compare the odds of hostile AI becoming extant given high funding for SI. The difference between those two is proportional to the value of SI (with regards to preventing hostile AI).
SI being the only one ought to lower your probability that this whole enterprise is worthwhile in any way.
With regards to the ‘message’, i think you grossly over estimate value of a rather easy insight that anyone who has watched Terminator could have. With regards to “rationally discussing”, what I have seen so far here is pure rationalization and very little, if any, rationality. What the SI has on the track record is, once again, a lot of rationalizations and not enough rationality to even have had an accountant through it’s first 10 years and first over 2 millions dollars in other people’s money.
Note that that second paragraph is one of Holden Karnofsky’s objections to SIAI: a high opinion of its own rationality that is not so far substantiable from the outside view.
Yes. I am sure Holden is being very polite, which is generally good but I’ve been getting impression that the point he was making did not in full carry across the same barrier that has resulted in the above-mentioned high opinion of own rationality despite complete lack of results for which rationality would be better explanation than irrationality (and presence of results which set rather low ceiling for the rationality). The ‘resistance to feedback’ is even stronger point, suggestive that the belief in own rationality is, at least to some extent, combined with expectation that it won’t pass the test and subsequent avoidance (rather than seeking) of tests; as when psychics do believe in their powers but do avoid any reliable test.
Could you give me some examples of other people and organizations trying to prevent the risk of an Unfriendy AI? Because for me, it’s not like I believe that SI has a great chance to develop the theory and prevent the danger, but rather like they are the only people who even care about this specific risk (which I believe to be real).
As soon as the message becomes widely known, and smart people and organizations will start rationally discussing the dangers of Unfriendly AI, and how to make a Friendly AI (avoiding some obvious errors, such as “a smart AI simply must develop a human-compatible morality, because it would be too horrible to think otherwise”), then there is a pretty good chance that some of those organization will be more capable than SI to reach that goal: more smart people, better funding, etc. But at this moment, SI seems to be the only one paying attention to this topic.
It’s a crooked game, but it’s the only game in town?
None of that is evidence that SI would be more effective if it had more money. Assign odds to hostile AI becoming extant given low funding for SI, and compare the odds of hostile AI becoming extant given high funding for SI. The difference between those two is proportional to the value of SI (with regards to preventing hostile AI).
SI being the only one ought to lower your probability that this whole enterprise is worthwhile in any way.
With regards to the ‘message’, i think you grossly over estimate value of a rather easy insight that anyone who has watched Terminator could have. With regards to “rationally discussing”, what I have seen so far here is pure rationalization and very little, if any, rationality. What the SI has on the track record is, once again, a lot of rationalizations and not enough rationality to even have had an accountant through it’s first 10 years and first over 2 millions dollars in other people’s money.
Note that that second paragraph is one of Holden Karnofsky’s objections to SIAI: a high opinion of its own rationality that is not so far substantiable from the outside view.
Yes. I am sure Holden is being very polite, which is generally good but I’ve been getting impression that the point he was making did not in full carry across the same barrier that has resulted in the above-mentioned high opinion of own rationality despite complete lack of results for which rationality would be better explanation than irrationality (and presence of results which set rather low ceiling for the rationality). The ‘resistance to feedback’ is even stronger point, suggestive that the belief in own rationality is, at least to some extent, combined with expectation that it won’t pass the test and subsequent avoidance (rather than seeking) of tests; as when psychics do believe in their powers but do avoid any reliable test.