I have found it fruitful to argue this case back and forth with you, thank you for defending and explaining your perspective.
I will restate my overall position, I invite you to do the same, and then I propose that we consider this 40+ comment thread concluded for now.
———
The comment of yours that (to me) started this thread was the following.
If the default path is AI’s taking over control from humans, then what is the current plan in leading AI labs? Surely all the work they put in AI safety is done to prevent exactly such scenarios. I would find it quite hard to believe that a large group of people would vigorously do something if they believed that their efforts will go to vain.
I primarily wish to argue that, given the general lack of accountability for developing machine learning systems in worlds where indeed the default outcome is doom, it should not be surprising to find out that there is a large corporation (or multiple) doing so. One should not assume that the incentives are aligned – anyone who is risking omnicide-level outcomes via investing in novel tech development currently faces no criminal penalties, fines, or international sanctions.
Given the current intellectual elite scene where a substantial number of prestigious people care about extinction level outcomes, it is also not surprising that glory-seeking companies have large departments focused on ‘ethics’ and ‘safety’ in order to look respectable to such people. Separately from any intrinsic interest, it has been a useful political chip for enticing a great deal of talent from scientific communities and communities interested in ethics to work for them (not dissimilar to how Sam Bankman-Fried managed to cause a lot of card-carrying members of the Effective Altruist philosophy and scene to work very hard to help build his criminal empire by talking a good game about utilitarianism, veganism, and the rest).
Looking at a given company’s plan for preventing doom, and noticing it does not check out, should not be followed by an assumption of adequacy and good incentives such that surely this company would not exist nor do work on AI safety if it did not have a strong plan, I must be mistaken. I believe that there is no good plan and that these companies would exist regardless of whether a good plan existed or not. Given the lack of accountability, and my belief that alignment is clearly unsolved and we fundamentally do not knowing what we’re doing, I believe the people involved are getting rich risking all of our lives and there is (currently) no justice here.
We have agreed on many points, and from the outset I believe you felt my position had some truth to it (e.g. “I do get that point that you are making, but I think this is a little bit unfair to these organizations.”). I will leave you to outline whichever overall thoughts and remaining points of agreement or disagreement that you wish.
I have found it fruitful to argue this case back and forth with you, thank you for defending and explaining your perspective.
I will restate my overall position, I invite you to do the same, and then I propose that we consider this 40+ comment thread concluded for now.
———
The comment of yours that (to me) started this thread was the following.
I primarily wish to argue that, given the general lack of accountability for developing machine learning systems in worlds where indeed the default outcome is doom, it should not be surprising to find out that there is a large corporation (or multiple) doing so. One should not assume that the incentives are aligned – anyone who is risking omnicide-level outcomes via investing in novel tech development currently faces no criminal penalties, fines, or international sanctions.
Given the current intellectual elite scene where a substantial number of prestigious people care about extinction level outcomes, it is also not surprising that glory-seeking companies have large departments focused on ‘ethics’ and ‘safety’ in order to look respectable to such people. Separately from any intrinsic interest, it has been a useful political chip for enticing a great deal of talent from scientific communities and communities interested in ethics to work for them (not dissimilar to how Sam Bankman-Fried managed to cause a lot of card-carrying members of the Effective Altruist philosophy and scene to work very hard to help build his criminal empire by talking a good game about utilitarianism, veganism, and the rest).
Looking at a given company’s plan for preventing doom, and noticing it does not check out, should not be followed by an assumption of adequacy and good incentives such that surely this company would not exist nor do work on AI safety if it did not have a strong plan, I must be mistaken. I believe that there is no good plan and that these companies would exist regardless of whether a good plan existed or not. Given the lack of accountability, and my belief that alignment is clearly unsolved and we fundamentally do not knowing what we’re doing, I believe the people involved are getting rich risking all of our lives and there is (currently) no justice here.
We have agreed on many points, and from the outset I believe you felt my position had some truth to it (e.g. “I do get that point that you are making, but I think this is a little bit unfair to these organizations.”). I will leave you to outline whichever overall thoughts and remaining points of agreement or disagreement that you wish.