An “outside view” might be to put the SI in the reference class of “groups who are trying to create a utopia” and observe that previous such efforts that have managed to gain momentum have tended to make the world worse.
I think the reality is more complicated than that, but that might be part of what motivates these kind of questions.
I think the biggest specific trust-related issue I have is with CEV—getting the utility function generation process right is really important, and in an optimal world I’d expect to see CEV subjected to a process of continual improvement and informed discussion. I haven’t seen that, but it’s hard to tell whether the SI are being overly protective of their CEV document or whether it’s just really hard getting the right people talking about it in the right way.
An “outside view” might be to put the SI in the reference class of “groups who are trying to create a utopia” and observe that previous such efforts that have managed to gain momentum have tended to make the world worse.
I think the reality is more complicated than that, but that might be part of what motivates these kind of questions.
I think the biggest specific trust-related issue I have is with CEV—getting the utility function generation process right is really important, and in an optimal world I’d expect to see CEV subjected to a process of continual improvement and informed discussion. I haven’t seen that, but it’s hard to tell whether the SI are being overly protective of their CEV document or whether it’s just really hard getting the right people talking about it in the right way.