I think that makes sense for “building a useful product”, but less so for “test the hypothesis that you can get aligned superhuman performance out of an unaligned-by-default intelligence, for purposes of later being more informed when you go to build an aligned, godlike intelligence.”
Right, but I’m not sure how you’d “test” for success in that scenario. Usefulness to humanity, as demonstrated by effective product use, seems to me like the only way to get a rigorous result. If you can’t measure the success or failure of an idea objectively, then the idea probably isn’t going to matter much.
I think that makes sense for “building a useful product”, but less so for “test the hypothesis that you can get aligned superhuman performance out of an unaligned-by-default intelligence, for purposes of later being more informed when you go to build an aligned, godlike intelligence.”
Right, but I’m not sure how you’d “test” for success in that scenario. Usefulness to humanity, as demonstrated by effective product use, seems to me like the only way to get a rigorous result. If you can’t measure the success or failure of an idea objectively, then the idea probably isn’t going to matter much.