One caveat is that on my models of AI development I don’t expect the CEO could just copy model parameters to the intern. I think it’s more likely that we have something along the lines of “graduate of <specific college major>” AI systems that you then copy and use as needed. But I don’t think this really affects your point.
Sure, you could assert that AI systems replace humans one-for-one, but if this is not the best design, then there may be competitive pressure to move away from this towards something less interpretable.
Yeah jtbc I definitely would not assert this. If I had to make an argument for as-much-interpretability, it would be something like “in the scenario we’re considering, AI systems are roughly human-level in capability; at this level of capability societal organization will still require a lot of modularity; if we know nothing else and assume agents are as black-boxy as humans, it seems reasonable to assume this will lead to a roughly similar amount of interpretability as current society”. But this is not a particularly strong argument, especially in the face of vast uncertainty about what the future looks like.
Yup, all of that sounds right to me!
One caveat is that on my models of AI development I don’t expect the CEO could just copy model parameters to the intern. I think it’s more likely that we have something along the lines of “graduate of <specific college major>” AI systems that you then copy and use as needed. But I don’t think this really affects your point.
Yeah jtbc I definitely would not assert this. If I had to make an argument for as-much-interpretability, it would be something like “in the scenario we’re considering, AI systems are roughly human-level in capability; at this level of capability societal organization will still require a lot of modularity; if we know nothing else and assume agents are as black-boxy as humans, it seems reasonable to assume this will lead to a roughly similar amount of interpretability as current society”. But this is not a particularly strong argument, especially in the face of vast uncertainty about what the future looks like.