It may be the case that solving inner alignment problems means hitting a narrow target; meaning that if we naively carry out a super-large-scale training process that spits out a huge AGI-level NN, dangerous logic is very likely to arise somewhere in the NN at some point during training. Since this concern doesn’t point at any specific-type-of-dangerous-logic I guess it’s not what you’re after in this post; but I wouldn’t classify it as part of the threat model that “we don’t know what we don’t know”.
Having said all that, here’s an attempt at describing a specific scenario as requested:
Suppose we finally train our AGI-level GPT-N and we think that the distribution it learned is “the human writing distribution”, HWD for short. HWD is a distribution that roughly corresponds to our credences when answering questions like “which of these two strings is more likely to have appeared on the internet prior to 2020-07-28?”. But unbeknown to us, the inductive bias of our training process made GPT-N learn the distribution HWD*, which is just like HWD except that some fraction of [the strings with a prefix that looks like “a prompt by humans-trying-to-automate-AI-safety”] are manipulative and make AI safety researchers, upon reading, invoke an AGI with a goal system X. Turns out that the inductive bias of our training process caused GPT-N to model agents-with-goal-system-X and such agents tend to sample lots of strings from the HWD* distribution in order to “steal” the cosmic endowment of reckless civilizations like ours. This would be a manifestation of is the same type of failure mode as the universal prior problem.
It may be the case that solving inner alignment problems means hitting a narrow target; meaning that if we naively carry out a super-large-scale training process that spits out a huge AGI-level NN, dangerous logic is very likely to arise somewhere in the NN at some point during training. Since this concern doesn’t point at any specific-type-of-dangerous-logic I guess it’s not what you’re after in this post; but I wouldn’t classify it as part of the threat model that “we don’t know what we don’t know”.
Having said all that, here’s an attempt at describing a specific scenario as requested:
Suppose we finally train our AGI-level GPT-N and we think that the distribution it learned is “the human writing distribution”, HWD for short. HWD is a distribution that roughly corresponds to our credences when answering questions like “which of these two strings is more likely to have appeared on the internet prior to 2020-07-28?”. But unbeknown to us, the inductive bias of our training process made GPT-N learn the distribution HWD*, which is just like HWD except that some fraction of [the strings with a prefix that looks like “a prompt by humans-trying-to-automate-AI-safety”] are manipulative and make AI safety researchers, upon reading, invoke an AGI with a goal system X. Turns out that the inductive bias of our training process caused GPT-N to model agents-with-goal-system-X and such agents tend to sample lots of strings from the HWD* distribution in order to “steal” the cosmic endowment of reckless civilizations like ours. This
would be a manifestation ofis the same type of failure mode as the universal prior problem.