Arbitrarily good prediction of human-generated text can demand arbitrarily high superhuman intelligence.
Simple demonstration #1: Somewhere on the net, probably even in the GPT training sets, is a list of <hash, plaintext> pairs, in that order.
Simple demonstration #2: Train on only science papers up until 2010, each preceded by date and title, and then ask the model to generate starting from titles and dates in 2020.
My reply to a similar statement Eliezer made on Twitter today:
Reversing hashes nicely drives home that the training objective is technically vastly superhuman. But such far-fetched examples might give the impression that we’re not going to get any superhuman capabilities realistically/any time soon with SSL.
There are much more tractable superhuman capabilities that I expect current and near future LLMs to learn, such as having a much more universal “type of guy” prior than any individual human, modeling statistical regularities that no humans think about, stream-of-thought outputting of types of text that would require painstaking effort/ revision/collaboration for humans to create. Etc.
Statistical analysis of an audio recording of someone typing at a keyboard is sufficient to reverse engineer keystrokes. Clever humans figured this out, but there are many more evidential entanglements like this lurking that no one has thought of, but will be transparent to LLMs.
The 2020 extrapolation example gets at a more realistic class of capability that even GPT-3 has to a nonzero extent, and which will scale more continuously in the current regime with practical implications.
Scientific papers describe facts about the real world that aren’t fully determined by previous scientific papers.
Take for example the scientific papers describing a new species of bacteria that was unknown a decade earlier. Nothing in the training data describes it. You can also not determine the properties of the species based on first principles.
On the other hand, it might be possible to figure out an algorithm that does create texts that fit to given hash values.
If you are intelligent enough, you can deduce the laws of the universe from a surprisingly small amount of data. In the vein of your example, there is the story of Darwin deducing the existence of a moth with a long proboscis after seeing an orchid with a particular shape, and proving to be right. Perhaps papers from pre-2010 don’t have the right models, but maybe they have enough information and data for a sufficiently intelligent being to piece together from them whatever is missing?
Sure. What isn’t clear is that you get a real paper from 2020, not a piece of fiction that could have been written in 2010. (Or just a typo filled science paper)
Simple demonstration #2: Train on only science papers up until 2010, each preceded by date and title, and then ask the model to generate starting from titles and dates in 2020.
Arbitrarily superintelligent non-causally-trained models will probably still fail at this. IID breaks that kind of prediction. you’d need to train them in a way that makes causally invalid models implausible hypotheses.
They’re folk theorems, not conjectures. The demonstration is that, in principle, you can go on reducing the losses at prediction of human-generated text by spending more and more and more intelligence, far far past the level of human intelligence or even what we think could be computed by using all the negentropy in the reachable universe. There’s no realistic limit on required intelligence inherent in the training problem; any limits on the intelligence of the system come from the limitations of the trainer, not the loss being minimized as far as theoretically possible by a moderate level of intelligence. If this isn’t mathematically self-evident then you have not yet understood what’s being stated.
No, I didn’t understand what you said. It seemed like you simplified ML systems with a look up table in #1. In #2, it seems like you know what exactly is used to train these systems, and somehow papers before or after 2010 is of meaningful indicators for ML systems, which I don’t know where the reasoning came from. My apologies for not being knowledgeable in this area.
The two examples were (mostly) unrelated and served to demonstrate two cases where a perfect text predictor needs to do incredibly complex calculation to correctly predict text. Thus a perfect text predictor is vast superintelligence (and we won’t achieve perfect text prediction, but as we get better and better we might get closer to superintelligence)
In the first case, if the training data contains series of [hash] then [plain text], then a correct predictor must be able to retrieve the plain text from the hash (and because there are multiple plain texts with the same hash, it would have to calculate through all of them and evaluate which is most probable to appear). Thus correctly predicting text can mean being able to calculate an incredibly large amount of hashes on all combinations of text of certain lengths and evaluating which is the most probable.
In the second case, the task is to predict future papers based on past papers, which is kinda obviously very hard.
It doesn’t seem clear to me what those two demonstrations are trying to test. 1 seems like a case of over-fitting. 2 seems like an extension of 1 except it’s the case with papers, not sure how the papers case has anything to do with the generalized capabilities of ChatGPT. If you think ChatGPT is merely a complex lookup-table, then I don’t really know what to say. Lookup-table or NLP, I don’t know how either has much to do with general intelligence. Both are models that may seem intelligent if that’s where the discussion is focusing on. Honestly, I don’t really understand a lot of the stuff discussed on this site.
Arbitrarily good prediction of human-generated text can demand arbitrarily high superhuman intelligence.
Simple demonstration #1: Somewhere on the net, probably even in the GPT training sets, is a list of <hash, plaintext> pairs, in that order.
Simple demonstration #2: Train on only science papers up until 2010, each preceded by date and title, and then ask the model to generate starting from titles and dates in 2020.
My reply to a similar statement Eliezer made on Twitter today:
The 2020 extrapolation example gets at a more realistic class of capability that even GPT-3 has to a nonzero extent, and which will scale more continuously in the current regime with practical implications.
It’s not clear that it’s possible for a transformer model to do #2 no matter how much training went into it.
It’d take less computing power than #1.
Scientific papers describe facts about the real world that aren’t fully determined by previous scientific papers.
Take for example the scientific papers describing a new species of bacteria that was unknown a decade earlier. Nothing in the training data describes it. You can also not determine the properties of the species based on first principles.
On the other hand, it might be possible to figure out an algorithm that does create texts that fit to given hash values.
If you are intelligent enough, you can deduce the laws of the universe from a surprisingly small amount of data. In the vein of your example, there is the story of Darwin deducing the existence of a moth with a long proboscis after seeing an orchid with a particular shape, and proving to be right. Perhaps papers from pre-2010 don’t have the right models, but maybe they have enough information and data for a sufficiently intelligent being to piece together from them whatever is missing?
You can piece together some things, but there’s a lot of randomness in our world. A lot of important science is about discovering black swans.
Some things is enough, you’d still get less loss if you’re just right about the stuff that can be pieced together.
Sure. What isn’t clear is that you get a real paper from 2020, not a piece of fiction that could have been written in 2010. (Or just a typo filled science paper)
Arbitrarily superintelligent non-causally-trained models will probably still fail at this. IID breaks that kind of prediction. you’d need to train them in a way that makes causally invalid models implausible hypotheses.
But, also, if you did that, then yes, agreed.
These demonstrations seem like grossly over-simplified conjectures. Is this just a thought experiment or actual research interests in the field?
They’re folk theorems, not conjectures. The demonstration is that, in principle, you can go on reducing the losses at prediction of human-generated text by spending more and more and more intelligence, far far past the level of human intelligence or even what we think could be computed by using all the negentropy in the reachable universe. There’s no realistic limit on required intelligence inherent in the training problem; any limits on the intelligence of the system come from the limitations of the trainer, not the loss being minimized as far as theoretically possible by a moderate level of intelligence. If this isn’t mathematically self-evident then you have not yet understood what’s being stated.
No, I didn’t understand what you said. It seemed like you simplified ML systems with a look up table in #1. In #2, it seems like you know what exactly is used to train these systems, and somehow papers before or after 2010 is of meaningful indicators for ML systems, which I don’t know where the reasoning came from. My apologies for not being knowledgeable in this area.
The two examples were (mostly) unrelated and served to demonstrate two cases where a perfect text predictor needs to do incredibly complex calculation to correctly predict text. Thus a perfect text predictor is vast superintelligence (and we won’t achieve perfect text prediction, but as we get better and better we might get closer to superintelligence)
In the first case, if the training data contains series of [hash] then [plain text], then a correct predictor must be able to retrieve the plain text from the hash (and because there are multiple plain texts with the same hash, it would have to calculate through all of them and evaluate which is most probable to appear). Thus correctly predicting text can mean being able to calculate an incredibly large amount of hashes on all combinations of text of certain lengths and evaluating which is the most probable.
In the second case, the task is to predict future papers based on past papers, which is kinda obviously very hard.
It doesn’t seem clear to me what those two demonstrations are trying to test. 1 seems like a case of over-fitting. 2 seems like an extension of 1 except it’s the case with papers, not sure how the papers case has anything to do with the generalized capabilities of ChatGPT. If you think ChatGPT is merely a complex lookup-table, then I don’t really know what to say. Lookup-table or NLP, I don’t know how either has much to do with general intelligence. Both are models that may seem intelligent if that’s where the discussion is focusing on. Honestly, I don’t really understand a lot of the stuff discussed on this site.