GPT-2 is best described IMHO as “DeepDream for text.” They use different neural network architectures, but that’s because analyzing images and natural language require different architectures. Fundamentally their complete-the-prompt-using-training-data design is the same.
If by ‘fundamentally the same’ you mean ‘actually they’re completely different and optimize completely different things and give completely different results on completely different modalities’, then yeah, sure. (Also, a dog is an octopus.) DeepDream is a iterative optimization process which tries to maximize the class-ness of an image input (usually, dogs); a language model like GPT-2 is predicting the most likely next observation in a natural text dataset which can be fed its own guesses. They bear about as much relation as a propaganda poster and a political science paper.
Gwern, I respect you but sometimes you miss the mark. I was describing a particular application of deep dream in which the output is fed in as input, which doesn’t strike me as any different from your own description of GPT-2.
A little less hostility in your comment and it would be received better.
Feeding in output as input is exactly what is iterative about DeepDream, and the scenario does not change the fact that GPT-2 and DeepDream are fundamentally different in many important ways and there is no sense in which they are ‘fundamentally the same’, not even close.
And let’s consider the chutzpah of complaining about tone when you ended your own highly misleading comment with the snide
But by all means, spend your $1000 on it. Maybe you’ll learn something in the process.
There was no snide there. I honestly think he’ll learn something of value. I don’t think he’ll get the result he wanted, but he will learn something in the process.
If by ‘fundamentally the same’ you mean ‘actually they’re completely different and optimize completely different things and give completely different results on completely different modalities’, then yeah, sure. (Also, a dog is an octopus.) DeepDream is a iterative optimization process which tries to maximize the class-ness of an image input (usually, dogs); a language model like GPT-2 is predicting the most likely next observation in a natural text dataset which can be fed its own guesses. They bear about as much relation as a propaganda poster and a political science paper.
Gwern, I respect you but sometimes you miss the mark. I was describing a particular application of deep dream in which the output is fed in as input, which doesn’t strike me as any different from your own description of GPT-2.
A little less hostility in your comment and it would be received better.
Feeding in output as input is exactly what is iterative about DeepDream, and the scenario does not change the fact that GPT-2 and DeepDream are fundamentally different in many important ways and there is no sense in which they are ‘fundamentally the same’, not even close.
And let’s consider the chutzpah of complaining about tone when you ended your own highly misleading comment with the snide
There was no snide there. I honestly think he’ll learn something of value. I don’t think he’ll get the result he wanted, but he will learn something in the process.