The approach to images here is very different from Image GPT. (Though this is not the first time OpenAI has written about this approach—see the “Image VQ” results from the multi-modal scaling paper.)
In Image GPT, an image is represented as a 1D sequence of pixel colors. The pixel colors are quantized to a palette of size 512, but still represent “raw colors” as opposed to anything more abstract. Each token in the sequence represents 1 pixel.
In DALL-E, an image is represented as a 2D array of tokens from a latent code. There are 8192 possible tokens. Each token in the sequence represents “what’s going on” in a roughly 8x8 pixel region (because they use 32x32 codes for 256x256 images).
(Caveat: The mappings from pixels-->tokens and tokens-->pixels are contextual, so a token can influence pixels outside “its” 8x8 region.)
This latent code is analogous to the BPE code used to represent tokens (generally words) for text GPT. Like BPE, the code is defined before doing generative training, and is presumably fixed during generative training. Like BPE, it chunks the “raw” signal (pixels here, characters in BPE) into larger, more meaningful units.
This is like a vocabulary of 8192 “image words.” DALL-E “writes” an 32x32 array of these image words, and then a separate network “decodes” this discrete array to a 256x256 array of pixel colors.
Intuitively, this feels closer than Image GPT to mimicking what text GPT does with text. Pixels are way lower-level than words; 8x8 regions with contextual information feel closer to the level of words.
As with BPE, you get a head start over modeling the raw signal. As with BPE, the chunking may ultimately be a limiting factor. Although the chunking process here is differentiable (a neural auto-encoder), so it ought to be adaptable in a way BPE is not.
(Trivia: I’m amused that one of their visuals allows you to ask for images of triangular light bulbs—the example Yudkowsky used in LOGI to illustrate the internal complexity of superficially atomic concepts.)
Very interesting!
The approach to images here is very different from Image GPT. (Though this is not the first time OpenAI has written about this approach—see the “Image VQ” results from the multi-modal scaling paper.)
In Image GPT, an image is represented as a 1D sequence of pixel colors. The pixel colors are quantized to a palette of size 512, but still represent “raw colors” as opposed to anything more abstract. Each token in the sequence represents 1 pixel.
In DALL-E, an image is represented as a 2D array of tokens from a latent code. There are 8192 possible tokens. Each token in the sequence represents “what’s going on” in a roughly 8x8 pixel region (because they use 32x32 codes for 256x256 images).
(Caveat: The mappings from pixels-->tokens and tokens-->pixels are contextual, so a token can influence pixels outside “its” 8x8 region.)
This latent code is analogous to the BPE code used to represent tokens (generally words) for text GPT. Like BPE, the code is defined before doing generative training, and is presumably fixed during generative training. Like BPE, it chunks the “raw” signal (pixels here, characters in BPE) into larger, more meaningful units.
This is like a vocabulary of 8192 “image words.” DALL-E “writes” an 32x32 array of these image words, and then a separate network “decodes” this discrete array to a 256x256 array of pixel colors.
Intuitively, this feels closer than Image GPT to mimicking what text GPT does with text. Pixels are way lower-level than words; 8x8 regions with contextual information feel closer to the level of words.
As with BPE, you get a head start over modeling the raw signal. As with BPE, the chunking may ultimately be a limiting factor. Although the chunking process here is differentiable (a neural auto-encoder), so it ought to be adaptable in a way BPE is not.
(Trivia: I’m amused that one of their visuals allows you to ask for images of triangular light bulbs—the example Yudkowsky used in LOGI to illustrate the internal complexity of superficially atomic concepts.)
This is the link to Yudkowsky discussion of concept merging with the triangular lightbulb example: https://intelligence.org/files/LOGI.pdf#page=10
Generated lightbulb images: https://i.imgur.com/EHPwELf.png