Is there any evidence that this is actually a general inductor, i.e. that as a prior it dominates some large class of functions? From skimming the paper it sounds like this could be interesting progress in ILP, but not necessarily groundbreaking or close to being a fully general inductor. At the moment I’d be more concerned about the transformer architecture potentially being used as (part of) a general inductor.
My impression is that it’s interesting because it’s good at some functions that deep learning is bad at (although unfortunately the paper doesn’t make any toe-to-toe comparisons), but certainly there’s a lot of things in which transformers would beat it. In particular I would be very surprised if it could reproduce GPT3 or DALL-E. So, if this leads to a major breakthrough it would probably be through merging it with deep learning somehow.
Is there any evidence that this is actually a general inductor, i.e. that as a prior it dominates some large class of functions? From skimming the paper it sounds like this could be interesting progress in ILP, but not necessarily groundbreaking or close to being a fully general inductor. At the moment I’d be more concerned about the transformer architecture potentially being used as (part of) a general inductor.
My impression is that it’s interesting because it’s good at some functions that deep learning is bad at (although unfortunately the paper doesn’t make any toe-to-toe comparisons), but certainly there’s a lot of things in which transformers would beat it. In particular I would be very surprised if it could reproduce GPT3 or DALL-E. So, if this leads to a major breakthrough it would probably be through merging it with deep learning somehow.
I’m not aware of a technical definition of “general inductor”. I meant that it’s an inductor that is quite general.