I don’t know who first said it, but the popular saying “Computer vision is the inverse of computer graphics” encompasses much of this viewpoint.
Computer graphics is the study/art of the approximation theory you mention and fairly well developed & understood in terms of how to best simulate worlds & observations in real-time from the perspective of an observer. But of course traditional graphics uses human-designed world models and algorithms.
Diffusion models provide a general framework for learning a generative model in the other direction—in part by inverting trained vision and noise models.
So naturally there is also diffusion planning which is an example of the symmetry you discuss: using general diffusion inference for planning. The graph dimensions end up being both space-time and abstraction level with the latter being more important: sensor inference moves up the abstraction/compression hierarchy, whereas planning/acting/generating moves down.
I don’t know who first said it, but the popular saying “Computer vision is the inverse of computer graphics” encompasses much of this viewpoint.
Computer graphics is the study/art of the approximation theory you mention and fairly well developed & understood in terms of how to best simulate worlds & observations in real-time from the perspective of an observer. But of course traditional graphics uses human-designed world models and algorithms.
Diffusion models provide a general framework for learning a generative model in the other direction—in part by inverting trained vision and noise models.
So naturally there is also diffusion planning which is an example of the symmetry you discuss: using general diffusion inference for planning. The graph dimensions end up being both space-time and abstraction level with the latter being more important: sensor inference moves up the abstraction/compression hierarchy, whereas planning/acting/generating moves down.
Yeah, I’d looked at computer graphics myself. I expect that field does have some generalizable lessons.
Great addition regarding diffusion planning.