Silmer’s thesis is that desire more or less covers the simple notion of subjective beauty, IE, liking what you see. But when a second player enters the game, optimizing for desirability, things get much more interesting; this phenomenon has its own special indicators, such as bright colors and symmetry. Often, “beauty” is much more about this special phenomenon.
My thesis is that mutual information captures a simple, practical notion of “aboutness”; but optimizing for mutual information carries its own special signature, such as language (IE, codified mappings). Often, “aboutness” is much more about this special phenomenon.
I just read an article that reminded me of this post. The relevant section starts with “Bender and Manning’s biggest disagreement is over how meaning is created”. Bender’s position seems to have some similarities with the thesis you present here, especially when viewed in contrast to what Manning claims is the currently more popular position that meaning can arise purely from distributional properties of language.
This got me wondering: if Bender is correct, then there is a fundamental limitation in how well (pure) language models can understand the world; are there ways to test this hypothesis, and what does it mean for alignment?
This got me wondering: if Bender is correct, then there is a fundamental limitation in how well (pure) language models can understand the world; are there ways to test this hypothesis, and what does it mean for alignment?
Well, obviously, there’s a huge problem right now with LLMs having no truth-grounding, IE not being able to distinguish between making stuff up vs trying to figure things out. I think that’s a direct consequence of only having a ‘correlational’ picture (IE the ‘manning’ view).
Interesting. I’m reminded of this definition of “beauty”.
Interesting comparison!
To spell it out a little,
Silmer’s thesis is that desire more or less covers the simple notion of subjective beauty, IE, liking what you see. But when a second player enters the game, optimizing for desirability, things get much more interesting; this phenomenon has its own special indicators, such as bright colors and symmetry. Often, “beauty” is much more about this special phenomenon.
My thesis is that mutual information captures a simple, practical notion of “aboutness”; but optimizing for mutual information carries its own special signature, such as language (IE, codified mappings). Often, “aboutness” is much more about this special phenomenon.
I just read an article that reminded me of this post. The relevant section starts with “Bender and Manning’s biggest disagreement is over how meaning is created”. Bender’s position seems to have some similarities with the thesis you present here, especially when viewed in contrast to what Manning claims is the currently more popular position that meaning can arise purely from distributional properties of language.
This got me wondering: if Bender is correct, then there is a fundamental limitation in how well (pure) language models can understand the world; are there ways to test this hypothesis, and what does it mean for alignment?
Thoughts?
Well, obviously, there’s a huge problem right now with LLMs having no truth-grounding, IE not being able to distinguish between making stuff up vs trying to figure things out. I think that’s a direct consequence of only having a ‘correlational’ picture (IE the ‘manning’ view).