I very much agree and really like the coining of the term “teleosemantics”. I might steal it! :-)
I’m not sure how much you’ve read my work on this topic or how much it influenced you, but in case you’re not very aware of it I think it’s worth pointing out some things I’ve been working on in this space for a while that you might find interesting.
I’ve tried to explain these ideas but they seem tricky for folks to grok. They either bounce off because it seems obvious and like they have nothing to learn (I think this is not true because while it’s obvious because it’s just how things are, understanding why things are the way they are in detail is where all the value is) or they have a worldview that strongly rejects the idea that things like truth might be teleological. So I’m now writing a book about the subject to help people both appreciate the depth of the teleology of truth and argue that it’s how the world is by breaking down the ideas that prop up the worldview that disagrees. I hope to publish the first draft of the chapter on the problem of the criterion this week, and then I’ll get starting on writing the chapter about telos.
I think understanding these ideas are essential for solving alignment because we can’t even agree on what we’re fundamentally trying to align to if we don’t adequately consider this problem. Maybe we get lucky and schemes we attempt to keep AI aligned end up accounting for the inherently teleological nature of knowledge, but I’m not will to leave it to chance.
One thing I see as different between your perspective and (my understanding of) teleosemantics, so far:
You make a general case that values underlie beliefs.
Teleosemantics makes a specific claim that the meaning of semantic constructs (such as beliefs and messages) is pinned down by what it is trying to correspond to.
Your picture seems very compatible with, EG, the old LW claim that UDT’s probabilities are really a measure of caring—how much you care about doing well in a variety of scenarios.
Teleosemantics might fail to analyze such probabilities as beliefs at all; certainly not beliefs about the world. (Perhaps beliefs about how important different scenarios are, where “importance” gets some further analysis...)
The teleosemantic picture is that epistemic accuracy is a common, instrumentally convergent subgoal; and “meaning” (in the sense of semantic content) arises precisely where this subgoal is being optimized.
That’s my guess at the biggest difference between our two pictures, anyway.
The teleosemantic picture is that epistemic accuracy is a common, instrumentally convergent subgoal; and “meaning” (in the sense of semantic content) arises precisely where this subgoal is being optimized.
I think this is exactly right. I often say things like “accurate maps are extremely useful to things like survival, so you and every other living thing has strong incentives to draw accurate maps, but this is contingent on the extent to which you care about e.g. survival”.
So to see if I have this right, the difference is I’m trying to point at a larger phenomenon and you mean teleosemantics to point just at the way beliefs get constrained to be useful.
So to see if I have this right, the difference is I’m trying to point at a larger phenomenon and you mean teleosemantics to point just at the way beliefs get constrained to be useful.
This doesn’t sound quite right to me. Teleosemantics is a purported definition of belief. So according to the teleosemantic picture, it isn’t a belief if it’s not trying to accurately reflect something.
The additional statement I prefaced this with, that accuracy is an instrumentally convergent subgoal, was intended to be an explanation of why this sort of “belief” is a common phenomenon, rather than part of the definition of “belief”.
In principle, there could be a process which only optimizes accuracy and doesn’t serve any larger goal. This would still be creating and maintaining beliefs according to the definition of teleosemantics, although it would be an oddity. (How did it get there? How did a non-agentic process end up creating it?)
(Following some links...) What’s the deal with Holons?
Your linked article on epistemic circularity doesn’t really try to explain itself, but rather links to this article, which LOUDLY doesn’t explain itself.
I haven’t read much else yet, but here is what I think I get:
You use Godel’s incompleteness theorem as part of an argument that meta-rationalism can’t make itself comprehensible to rationalism.
You think (or thought at the time) that there’s a thing, Holons, or Holonic thinking, which is fundamentally really really hard to explain, but which a bunch of people (mainly Buddhists and a few of the best postmodernists) already get. Kensho vibes.
There is an “everything of everythings”, exceeding all systems, something like the highest level Tegmark multiverse only much more awesome, which is called “holon”, or God, or Buddha. We cannot approach it in far mode, but we can… somehow… fruitfully interact with it in near mode. Rationalists deny it because their preferred far-mode approach is fruitless here. But you can still “get it” without necessarily being able to explain it by words. Maybe it is actually inexplicable by words in principle, because the only sufficiently good explanation for holon/God/Buddha is the holon/God/Buddha itself. If you “get it”, you become the Kegan-level-5 meta-rationalist, and everything will start making sense. If you don’t “get it”, you will probably construct some Kegan-level-4 rationalist verbal argument for why it doesn’t make sense at all.
I’m curious whether you see any similarity between holons and object oriented ontology (if you’re at all familiar with that).
I was vibing with object oriented ontology when I wrote this, particularly the “nontrivial implication” at the end.
Here’s my terrible summary of OOO:
Everything is ‘objects’.
For practically every realism debate, OOO lands on the realist side. There is an external world, there are tables, there are chairs, and triangles, and squares, numbers, letters, moral facts, possible worlds, and even fictional characters.
Philosophy is properly understood as a form of art; and as art, especially closely related to theater.
Sciences deal with objective (measurable, quantifiable) facts; arts deal with subjective/intersubjective (immeasurable, unquantifiable) facts.
Objects are a kind of intersection of these two sorts of thing.
To understand there as being an object is in some sense to be able to put yourself in its place, empathize with it, imagine it were you. This is our way of fruitfully relating to the immeasurable/unquantifiable. So panpsychism is in some sense a true fact about our ontology.
I find OOO to be an odd mix of interesting ideas and very weird ideas.
Feel free to ignore the OOO comparison if it’s not a terribly useful comparison for holons.
Oh man I kind of wish I could go back in time and wipe out all the cringe stuff I wrote when I was trying to figure things out (like why did I need to pull in Godel or reify my confusion?). With that said, here’s some updated thoughts on holons. I’m not really familiar with OOO, so I’ll be going off your summary here.
I think I started out really not getting what the holon idea points at, but I understood enough to get myself confused in new ways for a while. So first off there’s only ~1 holon, such that it doesn’t make sense to talk about it as anything other than the whole world. Maybe you could make some case for many overlapping holons centered around each point in the universe expanding out to it’s Hubble volume, but I think that’s probably not helpful. Better to think of the holon as just the whole world, so really it’s just a weird cybernetics term for talking about the world.
The trouble was I really didn’t fully grasp the way that relative and absolute truth are not one and the same. So I was actually still fully trapped within my ontology, but holons seemed like a way to pull pre-ontological reality existing on its own inside of ontology.
OOO mostly sounds like being confused about ontology, specifically a kind of reification of the confusion that comes from not realizing that it’s maps all the way down, i.e. you only experience the world through, and it’s only through experiencing non-experience that you get to taste reality, which is an extremely mysterious answer trying to point at a thing that happens all the time but we literally can’t notice it because noticing it destroys it.
OK. So far it seems to me like we share a similar overall take, but I disagree with some of your specific framings and such. I guess I’ll try and comment on the relevant posts, even though this might imply commenting on some old stuff that you’ll end up disclaiming.
Cool. For what it’s worth, I also disagree with many of my old framings. Basically anything written more than ~1 year ago is probably vaguely but not specifically endorsed.
I very much agree and really like the coining of the term “teleosemantics”. I might steal it! :-)
I’m not sure how much you’ve read my work on this topic or how much it influenced you, but in case you’re not very aware of it I think it’s worth pointing out some things I’ve been working on in this space for a while that you might find interesting.
I got nervous about how truth works when I tried to tackle the alignment problem head on. I ended up having to write a sequence of posts to sort out my ideas. At the time, I really failed to appreciate how deep telos ran.
My later work on alignment led me down the path of trying to understand human values as a prerequisite for verifying if an alignment scheme could work in theory or if a particular AI was aligned, which took me on a side quest into metaethical uncertainty. During that side quest I learned about epistemic circularity and the problem of the criterion.
I’ve tried to explain these ideas but they seem tricky for folks to grok. They either bounce off because it seems obvious and like they have nothing to learn (I think this is not true because while it’s obvious because it’s just how things are, understanding why things are the way they are in detail is where all the value is) or they have a worldview that strongly rejects the idea that things like truth might be teleological. So I’m now writing a book about the subject to help people both appreciate the depth of the teleology of truth and argue that it’s how the world is by breaking down the ideas that prop up the worldview that disagrees. I hope to publish the first draft of the chapter on the problem of the criterion this week, and then I’ll get starting on writing the chapter about telos.
I think understanding these ideas are essential for solving alignment because we can’t even agree on what we’re fundamentally trying to align to if we don’t adequately consider this problem. Maybe we get lucky and schemes we attempt to keep AI aligned end up accounting for the inherently teleological nature of knowledge, but I’m not will to leave it to chance.
One thing I see as different between your perspective and (my understanding of) teleosemantics, so far:
You make a general case that values underlie beliefs.
Teleosemantics makes a specific claim that the meaning of semantic constructs (such as beliefs and messages) is pinned down by what it is trying to correspond to.
Your picture seems very compatible with, EG, the old LW claim that UDT’s probabilities are really a measure of caring—how much you care about doing well in a variety of scenarios.
Teleosemantics might fail to analyze such probabilities as beliefs at all; certainly not beliefs about the world. (Perhaps beliefs about how important different scenarios are, where “importance” gets some further analysis...)
The teleosemantic picture is that epistemic accuracy is a common, instrumentally convergent subgoal; and “meaning” (in the sense of semantic content) arises precisely where this subgoal is being optimized.
That’s my guess at the biggest difference between our two pictures, anyway.
I think this is exactly right. I often say things like “accurate maps are extremely useful to things like survival, so you and every other living thing has strong incentives to draw accurate maps, but this is contingent on the extent to which you care about e.g. survival”.
So to see if I have this right, the difference is I’m trying to point at a larger phenomenon and you mean teleosemantics to point just at the way beliefs get constrained to be useful.
This doesn’t sound quite right to me. Teleosemantics is a purported definition of belief. So according to the teleosemantic picture, it isn’t a belief if it’s not trying to accurately reflect something.
The additional statement I prefaced this with, that accuracy is an instrumentally convergent subgoal, was intended to be an explanation of why this sort of “belief” is a common phenomenon, rather than part of the definition of “belief”.
In principle, there could be a process which only optimizes accuracy and doesn’t serve any larger goal. This would still be creating and maintaining beliefs according to the definition of teleosemantics, although it would be an oddity. (How did it get there? How did a non-agentic process end up creating it?)
(Following some links...) What’s the deal with Holons?
Your linked article on epistemic circularity doesn’t really try to explain itself, but rather links to this article, which LOUDLY doesn’t explain itself.
I haven’t read much else yet, but here is what I think I get:
You use Godel’s incompleteness theorem as part of an argument that meta-rationalism can’t make itself comprehensible to rationalism.
You think (or thought at the time) that there’s a thing, Holons, or Holonic thinking, which is fundamentally really really hard to explain, but which a bunch of people (mainly Buddhists and a few of the best postmodernists) already get. Kensho vibes.
Not something you wrote, but Viliam trying to explain you:
I’m curious whether you see any similarity between holons and object oriented ontology (if you’re at all familiar with that).
I was vibing with object oriented ontology when I wrote this, particularly the “nontrivial implication” at the end.
Here’s my terrible summary of OOO:
Everything is ‘objects’.
For practically every realism debate, OOO lands on the realist side. There is an external world, there are tables, there are chairs, and triangles, and squares, numbers, letters, moral facts, possible worlds, and even fictional characters.
Philosophy is properly understood as a form of art; and as art, especially closely related to theater.
Sciences deal with objective (measurable, quantifiable) facts; arts deal with subjective/intersubjective (immeasurable, unquantifiable) facts.
Objects are a kind of intersection of these two sorts of thing.
To understand there as being an object is in some sense to be able to put yourself in its place, empathize with it, imagine it were you. This is our way of fruitfully relating to the immeasurable/unquantifiable. So panpsychism is in some sense a true fact about our ontology.
I find OOO to be an odd mix of interesting ideas and very weird ideas.
Feel free to ignore the OOO comparison if it’s not a terribly useful comparison for holons.
Oh man I kind of wish I could go back in time and wipe out all the cringe stuff I wrote when I was trying to figure things out (like why did I need to pull in Godel or reify my confusion?). With that said, here’s some updated thoughts on holons. I’m not really familiar with OOO, so I’ll be going off your summary here.
I think I started out really not getting what the holon idea points at, but I understood enough to get myself confused in new ways for a while. So first off there’s only ~1 holon, such that it doesn’t make sense to talk about it as anything other than the whole world. Maybe you could make some case for many overlapping holons centered around each point in the universe expanding out to it’s Hubble volume, but I think that’s probably not helpful. Better to think of the holon as just the whole world, so really it’s just a weird cybernetics term for talking about the world.
The trouble was I really didn’t fully grasp the way that relative and absolute truth are not one and the same. So I was actually still fully trapped within my ontology, but holons seemed like a way to pull pre-ontological reality existing on its own inside of ontology.
OOO mostly sounds like being confused about ontology, specifically a kind of reification of the confusion that comes from not realizing that it’s maps all the way down, i.e. you only experience the world through, and it’s only through experiencing non-experience that you get to taste reality, which is an extremely mysterious answer trying to point at a thing that happens all the time but we literally can’t notice it because noticing it destroys it.
OK. So far it seems to me like we share a similar overall take, but I disagree with some of your specific framings and such. I guess I’ll try and comment on the relevant posts, even though this might imply commenting on some old stuff that you’ll end up disclaiming.
Cool. For what it’s worth, I also disagree with many of my old framings. Basically anything written more than ~1 year ago is probably vaguely but not specifically endorsed.