But that isn’t relevant to what you are saying, because you are making a normative point: you are saying some concepts are wrong.
You know, I think I agree that the reliance on normativity intuitions is a weakness of the original post as written in April 2019. I’ve thought a lot more in the intervening 20 months, and have been working on a sequel that I hope to finish very soon (working title “Unnatural Categories Are Optimized for Deception”, current draft sitting at 8,650 words) that I think does a much better job at reducing that black box. (That is, I think the original normative claim is basically “right”, but I now have a deeper understanding of what that’s even supposed to mean.)
In summary: when I say that some concepts are wrong, or more wrong than others, I just mean that some concepts are worse than others at making probabilistic predictions. We can formalize this with specific calculations in simple examples (like the Foos clustered at [1, 2, 3] in ℝ³ in the original post) and be confident that the underlying mathematical principles apply to the real world, even if the real world is usually too complicated for us to do explicit calculations for.
This is most straightforward in cases where the causal interaction between “the map” and “the territory” goes only in the one direction “territory → map”, and where where we only have to consider one agent’s map. As we relax those simplifying assumptions, the theory has to get more complicated.
First complication: if there are multiple agents with aligned perferences but limited ability to communicate, then they potentially face coordination problems: that’s what “Schelling Categories” is about.
Second complication: if there are multiple agents whose preferences aren’t aligned, then they might have an incentive to decieve each other, making the other agent have a worse map in a way that will trick it into making decisions that benefit the first agent. (Or, a poorly-designed agent might have an incentive to deceive itself, “wireheading” on making the map look good, instead of using a map that reflects the territory to formulate plans that make the territory better.) This is what my forthcoming sequel post is about.
Third complication: if the map can affect the territory, you can have self-fulfilling (or partially-self-fulfilling, or self-negating) prophecies. I’m not sure I understand the theory of this yet.
The sense in which I deny that scientifically inaccurate maps can have compensatory kinds of usefulness, is that I think they have to fall into the second case: the apparent usefulness has to derive from deception (or wireheading). Why else would you want a model/map that makes worse predictions rather than better predictions? (Note: self-fulfilling prophecies aren’t inaccurate!)
You’re one of them.
Well, yes. I mean, I think I’m fighting for more accurate maps, but that’s (trivially) still fighting! I don’t doubt that the feeling is mutual.
I’m reminded of discussions where one person argues that a shared interest group (for concreteness, let’s say, a chess club) should remain politically neutral (as opposed to, say, issuing a collective condemnation of puppy-kicking), to which someone responds that everything is political and that therefore neutrality is just supporting the status quo (in which some number of puppies per day will continue to be kicked). There’s a sense in which it’s true that everything is political! (As it is written, refusing to act is like refusing to allow time to pass.)
I think a better counter-counter reply is not to repeat that Chess Club should be “neutral” (because I don’t know what that means, either), but rather to contend that it’s not Chess Club’s job to save the puppies of the world: we can save more puppies with a division of labor in which Chess Club focuses on Society’s chess needs, and an Anti-Puppy-Kicking League focuses on Society’s interest in saving puppies. (And if you think Society should care more about puppies and less about chess, you should want to defund Chess Club rather than having it issue collective statements.)
Similarly, but even more fundamentally, it’s not the map’s job to provide compensatory usefulness; the map’s job is to reflect the territory. In a world where agents are using maps to make decisions, you probably can affect the territory by distorting the map for purposes that aren’t about maximizing predictive accuracy! It’s just really bad AI design, because by the very nature of the operation, you’re sabotaging your ability to tell whether your intervention is actually making things better.
In summary: when I say that some concepts are wrong, or more wrong than others, I just mean that some concepts are worse than others at making probabilistic predictions.
That would be true if the only useful thing, or the only thing anyone does, is making probability calculations
The sense in which I deny that scientifically inaccurate maps can have compensatory kinds of usefulness, is that I think they have to fall into the second case: the apparent usefulness has to derive from deception (or wireheading). Why else would you want a model/map that makes worse predictions rather than better predictions?
Because you are doing something other than prediction.
What specific other thing are you doing besides prediction? If you can give me a specific example, I think I should be able to reply with either (a) “that’s a prediction”, (b) “that’s coordination”, (c) “here’s an explanation of why that’s deception/wireheading in the technical sense I’ve described”, (d) “that’s a self-fulfilling prophecy”, or (e) “whoops, looks like my philosophical thesis isn’t quite right and I need to do some more thinking; thanks TAG!!”.
(I should be able to reply eventually; no promises on turnaround time because I’m coping with the aftermath of a crisis that I’m no longer involved in, but for which I have both a moral responsibility and selfish interest to reflect and repent on my role in.)
Why are (b) and (d) not exceptions to your thesis, already?
FYI I am also pretty confused about this. Have you (Zack) previously noted something somewhere about “that’s coordination”… and… somehow wrapping that around to “but words are just for prediction anyway?”.
“That’s deception/wireheading” feels like a reasonable, key thing to be aware of. I think you’re maybe trying to build towards something like “and a lot of coordination is oriented around deception, and that’s bad, or suboptimal, or at least sad”, but not sure.
The newer “Unnatural Categories” post seemed to build towards that, but then completely ignored the question of nation-border category boundaries which seemed pretty key.
(Overall I feel pretty happy to watch you explore this entire line of reasoning deeply over the years and learn from it. I think intellectual progress depends a lot on people picking a bunch of assumptions and running with them deeply and then reporting their findings publicly. But I currently feel like there’s a pretty gaping hole in your arguments that have something-or-other-to-do-with “but, like, coordination tho”)
Have you (Zack) previously noted something somewhere about “that’s coordination”… and… somehow wrapping that around to “but words are just for prediction anyway?”.
Why are (b) and (d) not exceptions to your thesis, already?
You surely need to argue that exceptions to everything-is-prediction are i) non existent, or ii) minor or iii) undesirable, normatively wrong.
But co ordination is extremely valuable.
And “self fulfilling prophecy” is basically looking at creation and construction through the lens of prediction.
Making things is important. If you build something according to a blueprint, it will happen to be the case that once it is built, the blueprint describes it, but that is incidental.
You can make predictions about money, but that is not the central purpose of money.
In summary: when I say that some concepts are wrong, or more wrong than others, I just mean that some concepts are worse than others at making probabilistic predictions.
That would be true if the only useful thing, or the only thing anyone does, is making probability calculations.
We can formalize this with specific calculations
You can formalise the claim that some concepts are worse than others at making probabilistic predictions, as such, but that doesnt give you the further claim that ” the only useful thing, or the only thing anyone does, is making probability calculations”.
You know, I think I agree that the reliance on normativity intuitions is a weakness of the original post as written in April 2019. I’ve thought a lot more in the intervening 20 months, and have been working on a sequel that I hope to finish very soon (working title “Unnatural Categories Are Optimized for Deception”, current draft sitting at 8,650 words) that I think does a much better job at reducing that black box. (That is, I think the original normative claim is basically “right”, but I now have a deeper understanding of what that’s even supposed to mean.)
In summary: when I say that some concepts are wrong, or more wrong than others, I just mean that some concepts are worse than others at making probabilistic predictions. We can formalize this with specific calculations in simple examples (like the Foos clustered at [1, 2, 3] in ℝ³ in the original post) and be confident that the underlying mathematical principles apply to the real world, even if the real world is usually too complicated for us to do explicit calculations for.
This is most straightforward in cases where the causal interaction between “the map” and “the territory” goes only in the one direction “territory → map”, and where where we only have to consider one agent’s map. As we relax those simplifying assumptions, the theory has to get more complicated.
First complication: if there are multiple agents with aligned perferences but limited ability to communicate, then they potentially face coordination problems: that’s what “Schelling Categories” is about.
Second complication: if there are multiple agents whose preferences aren’t aligned, then they might have an incentive to decieve each other, making the other agent have a worse map in a way that will trick it into making decisions that benefit the first agent. (Or, a poorly-designed agent might have an incentive to deceive itself, “wireheading” on making the map look good, instead of using a map that reflects the territory to formulate plans that make the territory better.) This is what my forthcoming sequel post is about.
Third complication: if the map can affect the territory, you can have self-fulfilling (or partially-self-fulfilling, or self-negating) prophecies. I’m not sure I understand the theory of this yet.
The sense in which I deny that scientifically inaccurate maps can have compensatory kinds of usefulness, is that I think they have to fall into the second case: the apparent usefulness has to derive from deception (or wireheading). Why else would you want a model/map that makes worse predictions rather than better predictions? (Note: self-fulfilling prophecies aren’t inaccurate!)
Well, yes. I mean, I think I’m fighting for more accurate maps, but that’s (trivially) still fighting! I don’t doubt that the feeling is mutual.
I’m reminded of discussions where one person argues that a shared interest group (for concreteness, let’s say, a chess club) should remain politically neutral (as opposed to, say, issuing a collective condemnation of puppy-kicking), to which someone responds that everything is political and that therefore neutrality is just supporting the status quo (in which some number of puppies per day will continue to be kicked). There’s a sense in which it’s true that everything is political! (As it is written, refusing to act is like refusing to allow time to pass.)
I think a better counter-counter reply is not to repeat that Chess Club should be “neutral” (because I don’t know what that means, either), but rather to contend that it’s not Chess Club’s job to save the puppies of the world: we can save more puppies with a division of labor in which Chess Club focuses on Society’s chess needs, and an Anti-Puppy-Kicking League focuses on Society’s interest in saving puppies. (And if you think Society should care more about puppies and less about chess, you should want to defund Chess Club rather than having it issue collective statements.)
Similarly, but even more fundamentally, it’s not the map’s job to provide compensatory usefulness; the map’s job is to reflect the territory. In a world where agents are using maps to make decisions, you probably can affect the territory by distorting the map for purposes that aren’t about maximizing predictive accuracy! It’s just really bad AI design, because by the very nature of the operation, you’re sabotaging your ability to tell whether your intervention is actually making things better.
That would be true if the only useful thing, or the only thing anyone does, is making probability calculations
Because you are doing something other than prediction.
What specific other thing are you doing besides prediction? If you can give me a specific example, I think I should be able to reply with either (a) “that’s a prediction”, (b) “that’s coordination”, (c) “here’s an explanation of why that’s deception/wireheading in the technical sense I’ve described”, (d) “that’s a self-fulfilling prophecy”, or (e) “whoops, looks like my philosophical thesis isn’t quite right and I need to do some more thinking; thanks TAG!!”.
(I should be able to reply eventually; no promises on turnaround time because I’m coping with the aftermath of a crisis that I’m no longer involved in, but for which I have both a moral responsibility and selfish interest to reflect and repent on my role in.)
Seconding TAG’s:
FYI I am also pretty confused about this. Have you (Zack) previously noted something somewhere about “that’s coordination”… and… somehow wrapping that around to “but words are just for prediction anyway?”.
“That’s deception/wireheading” feels like a reasonable, key thing to be aware of. I think you’re maybe trying to build towards something like “and a lot of coordination is oriented around deception, and that’s bad, or suboptimal, or at least sad”, but not sure.
The newer “Unnatural Categories” post seemed to build towards that, but then completely ignored the question of nation-border category boundaries which seemed pretty key.
(Overall I feel pretty happy to watch you explore this entire line of reasoning deeply over the years and learn from it. I think intellectual progress depends a lot on people picking a bunch of assumptions and running with them deeply and then reporting their findings publicly. But I currently feel like there’s a pretty gaping hole in your arguments that have something-or-other-to-do-with “but, like, coordination tho”)
Yes! You commented on it!
Have now re-read. Am actually a bit sad I didn’t notice that post to nominate it.
Whelp! Off to re-read. Thank you sir.
Why are (b) and (d) not exceptions to your thesis, already?
You surely need to argue that exceptions to everything-is-prediction are i) non existent, or ii) minor or iii) undesirable, normatively wrong.
But co ordination is extremely valuable.
And “self fulfilling prophecy” is basically looking at creation and construction through the lens of prediction. Making things is important. If you build something according to a blueprint, it will happen to be the case that once it is built, the blueprint describes it, but that is incidental.
You can make predictions about money, but that is not the central purpose of money.
That would be true if the only useful thing, or the only thing anyone does, is making probability calculations.
You can formalise the claim that some concepts are worse than others at making probabilistic predictions, as such, but that doesnt give you the further claim that ” the only useful thing, or the only thing anyone does, is making probability calculations”.