There are two things that should be distinguished:
Given one’s current beliefs, how to update them given new evidence.
How to get the machine started: how to get an initial set of beliefs on the basis of no evidence
The answer to the first is very simple: Bayes’ rule. Even if you do not have numbers to plug in, there are nevertheless some principles following from Bayes’ rule that can still be applied. For example, avoiding the conjunction fallacy, bottom-line reasoning, suggestive variable names, failure to entangle with reality, taking P and not-P both as evidence for a favoured idea, and so on. These are all written of in the Sequences.
Answering the second generally leads to speculation about Solomonoff priors, the maxent principle, improper priors, and so on, and I am not sure has contributed much. But the same is true of every other attempt to find an ultimate foundation for knowledge. A rock cannot discover Bayes’ rule, and what is the tabula rasa of an agent but a rock?
Another direction which has so far led only to more philosophical floundering is trying to apply probabilistic reasoning at the meta-level: what is the probability that our methods of logical reasoning are sound? Whatever conclusions we come to about that, what is the probability that those conclusions are true? And so on. Nothing good has yet come of this. It is a standard observation that despite all the work that has been done on non-standard logic, at the meta-level everyone reverts to good old standard logic. The only innovation that is used at the meta-level, because it was there all along, is constructiveness. When a mathematician proves something, he actually exhibits a proof, not a non-constructive argument that there exists a proof. But Euclid did that already.
It’s not trivially clear at all to me how bayes rule leads into such things as bottom-line reasoning avoidance. It seems plausible for me that for a lot of people there is enough handwaveing that the actual words put forward don’t do the majority of the lifting. That is a person talking about epistemology refers to bayes rule and explains why habits like avoiding bottom-line reasoning are good but they don’t materially need bayes rule for that. There might be belief in entailment rather than actual reproducable / objectively verifable entailment. If I wave a giant “2+2=4” flag while robbing a bunch of banks in one sense that fact has caused theft and in another it has not. Neither is is clear that anyone that robs banks must believe “2+2=4″.
You can avoid these things while knowing nothing of the Way of Bayes, but the Way of Bayes shows the underlying structure of reality that unifies the reasons for all of them being faults of reasoning.
I am unsure whether the dogmatic tone is put forward in sarcasm or as just plain straight.
One could argue that God is a convenient and unified way about thinking what is moral. And it is quite common for prisoners to find great utility in faith. But beliefs with the structure of “godlessness is dangerous as then there would be no right and wrong.” cloud thinking a lot and tie the beliefs to a specific ontology. Are beliefs like “it’s only good in so far that it aligns with God” and “it’s reasonable only so far as it aligns with bayes” meaningfully structurally different?
What if there is a deeper way of thinking why certain cognitve moves are good at even more unified view? What principles do we use to verify that the way of bayes checks out?
I am being quite straight, although consciously adopting some of Eliezer’s rhetorical style.
Are beliefs like “it’s only good in so far that it aligns with God” and “it’s reasonable only so far as it aligns with bayes” meaningfully structurally different?
They are meaningfully different. The Way of Bayes works; the Way of God does not. “Works” means “capable of leading one’s beliefs to align more closely with reality.” For more about this, it’s all there in the Sequences.
The analog I was shooting for was “thing is X only in so far that it approximates Y” where in one case X=good, Y= God and another case X=reasonable and Y=Bayes. The case of X=reasonable and Y=God doesn’t impact anything (althought I guess that the stance that there is a divine gatekeeper to truth isn’t a totally alien one, but I was not referencing it here).
Part of the reason as far as I understood for the rhetorical style is to make the silly things jump out as silly to not vest too serious weight in it.
There is the addiotional issue that rationalist are not particularly winning so the case of “one is broken, one is legit” can be questioned. Because of the heavy redefinition or questinioning of definitions it can be hard to verify that epikunfukas succeed in a metric other than the one defined by their teacher. This despite one of the central points being the reliance on external measures for success. If you follow fervently the teacher that teaches that you should not follow your teacher blindly you are still fervently following. That you have a model that refers to itself as making two variables close to each other doesn’t say whether it is a good model (“I am a true model” is not informative).
Part of the reason as far as I understood for the rhetorical style is to make the silly things jump out as silly to not vest too serious weight in it.
I can’t speak for Eliezer, but my intention is to imply that this is actually as important as the linguistic devices say it is. There is no irony intended here, no buffer of plausible deniability against being thought to be serious.
I can’t make any sense of your last paragraph, and the non-word “epikunfukas” is the least of it.
That depends on lot how narrowly or widely we interpret things. It could make a lot of sense that “updating correctly” also includes not updating into a eventual deadend and updating in a way that can faciliate paradigm shifts.
It might be worth noting that for some “updating” can refer to a very narrow process involving explicitly and conciously formulated numbers. but theorethical mullings over bayes rule can also include appriciation for the wiggleroom ie 99.999% vs 99.9% differing in the degree how much emphasis for totally different paradigms is given.
That depends on lot how narrowly or widely we interpret things
Indeed. Where Bayes is a taken loosely series of maxims, you can add some n advice about not flogging a dead horse.
But if Bayes means a formal, mathematical method, there is nothing in it to tell you to stop incrementally updating the same framing of a problem,and nothing to help you come up with a bold new hypothesis.
Whether there is nothing when things are interpreted formally is a very hard claim to prove. If a framing of a problem dissatisfactory it can be incrementally updated away from too. If you have a problem and try to solve it with epicycles that there are model deficiencies that do weaken the reliability of the whole paradigm.
Or stated in another way every hypothesis always has a positive probability, we never stop considering any hypothesis so we are guaranteed to not “miss out” on any of them (0 doesn’t occur as a probability and can never arise). Only in approximations do we have a cutout that sufficiently small probabilities are not given any thought.
There migth be a problem how to design the hypothesis space to be truly universally representative. In approximations we define hypotheses to be some pretty generic class of statements which has something parameter like and we try to learn what are the appropriate values of the parameters. But coming up with an even more generic class might allow for hypotheses which have a better structure that better fit. In the non-approximation case we don’t get “imagination starved” because we don’t specify what pool the hypotheses are drawn from.
For more concrete example, if have a set of points and ask what line best fits you are going to get a bad result if the data is in a shape of a parabola. You could catch parabolas if you asked “what polynomial best fits these points?”. But trying to ask a question like “what is the best thought that explains these observations” is resistant to be made formal because “thought” is so nebolous. No amount of line fitting will suggest a parabola but some amount of polynomial fittings will suggest parabolas over lines.
Whether there is nothing when things are interpreted formally
I didn’t say there is nothing when things are interpreted formally. I said the formalism of Bayesian probability does not include a formula for generating novel hypotheses, and that is easy to prove.
If a framing of a problem dissatisfactory it can be incrementally updated away from too.
Can it? That doesn’t seem to be how things work in practice. There is a set of revolutions in science, and inasmuch as they are revolutions,they are not slow incremental changes.
Or stated in another way every hypothesis always has a positive probability, we never stop considering any hypothesis
We don’t have every hypothesis pre existing in our heads. If you were some sort of ideal reasoner with an infinite memory , you could do things that way, but you’re not. Cognitive limitations may well explain the existence of revolutionary paradigm shifts.
But trying to ask a question like “what is the best thought that explains these observations” is resistant to be made formal because “thought” is so nebolous
That what I was saying. You can’t formalise hypothesis fornation, yet it is necessary. Therefore, formal Bayes is not the one epistemology to rule then all, because all formalisations have that shortcoming.
There are two things that should be distinguished:
Given one’s current beliefs, how to update them given new evidence.
How to get the machine started: how to get an initial set of beliefs on the basis of no evidence
The answer to the first is very simple: Bayes’ rule. Even if you do not have numbers to plug in, there are nevertheless some principles following from Bayes’ rule that can still be applied. For example, avoiding the conjunction fallacy, bottom-line reasoning, suggestive variable names, failure to entangle with reality, taking P and not-P both as evidence for a favoured idea, and so on. These are all written of in the Sequences.
Answering the second generally leads to speculation about Solomonoff priors, the maxent principle, improper priors, and so on, and I am not sure has contributed much. But the same is true of every other attempt to find an ultimate foundation for knowledge. A rock cannot discover Bayes’ rule, and what is the tabula rasa of an agent but a rock?
Another direction which has so far led only to more philosophical floundering is trying to apply probabilistic reasoning at the meta-level: what is the probability that our methods of logical reasoning are sound? Whatever conclusions we come to about that, what is the probability that those conclusions are true? And so on. Nothing good has yet come of this. It is a standard observation that despite all the work that has been done on non-standard logic, at the meta-level everyone reverts to good old standard logic. The only innovation that is used at the meta-level, because it was there all along, is constructiveness. When a mathematician proves something, he actually exhibits a proof, not a non-constructive argument that there exists a proof. But Euclid did that already.
It’s not trivially clear at all to me how bayes rule leads into such things as bottom-line reasoning avoidance. It seems plausible for me that for a lot of people there is enough handwaveing that the actual words put forward don’t do the majority of the lifting. That is a person talking about epistemology refers to bayes rule and explains why habits like avoiding bottom-line reasoning are good but they don’t materially need bayes rule for that. There might be belief in entailment rather than actual reproducable / objectively verifable entailment. If I wave a giant “2+2=4” flag while robbing a bunch of banks in one sense that fact has caused theft and in another it has not. Neither is is clear that anyone that robs banks must believe “2+2=4″.
You can avoid these things while knowing nothing of the Way of Bayes, but the Way of Bayes shows the underlying structure of reality that unifies the reasons for all of them being faults of reasoning.
I am unsure whether the dogmatic tone is put forward in sarcasm or as just plain straight.
One could argue that God is a convenient and unified way about thinking what is moral. And it is quite common for prisoners to find great utility in faith. But beliefs with the structure of “godlessness is dangerous as then there would be no right and wrong.” cloud thinking a lot and tie the beliefs to a specific ontology. Are beliefs like “it’s only good in so far that it aligns with God” and “it’s reasonable only so far as it aligns with bayes” meaningfully structurally different?
What if there is a deeper way of thinking why certain cognitve moves are good at even more unified view? What principles do we use to verify that the way of bayes checks out?
I am being quite straight, although consciously adopting some of Eliezer’s rhetorical style.
They are meaningfully different. The Way of Bayes works; the Way of God does not. “Works” means “capable of leading one’s beliefs to align more closely with reality.” For more about this, it’s all there in the Sequences.
The analog I was shooting for was “thing is X only in so far that it approximates Y” where in one case X=good, Y= God and another case X=reasonable and Y=Bayes. The case of X=reasonable and Y=God doesn’t impact anything (althought I guess that the stance that there is a divine gatekeeper to truth isn’t a totally alien one, but I was not referencing it here).
Part of the reason as far as I understood for the rhetorical style is to make the silly things jump out as silly to not vest too serious weight in it.
There is the addiotional issue that rationalist are not particularly winning so the case of “one is broken, one is legit” can be questioned. Because of the heavy redefinition or questinioning of definitions it can be hard to verify that epikunfukas succeed in a metric other than the one defined by their teacher. This despite one of the central points being the reliance on external measures for success. If you follow fervently the teacher that teaches that you should not follow your teacher blindly you are still fervently following. That you have a model that refers to itself as making two variables close to each other doesn’t say whether it is a good model (“I am a true model” is not informative).
I can’t speak for Eliezer, but my intention is to imply that this is actually as important as the linguistic devices say it is. There is no irony intended here, no buffer of plausible deniability against being thought to be serious.
I can’t make any sense of your last paragraph, and the non-word “epikunfukas” is the least of it.
There’s a third thing: how do you realise that incremental updated are no longer working,and you need a revolutionary shift to another paradigm.
That depends on lot how narrowly or widely we interpret things. It could make a lot of sense that “updating correctly” also includes not updating into a eventual deadend and updating in a way that can faciliate paradigm shifts.
It might be worth noting that for some “updating” can refer to a very narrow process involving explicitly and conciously formulated numbers. but theorethical mullings over bayes rule can also include appriciation for the wiggleroom ie 99.999% vs 99.9% differing in the degree how much emphasis for totally different paradigms is given.
Indeed. Where Bayes is a taken loosely series of maxims, you can add some n advice about not flogging a dead horse. But if Bayes means a formal, mathematical method, there is nothing in it to tell you to stop incrementally updating the same framing of a problem,and nothing to help you come up with a bold new hypothesis.
Whether there is nothing when things are interpreted formally is a very hard claim to prove. If a framing of a problem dissatisfactory it can be incrementally updated away from too. If you have a problem and try to solve it with epicycles that there are model deficiencies that do weaken the reliability of the whole paradigm.
Or stated in another way every hypothesis always has a positive probability, we never stop considering any hypothesis so we are guaranteed to not “miss out” on any of them (0 doesn’t occur as a probability and can never arise). Only in approximations do we have a cutout that sufficiently small probabilities are not given any thought.
There migth be a problem how to design the hypothesis space to be truly universally representative. In approximations we define hypotheses to be some pretty generic class of statements which has something parameter like and we try to learn what are the appropriate values of the parameters. But coming up with an even more generic class might allow for hypotheses which have a better structure that better fit. In the non-approximation case we don’t get “imagination starved” because we don’t specify what pool the hypotheses are drawn from.
For more concrete example, if have a set of points and ask what line best fits you are going to get a bad result if the data is in a shape of a parabola. You could catch parabolas if you asked “what polynomial best fits these points?”. But trying to ask a question like “what is the best thought that explains these observations” is resistant to be made formal because “thought” is so nebolous. No amount of line fitting will suggest a parabola but some amount of polynomial fittings will suggest parabolas over lines.
I didn’t say there is nothing when things are interpreted formally. I said the formalism of Bayesian probability does not include a formula for generating novel hypotheses, and that is easy to prove.
Can it? That doesn’t seem to be how things work in practice. There is a set of revolutions in science, and inasmuch as they are revolutions,they are not slow incremental changes.
We don’t have every hypothesis pre existing in our heads. If you were some sort of ideal reasoner with an infinite memory , you could do things that way, but you’re not. Cognitive limitations may well explain the existence of revolutionary paradigm shifts.
That what I was saying. You can’t formalise hypothesis fornation, yet it is necessary. Therefore, formal Bayes is not the one epistemology to rule then all, because all formalisations have that shortcoming.