So, while in the shower, an idea for an FAI came into my head.
My intuition tells me that if we manage to entirely formalize correct reasoning, the result will have a sort of adversarial quality: you can “prove” statements, but these proofs can be overturned by stronger disproofs. So, I figured that if you simply told two (or more) AGIs to fight over one database of information, the most rational AGI would be able to set the database to contain the correct information. (Another intuition of mine tells me that FAI is a problem of rationality: once you have a rational AGI, you can just feed it CEV or whatever.)
Of course, for this to work, two things would have to happen: one of the AGIs would have to be intelligent enough to discover the rational conclusions, and no AGI could be so much smarter than the others that it could find tons of evidence in favor of its pet truths and have the database favor them despite that they’re false.
So, I don’t think this will work very well. At least I came to it by despairing about how not everybody has an infinite amount of money and yet values it anyway, thereby making our economic system perfect!
My intuition tells me that if we manage to entirely formalize correct reasoning, the result will have a sort of adversarial quality: you can “prove” statements, but these proofs can be overturned by stronger disproofs.
That… doesn’t sound right at all. It does sound like how people intuitively think about proof/reasoning (even people smart enough to be thinking about things like, say, set theory, trying to overturn Cantor’s diagonal argument with a counterexample without actually discovering a flaw in the theorem), and how we think about debates (the guy on the left half of the screen says something, the guy on the right says the opposite, and they go back and forth taking turns making Valid Points until the CNN anchor says “We’ll have to leave it there” and the viewers are left agreeing with (1) whoever agreed with their existing beliefs, or, if neither, (2) whoever spoke last). But even if our current formal understanding of reasoning is incomplete, we know it’s not going to resemble that. Yes, Bayesian updating will cause your probability estimates to fluctuate up and down a bit as you acquire more evidence, but the pieces of evidence aren’t fighting each other, they’re collaborating on determining what your map should look like and how confident you should be.
Of course, for this to work . . . no AGI could be so much smarter than the others that it could find tons of evidence in favor of its pet truths and have the database favor them despite that they’re false.
Why would we build AGI to have “pet truths”, to engage in rationalization rather than rationality, in the first place?
But even if our current formal understanding of reasoning is incomplete, we know it’s not going to resemble that. Yes, Bayesian updating will cause your probability estimates to fluctuate up and down a bit as you acquire more evidence, but the pieces of evidence aren’t fighting each other, they’re collaborating on determining what your map should look like and how confident you should be.
Yeah. So if one guy presents only evidence in favor, and the other guy presents only evidence against, they’re adversaries. One guy can state a theory, show that all existing evidence supports it, and thereby have “proved” it, and then the other guy can state an even better theory, also supported by all the evidence but simpler, thereby overturning that proof.
Why would we build AGI to have “pet truths”, to engage in rationalization rather than rationality, in the first place?
My intuition tells me that if we manage to entirely formalize correct reasoning, the result will have a sort of adversarial quality: you can “prove” statements, but these proofs can be overturned by stronger disproofs.
Game semantics works somewhat like this; a proof is formalized as an “argument” between a Proponent and an Opponent. If an extension of game semantics to probabilistic reasoning exists, it will work much like the ‘theory of uncertain arguments’ you mention here.
I seem to have man-with-a-hammer syndrome, and my hammer is economics. Luckily, I’m using economics as a tool for designing stuff, not for understanding stuff; there is no One True Design the way there’s a One True Truth.
So, while in the shower, an idea for an FAI came into my head.
My intuition tells me that if we manage to entirely formalize correct reasoning, the result will have a sort of adversarial quality: you can “prove” statements, but these proofs can be overturned by stronger disproofs. So, I figured that if you simply told two (or more) AGIs to fight over one database of information, the most rational AGI would be able to set the database to contain the correct information. (Another intuition of mine tells me that FAI is a problem of rationality: once you have a rational AGI, you can just feed it CEV or whatever.)
Of course, for this to work, two things would have to happen: one of the AGIs would have to be intelligent enough to discover the rational conclusions, and no AGI could be so much smarter than the others that it could find tons of evidence in favor of its pet truths and have the database favor them despite that they’re false.
So, I don’t think this will work very well. At least I came to it by despairing about how not everybody has an infinite amount of money and yet values it anyway, thereby making our economic system perfect!
That… doesn’t sound right at all. It does sound like how people intuitively think about proof/reasoning (even people smart enough to be thinking about things like, say, set theory, trying to overturn Cantor’s diagonal argument with a counterexample without actually discovering a flaw in the theorem), and how we think about debates (the guy on the left half of the screen says something, the guy on the right says the opposite, and they go back and forth taking turns making Valid Points until the CNN anchor says “We’ll have to leave it there” and the viewers are left agreeing with (1) whoever agreed with their existing beliefs, or, if neither, (2) whoever spoke last). But even if our current formal understanding of reasoning is incomplete, we know it’s not going to resemble that. Yes, Bayesian updating will cause your probability estimates to fluctuate up and down a bit as you acquire more evidence, but the pieces of evidence aren’t fighting each other, they’re collaborating on determining what your map should look like and how confident you should be.
Why would we build AGI to have “pet truths”, to engage in rationalization rather than rationality, in the first place?
Yeah. So if one guy presents only evidence in favor, and the other guy presents only evidence against, they’re adversaries. One guy can state a theory, show that all existing evidence supports it, and thereby have “proved” it, and then the other guy can state an even better theory, also supported by all the evidence but simpler, thereby overturning that proof.
We wouldn’t do it on purpose!
Game semantics works somewhat like this; a proof is formalized as an “argument” between a Proponent and an Opponent. If an extension of game semantics to probabilistic reasoning exists, it will work much like the ‘theory of uncertain arguments’ you mention here.
I seem to have man-with-a-hammer syndrome, and my hammer is economics. Luckily, I’m using economics as a tool for designing stuff, not for understanding stuff; there is no One True Design the way there’s a One True Truth.