The people who finally find out how the planets move will be spiritual descendants of the group A. … The problem with the group B is that it has no energy to move forward.
In this particular example, it’s true that group A was more correct. This is because planetary physics can be formalised relatively easily, and also because it’s a field where you can only observe and not experiment. But imagine the same conversation between sociologists who are trying to find out what makes people happy, or between venture capitalists trying to find out what makes startups succeed. In those cases, Group B can move forward using the sort of “energy” that biologists and inventors and entrepreneurs have, driven an experimental and empirical mindset. Whereas Group A might spend a long time writing increasingly elegant equations which rely on unjustified simplifications.
Instinctively reasoning about intelligence using analogies from physics instead of the other domains I mentioned above is a very good example of rationality realism.
Uncontrolled argues along similar lines—that the physics/chemistry model of science, where we get to generalize a compact universal theory from a number of small experiments, is simply not applicable to biology/psychology/sociology/economics and that policy-makers should instead rely more on widespread, continuous experiments in real environments to generate many localized partial theories.
A prototypical argument is the paradox-of-choice jam experiment, which has since become solidified in pop psychology. But actual supermarkets run many 1000s of in-situ experiments and find that it actually depends on the product, the nature of the choices, the location of the supermarket, the time of year etc.
Uncontrolled argues along similar lines—that the physics/chemistry model of science, where we get to generalize a compact universal theory from a number of small experiments, is simply not applicable to biology/psychology/sociology/economics and that policy-makers should instead rely more on widespread, continuous experiments in real environments to generate many localized partial theories.
I’ll note that (non-extreme) versions of this position are consistent with ideas like “it’s possible to build non-opaque AGI systems.” The full answer to “how do birds work?” is incredibly complex, hard to formalize, and dependent on surprisingly detailed local conditions that need to be discovered empirically. But you don’t need to understand much of that complexity at all to build flying machines with superavian speed or carrying capacity, or to come up with useful theory and metrics for evaluating “goodness of flying” for various practical purposes; and the resultant machines can be a lot simpler and more reliable than a bird, rather than being “different from birds but equally opaque in their own alien way”.
This isn’t meant to be a response to the entire “rationality non-realism” suite of ideas, or a strong argument that AGI developers can steer toward less opaque systems than AlphaZero; it’s just me noting a particular distinction that I particularly care about.
The relevant realism-v.-antirealism disagreement won’t be about “can machines serve particular functions more transparently than biological organs that happen to serve a similar function (alongside many other functions)?”. In terms of the airplane analogy, I expect disagreements like “how much can marginal effort today increase transparency once we learn how to build airplanes?”, “how much useful understanding are we currently missing about how airplanes work?”, and “how much of that understanding will we develop by default on the path toward building airplanes?”.
“This is because planetary physics can be formalized relatively easily” - they can now, and could when they were, but not before. One can argue that we thought many “complex” and very “human” abilities could not be algroithmically emulated in the past, and recent advances in AI (with neural nets and all that) have proven otherwise. If a program can do/predict something, there is a set of mechanical rules that explain it. The set might not be as elegant as Newton’s laws of motion, but it is still a set of equations nonetheless. The idea behind Villam’s comment (I think) is that in the future someone might say, the same way you just did, that “We can formalize how happy people generally are in a given society because that’s relatively easy, but what about something truly complex like what an individual might imagine if we read him a specific story?”.
In other words, I don’t see the essential differentiation between biology and sociology questions and physics questions, that you try to point to. In the post itself you also talk about moral preference, and I tend to agree with you that some people just have very individually strongly valued axioms that might contradict themselves or others, but it doesn’t in itself mean that questions about rationality differ from questions about, say, molecular biology, in the sense that they can be hypothetically answered to a satisfactory level of accuracy.
Group A was most successful in the field of computation, so I have high confidence that their approach would be successful in intelligence as well (especially in intelligence of artificial agents).
In this particular example, it’s true that group A was more correct. This is because planetary physics can be formalised relatively easily, and also because it’s a field where you can only observe and not experiment. But imagine the same conversation between sociologists who are trying to find out what makes people happy, or between venture capitalists trying to find out what makes startups succeed. In those cases, Group B can move forward using the sort of “energy” that biologists and inventors and entrepreneurs have, driven an experimental and empirical mindset. Whereas Group A might spend a long time writing increasingly elegant equations which rely on unjustified simplifications.
Instinctively reasoning about intelligence using analogies from physics instead of the other domains I mentioned above is a very good example of rationality realism.
Uncontrolled argues along similar lines—that the physics/chemistry model of science, where we get to generalize a compact universal theory from a number of small experiments, is simply not applicable to biology/psychology/sociology/economics and that policy-makers should instead rely more on widespread, continuous experiments in real environments to generate many localized partial theories.
A prototypical argument is the paradox-of-choice jam experiment, which has since become solidified in pop psychology. But actual supermarkets run many 1000s of in-situ experiments and find that it actually depends on the product, the nature of the choices, the location of the supermarket, the time of year etc.
I’ll note that (non-extreme) versions of this position are consistent with ideas like “it’s possible to build non-opaque AGI systems.” The full answer to “how do birds work?” is incredibly complex, hard to formalize, and dependent on surprisingly detailed local conditions that need to be discovered empirically. But you don’t need to understand much of that complexity at all to build flying machines with superavian speed or carrying capacity, or to come up with useful theory and metrics for evaluating “goodness of flying” for various practical purposes; and the resultant machines can be a lot simpler and more reliable than a bird, rather than being “different from birds but equally opaque in their own alien way”.
This isn’t meant to be a response to the entire “rationality non-realism” suite of ideas, or a strong argument that AGI developers can steer toward less opaque systems than AlphaZero; it’s just me noting a particular distinction that I particularly care about.
The relevant realism-v.-antirealism disagreement won’t be about “can machines serve particular functions more transparently than biological organs that happen to serve a similar function (alongside many other functions)?”. In terms of the airplane analogy, I expect disagreements like “how much can marginal effort today increase transparency once we learn how to build airplanes?”, “how much useful understanding are we currently missing about how airplanes work?”, and “how much of that understanding will we develop by default on the path toward building airplanes?”.
“This is because planetary physics can be formalized relatively easily” - they can now, and could when they were, but not before. One can argue that we thought many “complex” and very “human” abilities could not be algroithmically emulated in the past, and recent advances in AI (with neural nets and all that) have proven otherwise. If a program can do/predict something, there is a set of mechanical rules that explain it. The set might not be as elegant as Newton’s laws of motion, but it is still a set of equations nonetheless. The idea behind Villam’s comment (I think) is that in the future someone might say, the same way you just did, that “We can formalize how happy people generally are in a given society because that’s relatively easy, but what about something truly complex like what an individual might imagine if we read him a specific story?”.
In other words, I don’t see the essential differentiation between biology and sociology questions and physics questions, that you try to point to. In the post itself you also talk about moral preference, and I tend to agree with you that some people just have very individually strongly valued axioms that might contradict themselves or others, but it doesn’t in itself mean that questions about rationality differ from questions about, say, molecular biology, in the sense that they can be hypothetically answered to a satisfactory level of accuracy.
Group A was most successful in the field of computation, so I have high confidence that their approach would be successful in intelligence as well (especially in intelligence of artificial agents).