Note that having type 1 processes is not an error. AIs will also need to have type 1 processes. Being perfectly rational is never optimal if time and resources are an issue. Being perfectly rational would make sense only given infinite time and resources and no competition.
I heard you bring this up in person a few times last weekend too. I wanted to follow up with you because I think I’m starting to understand the reason for your disagreements with others on this matter. It is not the case that you’re fighting the good fight and we are all just off in crazy land on this issue. I think instead, you’re conflating “rationality” with “deliberation”.
Naive “perfect rationality”—being infinitely deliberate—is of course a mistake. But that’s not what Yudkowsky, Omohundro and other careful thinkers are advocating when they discuss wanting to build an AI that is rational. They mean things like having a deliberate rational process that won’t knowably violate the axioms of choice or other basic sanity checks that our kludgey, broken minds don’t even try to do on their own.
Also, it’s possible to design algorithms that are ideally rational within time and resource constraints (ie, AIXItl, Godel Machines). There isn’t a false dichotomy between “quick and dirty” heuristic kludges and infinite rationality. You can use “quick and clean” methods which converge towards rationality as they compute, rather than “type 1 processes” which are unrelated to rational deliberation.
Basically, I completely disagree with your first two sentences and would agree with your second two if you replaced “perfectly rational” with words closer to what you actually mean, like “infinitely deliberate”.
Note that having type 1 processes is not an error. AIs will also need to have type 1 processes. Being perfectly rational is never optimal if time and resources are an issue. Being perfectly rational would make sense only given infinite time and resources and no competition.
I heard you bring this up in person a few times last weekend too. I wanted to follow up with you because I think I’m starting to understand the reason for your disagreements with others on this matter. It is not the case that you’re fighting the good fight and we are all just off in crazy land on this issue. I think instead, you’re conflating “rationality” with “deliberation”.
Naive “perfect rationality”—being infinitely deliberate—is of course a mistake. But that’s not what Yudkowsky, Omohundro and other careful thinkers are advocating when they discuss wanting to build an AI that is rational. They mean things like having a deliberate rational process that won’t knowably violate the axioms of choice or other basic sanity checks that our kludgey, broken minds don’t even try to do on their own.
Also, it’s possible to design algorithms that are ideally rational within time and resource constraints (ie, AIXItl, Godel Machines). There isn’t a false dichotomy between “quick and dirty” heuristic kludges and infinite rationality. You can use “quick and clean” methods which converge towards rationality as they compute, rather than “type 1 processes” which are unrelated to rational deliberation.
Basically, I completely disagree with your first two sentences and would agree with your second two if you replaced “perfectly rational” with words closer to what you actually mean, like “infinitely deliberate”.