If someone made what they thought was FAI and it wasn’t actually FAI, how could you tell until it was too late?
How can you tell that a theorem is correct? By having a convincing proof, or perhaps other arguments to the effect of the theorem being correct, and being trained to accept correct proofs (arguments) and not incorrect ones.
Even though we currently have no idea about how to construct a FAI or how an argument for a particular design being a FAI might look like, we can still refuse to believe that something is a FAI when we are not given sufficient grounds for believing it is one, provided the people who make such decisions are skilled enough to accept correct arguments but not incorrect ones.
The rule to follow is to not actually build or run things that you don’t know to be good. By default, they don’t even work, and if they do, they are not good, because value is complex and won’t be captured by chance. There need to be strong grounds for believing that a particular design will work, and here rationality training is essential, because by default people follow one of the many kinds of crazy reasoning about these things.
How can you tell that a theorem is correct? By having a convincing proof, or perhaps other arguments to the effect of the theorem being correct, and being trained to accept correct proofs (arguments) and not incorrect ones.
Even though we currently have no idea about how to construct a FAI or how an argument for a particular design being a FAI might look like, we can still refuse to believe that something is a FAI when we are not given sufficient grounds for believing it is one, provided the people who make such decisions are skilled enough to accept correct arguments but not incorrect ones.
The rule to follow is to not actually build or run things that you don’t know to be good. By default, they don’t even work, and if they do, they are not good, because value is complex and won’t be captured by chance. There need to be strong grounds for believing that a particular design will work, and here rationality training is essential, because by default people follow one of the many kinds of crazy reasoning about these things.