This is often true in the regular circumstances, but SI is clearly in a rush to avert the x-risk from UFAI, and the relevant math is apparently not yet available, so they have to develop it as they go along. I would compare it to theoretical physics, where available math is often a limiting factor in constructing better models.
This is actually a really interesting and potentially apt comparison. FAI may end up being something like String theory: a region in math space that has zero practical applications. (but given the published work in FAI to date, String Theorists may take offense at such a comparison)
Earlier I said:
If FAI is or can be made tractable, it will be a technological system: some combination of hardware and software, an actual practical invention.
SI’s conception of ‘FAI’ as math (whatever that means) is competing with the growing number of pragmatic mainstream approaches, most of which are loosely brain inspired. Humans have internal mechanisms for empathy and altruism which could be reverse engineered and magnified in machines.
But it all depends on what one means by “math”. If you count algorithms as new math, then the vast numbers of computer scientists and programmers, and most of the folks working on AGI designs, are thus mathematicians. If by “math”, you mean the stuff that academic mathematicians typically work on, then one is hard pressed to find any connection to AGI (friendly or not).
This is often true in the regular circumstances, but SI is clearly in a rush to avert the x-risk from UFAI, and the relevant math is apparently not yet available, so they have to develop it as they go along. I would compare it to theoretical physics, where available math is often a limiting factor in constructing better models.
This is actually a really interesting and potentially apt comparison. FAI may end up being something like String theory: a region in math space that has zero practical applications. (but given the published work in FAI to date, String Theorists may take offense at such a comparison)
Earlier I said:
SI’s conception of ‘FAI’ as math (whatever that means) is competing with the growing number of pragmatic mainstream approaches, most of which are loosely brain inspired. Humans have internal mechanisms for empathy and altruism which could be reverse engineered and magnified in machines.
But it all depends on what one means by “math”. If you count algorithms as new math, then the vast numbers of computer scientists and programmers, and most of the folks working on AGI designs, are thus mathematicians. If by “math”, you mean the stuff that academic mathematicians typically work on, then one is hard pressed to find any connection to AGI (friendly or not).