Highly intelligent, and especially skilled in maths, probably at the IMO medal-winning level. (FAI team members will need to create lots of new math during the course of the FAI research initiative.)
Trustworthy. (Most FAI work is not “Friendliness theory” but instead AI architectures work that could be made more dangerous if released to a wider community that is less concerned with AI safety.)
If FAI is or can be made tractable, it will be a technological system: some combination of hardware and software, an actual practical invention. The parenthetical comment in your second point indicates you seem to acknowledge that FAI work mainly consists of safe AI architecture work.
If you look back on the mountain of historical evidence concerning invention, there is a rough general pattern or archetype for inventors. They may have more names or synonyms today: entrepreneur, hacker, programmer, engineer, etc, but the historical pattern remains.
The inventor mentality is characterized by relentless curiosity, creativity, dedication, knowledge, and intelligence. Formal education appears to be of little importance, and may in fact slightly negatively correlate with invention capability. Bill Gates dropped out of college, but Orville Wright dropped out of high school, and this trend appears across the technological landscape. Early mathematical ability may correlate with later inventorhood, but studying formal mathematics (being a mathematician) has a strong negative correlation with later invention. One immediate explanation is that any time spent studying formal mathematics is a waste of extremely precious higher cortical capacity which rivals are entirely devoting to pure technological study.
The Wright brothers didn’t need much or any mathematics to innovate in flight. They needed to understand flight, understand the landscape of prior art, and then quickly iterate and innovate within that space. Their most powerful tool was not mathematics or ‘rationality’, but rather the wind tunnel.
We can attract some of the people meeting these criteria by . ..
You don’t need accomplished mathematicians, if anything they would actually reduce your chances of success.
You need to attract the specific people who are going to or would develop AGI before you. They will almost certainly not be mathematicians (think Wright, Farnsworth, Edison, Tesla, Bill Gates, and not Terrance Tao). The historical evidence says they are unlikely to have any existing record of high status work.
How could you identify a future AGI-inventor? That should be the question, and the answer is clearly not “recruit math contest winners”.
One immediate explanation is that any time spent studying formal mathematics is a waste of extremely precious higher cortical capacity which rivals are entirely devoting to pure technological study.
Aren’t Turing and von Neumann (surely they invented “computers” as much as anyone) counterexamples to your thesis?
Turing published some conceptual math papers that would eventually get the field of computability and thus computer science started, but by no means did he invent the computer.
Computer evolution was already well under way when Turing published his paper on computability introducing Turing Machines in 1936.
The early British programmable digital computer, the Colossus, was developed by colleagues/contemporaries of Turing, but Turing was not involved, and at the time his abstract Turing Machine concept was not viewed as important:
It was thus not a fully general Turing-complete computer, even though Alan Turing worked at Bletchley Park. It was not then realized that Turing completeness was significant; most of the other pioneering modern computing machines were also not Turing complete (e.g. the Atanasoff–Berry Computer, the Bell Labs relay machines (by George Stibitz et al.), or the first designs of Konrad Zuse). The notion of a computer as a general purpose machine—that is, as more than a calculator devoted to solving difficult but specific problems—did not become prominent for several years.
Colossus was designed by the engineer Tommy Flowers.
The first Turing complete computer was the Z3, developed in germany by the engineer Konrad Zuse. Zuse is unlikely to have even heard of Turing, and the Z3 wasn’t proven Turing Complete until many decades later.
This architecture is to this day the basis of modern computer design, unlike the earliest computers that were ‘programmed’ by altering the electronic circuitry. Although the single-memory, stored program architecture is commonly called von Neumann architecture as a result of von Neumann’s paper, the architecture’s description was based on the work of J. Presper Eckert and John William Mauchly, inventors of the ENIAC at the University of Pennsylvania.[51]
Eckert was an electrical engineer, Mauchly a physicist.
Turing and von Neumman both made lasting contributions in the world of ideas, but they did not invent computers, not even close.
You don’t need accomplished mathematicians, if anything they would actually reduce your chances of success.
Imagine the world without something as basic as public-private key cryptography, which is pure math (or used to be pure before computer engineers hijacked it). Suppose there is at least one essential technology on the way to constructing AGI that requires advanced math skills, and your team is ill-equipped to recognize it. Result: you lose.
Mathematicians publish their work, it is freely available. It doesn’t need to be purchased and privately developed.
But if you don’t have mathematicians on your team, you might never realize the importance of the work that the other mathematicians publish, presuming that you even hear about it.
In the world I live in, results in one field that are actually important in other fields have a funny way of becoming known.
In the world I live in, inventors use and read about math, without the services of some personal conduit to the higher math gods.
Who determines whats important? The actual inventors, period.
The historical example shows that inventors don’t have this problem. Perhaps you believe otherwise, that invention has proceeded sub-optimally to date and would have been faster if only mathematicians and their ideas had more status. I don’t see evidence for this.
Actually I see evidence that our society tends to overrate the historical contributions of mathematicians to technical inventions.
Also, like I said in the other thread, it depends what one means by math.
LW-folk in particular (and perhaps lukeprog in extra particular), appear to have an especially strange mathematician fetish.
This is often true in the regular circumstances, but SI is clearly in a rush to avert the x-risk from UFAI, and the relevant math is apparently not yet available, so they have to develop it as they go along. I would compare it to theoretical physics, where available math is often a limiting factor in constructing better models.
This is actually a really interesting and potentially apt comparison. FAI may end up being something like String theory: a region in math space that has zero practical applications. (but given the published work in FAI to date, String Theorists may take offense at such a comparison)
Earlier I said:
If FAI is or can be made tractable, it will be a technological system: some combination of hardware and software, an actual practical invention.
SI’s conception of ‘FAI’ as math (whatever that means) is competing with the growing number of pragmatic mainstream approaches, most of which are loosely brain inspired. Humans have internal mechanisms for empathy and altruism which could be reverse engineered and magnified in machines.
But it all depends on what one means by “math”. If you count algorithms as new math, then the vast numbers of computer scientists and programmers, and most of the folks working on AGI designs, are thus mathematicians. If by “math”, you mean the stuff that academic mathematicians typically work on, then one is hard pressed to find any connection to AGI (friendly or not).
The other part of the pattern is that the competent inventive ones are the ones doing recruiting, not other way around, especially as vast majority of inventions are not someone’s first inventions, and inventions tend to make money.
If FAI is or can be made tractable, it will be a technological system: some combination of hardware and software, an actual practical invention. The parenthetical comment in your second point indicates you seem to acknowledge that FAI work mainly consists of safe AI architecture work.
If you look back on the mountain of historical evidence concerning invention, there is a rough general pattern or archetype for inventors. They may have more names or synonyms today: entrepreneur, hacker, programmer, engineer, etc, but the historical pattern remains.
The inventor mentality is characterized by relentless curiosity, creativity, dedication, knowledge, and intelligence. Formal education appears to be of little importance, and may in fact slightly negatively correlate with invention capability. Bill Gates dropped out of college, but Orville Wright dropped out of high school, and this trend appears across the technological landscape. Early mathematical ability may correlate with later inventorhood, but studying formal mathematics (being a mathematician) has a strong negative correlation with later invention. One immediate explanation is that any time spent studying formal mathematics is a waste of extremely precious higher cortical capacity which rivals are entirely devoting to pure technological study.
The Wright brothers didn’t need much or any mathematics to innovate in flight. They needed to understand flight, understand the landscape of prior art, and then quickly iterate and innovate within that space. Their most powerful tool was not mathematics or ‘rationality’, but rather the wind tunnel.
You don’t need accomplished mathematicians, if anything they would actually reduce your chances of success.
You need to attract the specific people who are going to or would develop AGI before you. They will almost certainly not be mathematicians (think Wright, Farnsworth, Edison, Tesla, Bill Gates, and not Terrance Tao). The historical evidence says they are unlikely to have any existing record of high status work.
How could you identify a future AGI-inventor? That should be the question, and the answer is clearly not “recruit math contest winners”.
Aren’t Turing and von Neumann (surely they invented “computers” as much as anyone) counterexamples to your thesis?
No, not if you actually read into the history.
Turing published some conceptual math papers that would eventually get the field of computability and thus computer science started, but by no means did he invent the computer.
Computer evolution was already well under way when Turing published his paper on computability introducing Turing Machines in 1936.
The early British programmable digital computer, the Colossus, was developed by colleagues/contemporaries of Turing, but Turing was not involved, and at the time his abstract Turing Machine concept was not viewed as important:
Colossus was designed by the engineer Tommy Flowers.
The first Turing complete computer was the Z3, developed in germany by the engineer Konrad Zuse. Zuse is unlikely to have even heard of Turing, and the Z3 wasn’t proven Turing Complete until many decades later.
Concerning Von Neumman’s architecture:
Eckert was an electrical engineer, Mauchly a physicist.
Turing and von Neumman both made lasting contributions in the world of ideas, but they did not invent computers, not even close.
Imagine the world without something as basic as public-private key cryptography, which is pure math (or used to be pure before computer engineers hijacked it). Suppose there is at least one essential technology on the way to constructing AGI that requires advanced math skills, and your team is ill-equipped to recognize it. Result: you lose.
Hardly.
Mathematicians publish their work, it is freely available. It doesn’t need to be purchased and privately developed.
Engineering builds on conceptual advances and mathematical tools, but typically said tools are developed long before the engineering work begins.
But if you don’t have mathematicians on your team, you might never realize the importance of the work that the other mathematicians publish, presuming that you even hear about it.
In the world I live in, results in one field that are actually important in other fields have a funny way of becoming known.
In the world I live in, inventors use and read about math, without the services of some personal conduit to the higher math gods.
Who determines whats important? The actual inventors, period.
The historical example shows that inventors don’t have this problem. Perhaps you believe otherwise, that invention has proceeded sub-optimally to date and would have been faster if only mathematicians and their ideas had more status. I don’t see evidence for this.
Actually I see evidence that our society tends to overrate the historical contributions of mathematicians to technical inventions.
Also, like I said in the other thread, it depends what one means by math.
LW-folk in particular (and perhaps lukeprog in extra particular), appear to have an especially strange mathematician fetish.
This is often true in the regular circumstances, but SI is clearly in a rush to avert the x-risk from UFAI, and the relevant math is apparently not yet available, so they have to develop it as they go along. I would compare it to theoretical physics, where available math is often a limiting factor in constructing better models.
This is actually a really interesting and potentially apt comparison. FAI may end up being something like String theory: a region in math space that has zero practical applications. (but given the published work in FAI to date, String Theorists may take offense at such a comparison)
Earlier I said:
SI’s conception of ‘FAI’ as math (whatever that means) is competing with the growing number of pragmatic mainstream approaches, most of which are loosely brain inspired. Humans have internal mechanisms for empathy and altruism which could be reverse engineered and magnified in machines.
But it all depends on what one means by “math”. If you count algorithms as new math, then the vast numbers of computer scientists and programmers, and most of the folks working on AGI designs, are thus mathematicians. If by “math”, you mean the stuff that academic mathematicians typically work on, then one is hard pressed to find any connection to AGI (friendly or not).
The other part of the pattern is that the competent inventive ones are the ones doing recruiting, not other way around, especially as vast majority of inventions are not someone’s first inventions, and inventions tend to make money.