I take your point that theorists can appear to be concerned with problems that have very little impact. On the other hand, there are some great theoretical results and concepts that can prevent us futility wasting our time and guide us to areas where success is more likely.
I think you’re being ungenerous to Bolstrom. His paper on the possibility of Oracle type AIs is quite nuanced, and discusses many difficulties that would have to be overcome …
why would an AI become evil?
The worry isn’t that the AI would suddenly become evil by some human standard, rather that the AI’s goal system would be insufficiently considerate of human values. When humans build a skyscraper, they aren’t deliberately being “evil” towards the ants that lived in the earth that was excavated and had concrete poured over it, the humans just don’t value the communities and structures that the ants had established.
I think your criticism is a little harsh. Turing machines are impossible to implement as well, but they are still a useful theoretical concept.
There was quite a bit of commentary on the Jan 27 post …
tl;dr: reactions are mixed.
My personal reaction is that it is surprising that neural networks, even large ones fed with clever inputs and used in clever ways, could be used to boost Go play to this level. Although it has long been known that neural networks are universal function approximators, this achievement is a “no, really.”
Yes the AI would know what we would approve of.
Okay, to simplify, suppose the AI has a function …
Boolean humankind_approves(Outcome o)
… that returns 1 when humankind would approve of a particular outcome o, and zero otherwise.
At any given point, the AI needs to have a well specified utility function.
Okay, to simplify, suppose the AI has a function …
Outcome U(Input i)
… which returns the outcome(s) (e.g., answer, plan) that optimizes expected utility given the input i.
But it doesn’t have any reason to care.
Assuming the AI is corrigible (I think we all agree that if the AI is not corrigible, it shouldn’t be turned on), we modify its utility function to U’ where
U’(i) = U(i) when humankind_approves(U(i)) or null if there does not exist a U(i) such that humankind_approves(U(i))
I suggest that an AI with utility function U’ is a friendly AI.
It could look at the existing research
I think extrapolation from existing research is an interesting area of study, but I was attempting to evoke the surprise of a breakthrough invention. To me, the most interesting inventions are exactly those inventions that are not mundane extrapolations of existing techniques.
Emulating human brains is a rather convoluted solution to any problem.
Granted. In practice, it may be possible to represent aspects of humankind in a more compact form. But the point is that if …
The AI would be very familiar with humans and would have a good idea of our [inventive] abilities.
… then to me it seems likely that “the AI would be very familiar with humans and would have a good idea of actions that would meet human approval.”
Taking your analogy … if we can model chimp inventiveness to a useful degree, wouldn’t we also be able to model which human actions would earn chimp approval and disapproval? Couldn’t we build a chimp-friendly AI?
Consider a different scenario: a year ago, we asked the first AI to generate a Go playing program that could beat a professional Go player. The first AI submits AlphaGo as its solution after 1 day of processing. How does the second AI determine that AlphaGo is within or outside of human inventiveness at that time?
It’s easy to detect what solutions a human couldn’t have invented. That’s what the second AI does
I think, to make this detection, the second AI would have to maintain high resolution simulations of the world’s smartest people (if not the entire population), and basically ask the simulations to collaboratively come up with their best solutions to the problem.
Supposing that is the case, the second AI can be configured to maintain high resolution simulations of the entire population, and basically ask the simulations whether they collectively approve of a particular action.
Is there a way to “detect what solutions a human couldn’t have invented” that doesn’t involve emulating humankind?
There is regular structure in human values that can be learned without requiring detailed knowledge of physics, anatomy, or AI programming.
While there is some regular structure to human values, I don’t think you can say that the totality of human values has a completely regular structure. There are too many cases of nameless longings and generalized anxieties. Much of art is dedicated exactly to teasing out these feelings and experiences, often in counterintuitive contexts.
Can they be learned without detailed knowledge of X, Y and Z? I suppose it depends on what “detailed” means—I’ll assume it means “less detailed than the required knowledge of the structure of human values.” That said, the excluded set of knowledge you chose—“physics, anatomy, or AI programming”—seems really odd to me. I suppose you can poll people about their values (or use more sophisticated methods like prediction markets), but I don’t see how this can yield more than “the set of human values that humans can articulate.” It’s something, but this seems to be a small subset of the set of human values. To characterize all dimensions of human values, I do imagine that you’ll need to model human neural biophysics in detail. If successful, it will be a contribution to AI theory and practice.
Human values are so fragile that it would require a superintelligence to capture them with anything close to adequate fidelity.
To me, in this context, the term “fragile” means exactly that it is important to characterize and consider all dimensions of human values, as well as the potentially highly nonlinear relationships between those dimensions. An at-the-time invisible “blow” to at-the-time unarticulated dimension can result in unfathomable suffering 1000 years hence. Can a human intelligence capture the totality of human values? Some of our artists seem to have glimpses of the whole, but it seems unlikely to me that a baseline human can appreciate the whole clearly.
I mean, the ability to estimate the abilities of superintelligences appears to be an aspect of reliable Vingean reflection.
Although we use limited proxies (e.g., IQ test questions) to estimate human intelligence.
The opportunities for detecting superintelligence would definitely be rarer if the superintelligence is actively trying to conceal the status.
What about in the case where there is no attempted concealment? Or even weaker, where the AI voluntary submits to arbitrary tests. What tests would we use?
Presumably we would have a successful model of human intelligence by that point. It’s interesting to think about what dimensions of intelligence to measure. Number of variables simultaneously optimized? Optimization speed? Ability to apply nonlinear relationships? Search speed in a high dimensional, nonlinear solution space? I guess it is more the ability to generate appropriate search spaces in the first place. Something much simpler?
Whatever mechanism that you use to require the AI to discard “solutions that a human couldn’t invent”, use that same mechanism to require the AI to discard “actions of which humankind would not approve.”
I believe that the formal terminology is to add the condition to the AI’s utility function.
I wonder if this is true in general. Have you read a good discussion on detecting superintelligence?
I think that if you are able to emulate humankind to the extent that you can determine things like “solutions that a human couldn’t invent” and “what a human given a year to work on it, would produce,” then you have already solved FAI, because instead you can require the AI to “only take actions of which humankind would approve.”
To use AI to build FAI, don’t we need a way to avoid this Catch 22?
it doesn’t optimize without end to create the best solution possible, it just has to meet some minimum threshold, then stop.
It’s easy to ask hard questions. I think it can be argued that emulating a human is a hard problem. There doesn’t seem to be a guarantee that the “minimum threshold” doesn’t involve converting planetary volumes to computronium.
I think the same problem is present in trying to specify minimum required computing power for a task prior to prior to performing the task. It isn’t obvious to me that calculating “minimum required computing power for X” is any less difficult than performing some general task X.
provided the field is important within the context of human societal development and in engaging the material I gain a nuanced understanding of the content and a deep appreciation of how the originators created the system.
I’ll suggest investigating the problem of “squaring the circle.” It has it’s roots in the origins of mathematics, passes through geometric proofs (including the notions of formal proofs and proof from elementary axioms), was unsolved for 2000 years in the face of myriad attempts, and was proved impossible to solve using the relatively modern techniques of abstract algebra.
The linked site has references (some already mentioned in this thread) that may be helpful …
R.Courant and H.Robbins, What is Mathematics?, Oxford University Press, 1996
H.Dorrie, 100 Great Problems Of Elementary Mathematics, Dover Publications, NY, 1965.
W.Dunham, Journey through Genius, Penguin Books, 1991
M.Kac and S.M.Ulam, Mathematics and Logic, Dover Publications, NY, 1968.
including …
R.B.Nelsen, Proofs Without Words, MAA, 1993
which may be of special interest to you.
Yudkowsky seems to think it is significant …
How did Belmopan, Brasília, Abuja and Islamabad do it?
Well all of these are deliberate decisions to build a national capital. They overcame the bootstrap problem by being funded by a pre-existing national tax base.
dozens of new cities built just in Singapore during the past half century
Again, government funding is used to overcome the bootstrap problem. Singapore is also geographically small, and many of these “cities” would be characterized as neighborhoods if they were in the US.
Las Vegas
Well, wikipedia says it began life as a water resupply stop for steam trains, and then got lucky by being near a major government project—Hoover dam. Later it took advantage of regulatory differences. An eccentric billionaire seems to have played a key roll.
There seem to be several towns that exist because of regulatory differences, so this seems a factor to consider—at least one eccentric billionaire seems fairly serious about “seasteading” for this reason. Historically, religious and ideological differences have founded cites, if not nations, so this is one way to push through the bootstrap phase—Salt Lake City being a relatively modern example in the US. Masdar City—zero carbon, zero waste—is an interesting example—ironically funded by oil wealth.
Gentrification simply means that rents go up in certain parts of the city. It doesn’t have directly something to do with new investments.
In my experience gentrification is always associated with renovation and new business investment. The wikipedia article seems to confirm that this is not an uncommon experience.
I feel like there should be some constraint on harming group happiness while you “do what makes you happy.”