General intelligence is not some abstract thing approximated by heuristics, it is the cyclical heuristic generation and execution framework.
I largely agree—but what are the heuristics generated to do? If you can generate a practical “heuristic generation and execution framework” using bounded computing power, then you should be able to tell me how to do it using unbounded finite computing power: and I haven’t seen unbounded solutions that would reliably work. Finding an unbounded solution is a strictly easier problem, and if you can show me an unbounded solution, I’d feel a lot less confused.
(Serious question: if you did have unbounded computing power, and you did have access to strategies such as “search all possible heuristics and evaluate them against the real world,” how would you go about constructing an AGI?)
Asking for unbounded solutions does not seem to me like a hunt for El Dorado; rather, it seems more like asking that we understand the difference between pyrite and gold before attempting to build a city of pure gold using these gold-colored rocks from various places.
You say you agree, and then talk about something entirely unrelated. I suspect I failed in communicating my point. That’s okay, let’s try again.
If you want a general intelligence running on unbounded computational hardware, that’s what AIXI is, and approximations thereof. But I hope I am not making a controversial statement if I claim that the behavior of AIXI is qualitatively different than human-like general intelligence. There is absolutely nothing about human-like intelligence I can think of which approximates brute-force search over the space of universal Turing machines. If you want to study human-like intelligence, then you will learn nothing of value by looking at AIXI.
For the purpose of this discussion I’m going to make broad, generalizing assumptions about how human-like intelligence works. Please forgive me and don’t nit the purposefully oversimplified details. General intelligence arises primarily out of the neocortex, which is a network of hundreds of millions of cortical columns, which can be thought of as learned heuristics. Both the behavior of columns and their connections change over time via some learning process.
Architectures like CogPrime similarly consist of networks of learned behaviors / heuristic programs connected in a probability graph model. Both the heuristics and the connections between them can be learned over time, as can the underlying rules (a difference from the biological model which allows the AI to change its architecture over time).
In these models of human-like thinking, the generality of general intelligence is not represented in any particular snapshot of its state. Rather, it is the fact that its behavior drifts over time, in sometimes directed and sometime undirected ways. The drift is the source of generality. The heuristics are learned ways of guiding that drift in productive directions, but (1) it is the update process that gives generality, not the guiding heuristics; (2) it is an inherently unconstrained process; and (3) the abstract function being approximated by the heuristic network over time is constantly changing.
One way of looking at this is to say that a human-like general intelligence is not a static machine able to solve any problem, but rather a dynamic machine able to solve only certain problems at any given point in time, but is able to drift within problem solving space in response to its percepts. And the space of specialized problem solvers is sufficiently connected that the human-like intelligence is able to move from its current state to become any other specialized problem solver in reasonable time, a process we call learning.
One of the stated research objectives of MIRI is learning how to build a “reliable” / steadfast agent. I’ll go out on a limb and say it: the above description of human intelligence, if true, is evidence that a steadfast human-like general intelligence is a contradiction of terms. This is what I mean by making the comparison to El Dorado: you are looking for something of which there is no a priori evidence of its existence.
Maybe there are other architectures for general problem solving which look nothing like the neocortex or integrative AGI designs like CogPrime. But so far the evidence is lacking...
If you want a general intelligence running on unbounded computational hardware, that’s what AIXI is
I disagree. AIXI does not in fact solve the problem. It leaves many questions (of logical uncertainty, counterfactual reasoning, naturalized induction, etc.) unanswered, even in the unbounded case. (These points are touched upon in the technical agenda, and will be expanded upon in one of the forthcoming papers mentioned.) My failure to communicate this is probably why my previous comment looked like a non-sequitur; sorry about that. I am indeed claiming that we aren’t even far along enough to have an unbounded solution, and that I strongly expect that unbounded solutions will yield robust insights that help us build more reliable intelligent systems.
(The technical agenda covers questions that are not answered by AIXI, and these indicate places where we’re still confused about intelligence even in the unbounded case. I continue to expect that resolving these confusions will be necessary to create reliable AGI systems. I am under the impression that you believe that intelligence is not all that confusing, and that we instead simply need bigger collections of heuristics, better tools for learning heuristics, and better tools for selecting heuristics, but that this will largely arise from continued grinding on e.g. OpenCog. This seems like our core disagreement, to me—does that seem accurate to you?)
a human-like general intelligence is not a static machine able to solve any problem, but rather a dynamic machine able to solve only certain problems at any given point in time, but is able to drift within problem solving space in response to its percepts.
Yep, this seems quite likely in the bounded case. A generally intelligent reasoner would have to be able to figure out new ways to solve new problems, learn new heuristics, and so on. I agree.
I’ll go out on a limb and say it: the above description of human intelligence, if true, is evidence that a steadfast human-like general intelligence is a contradiction of terms.
This depends upon how you cash out the word “steadfast”, but I don’t think that the type of reliability we are looking for is a contradiction in terms. Can you think of another meaning of the word “reliability” that we are looking for, that allows me to simultaneously believe that generally intelligent systems are “dynamic machines [...] able to drift within problem solving space in response to its percepts” and that reliability doesn’t arise in generally intelligent systems by default? (Such an interpretation is more likely to be the thing I’m trying to communicate.)
This depends upon how you cash out the word “steadfast”, but I don’t think that the type of reliability we are looking for is a contradiction in terms.
I think I can see what Mark_Friedenbach is getting at here; I consider this sentence:
And the space of specialized problem solvers is sufficiently connected that the human-like intelligence is able to move from its current state to become any other specialized problem solver in reasonable time, a process we call learning.
And I note that “any other specialised problem solver” includes both Friendly and Unfriendly AIs; this implies that Mark’s definition of human-like includes the possibility that, at any point, the AI may learn to be Unfriendly. Which would be in direct contradiction to the idea of an AI which is steadfastly Friendly. (Interestingly, if I am parsing this correctly, it does not preclude the possibility of a Friendly non-human-like intelligence...)
This depends upon how you cash out the word “steadfast”, but I don’t think that the type of reliability we are looking for is a contradiction in terms. Can you think of another meaning of the word “reliability” that we are looking for, that allows me to simultaneously believe that generally intelligent systems are “dynamic machines [...] able to drift within problem solving space in response to its percepts” and that reliability doesn’t arise in generally intelligent systems by default?
No, I’ve been trying for a while and can’t. I think what I mean is what you are saying. Sorry, can you try another explanation? I use “steadfast goal” in the way that Goertzel defined the term:
If you literally can’t think of a meaning of the word “reliability” such that intelligent systems could be both dynamic problem-solvers (in the sense above) and “unreliable” by default, then I seriously doubt that I can communicate my point in the time available, sorry—I’m going to leave this conversation to others.
To reiterate where we are, an AI is described as steadfast by Goertzel “if, over a long period of time, it either continues to pursue the same goals it had at the start of the time period, or stops acting altogether.”[1] I took this to be a more technical specification of what you mean by “reliable”, you disagreed. I don’t see what other definition you could mean ….
Mark: So you think human-level intelligence by principle does not combine with goal stability. Aren’t you simply disagreeing with the orthogonality thesis, “that an artificial intelligence can have any combination of intelligence level and goal”?
So you think human-level intelligence by principle does not combine with goal stability.
To be clear I’ve been talking about human-like, which is a different distinction than human-level. Human-like intelligences operate similarly to human psychology. And it is demonstrably true that humans do not have a fixed set of fundamentally unchangeable goals, and human society even less so. For all its faults, the neoreactionaries get this part right in their critique of progressive society: the W-factor introduces a predictable drift in social values over time. And although people do tend to get “fixed in their ways”, it is rare indeed for a single person to remain absolutely rigidly so. So yes, in as far as we are talking about human-like intelligences, if they had fixed truly steadfast goals then that would be something which distinguishes them from humans.
Aren’t you simply disagreeing with the orthogonality thesis, “that an artificial intelligence can have any combination of intelligence level and goal”?
I don’t think the orthogonality thesis is well formed. The nature of an intelligence may indeed cause it to develop certain goals in due coarse, or for its overall goal set to drift in certain, expected if not predictable ways.
Of course denying the orthogonality thesis as stated does not mean endorsing a cosmist perspective either, which would be just as ludicrous. I’m not naive enough to think that there is some hidden universal morality that any smart intelligence naturally figures out—that’s bunk IMHO. But it’s just as naive to think that the structure of an intelligence and its goal drift over time are purely orthogonal issues. In real, implementable designs (e.g. not AIXI), one informs the other.
So you disagree with the premise of the orthogonality thesis. Then you know a central concept to probe to understand the arguments put forth here. For example, check out Stuart’s Armstrong’s paper: General purpose intelligence: arguing the Orthogonality thesis
I largely agree—but what are the heuristics generated to do? If you can generate a practical “heuristic generation and execution framework” using bounded computing power, then you should be able to tell me how to do it using unbounded finite computing power: and I haven’t seen unbounded solutions that would reliably work. Finding an unbounded solution is a strictly easier problem, and if you can show me an unbounded solution, I’d feel a lot less confused.
(Serious question: if you did have unbounded computing power, and you did have access to strategies such as “search all possible heuristics and evaluate them against the real world,” how would you go about constructing an AGI?)
Asking for unbounded solutions does not seem to me like a hunt for El Dorado; rather, it seems more like asking that we understand the difference between pyrite and gold before attempting to build a city of pure gold using these gold-colored rocks from various places.
You say you agree, and then talk about something entirely unrelated. I suspect I failed in communicating my point. That’s okay, let’s try again.
If you want a general intelligence running on unbounded computational hardware, that’s what AIXI is, and approximations thereof. But I hope I am not making a controversial statement if I claim that the behavior of AIXI is qualitatively different than human-like general intelligence. There is absolutely nothing about human-like intelligence I can think of which approximates brute-force search over the space of universal Turing machines. If you want to study human-like intelligence, then you will learn nothing of value by looking at AIXI.
For the purpose of this discussion I’m going to make broad, generalizing assumptions about how human-like intelligence works. Please forgive me and don’t nit the purposefully oversimplified details. General intelligence arises primarily out of the neocortex, which is a network of hundreds of millions of cortical columns, which can be thought of as learned heuristics. Both the behavior of columns and their connections change over time via some learning process.
Architectures like CogPrime similarly consist of networks of learned behaviors / heuristic programs connected in a probability graph model. Both the heuristics and the connections between them can be learned over time, as can the underlying rules (a difference from the biological model which allows the AI to change its architecture over time).
In these models of human-like thinking, the generality of general intelligence is not represented in any particular snapshot of its state. Rather, it is the fact that its behavior drifts over time, in sometimes directed and sometime undirected ways. The drift is the source of generality. The heuristics are learned ways of guiding that drift in productive directions, but (1) it is the update process that gives generality, not the guiding heuristics; (2) it is an inherently unconstrained process; and (3) the abstract function being approximated by the heuristic network over time is constantly changing.
One way of looking at this is to say that a human-like general intelligence is not a static machine able to solve any problem, but rather a dynamic machine able to solve only certain problems at any given point in time, but is able to drift within problem solving space in response to its percepts. And the space of specialized problem solvers is sufficiently connected that the human-like intelligence is able to move from its current state to become any other specialized problem solver in reasonable time, a process we call learning.
One of the stated research objectives of MIRI is learning how to build a “reliable” / steadfast agent. I’ll go out on a limb and say it: the above description of human intelligence, if true, is evidence that a steadfast human-like general intelligence is a contradiction of terms. This is what I mean by making the comparison to El Dorado: you are looking for something of which there is no a priori evidence of its existence.
Maybe there are other architectures for general problem solving which look nothing like the neocortex or integrative AGI designs like CogPrime. But so far the evidence is lacking...
I disagree. AIXI does not in fact solve the problem. It leaves many questions (of logical uncertainty, counterfactual reasoning, naturalized induction, etc.) unanswered, even in the unbounded case. (These points are touched upon in the technical agenda, and will be expanded upon in one of the forthcoming papers mentioned.) My failure to communicate this is probably why my previous comment looked like a non-sequitur; sorry about that. I am indeed claiming that we aren’t even far along enough to have an unbounded solution, and that I strongly expect that unbounded solutions will yield robust insights that help us build more reliable intelligent systems.
(The technical agenda covers questions that are not answered by AIXI, and these indicate places where we’re still confused about intelligence even in the unbounded case. I continue to expect that resolving these confusions will be necessary to create reliable AGI systems. I am under the impression that you believe that intelligence is not all that confusing, and that we instead simply need bigger collections of heuristics, better tools for learning heuristics, and better tools for selecting heuristics, but that this will largely arise from continued grinding on e.g. OpenCog. This seems like our core disagreement, to me—does that seem accurate to you?)
Yep, this seems quite likely in the bounded case. A generally intelligent reasoner would have to be able to figure out new ways to solve new problems, learn new heuristics, and so on. I agree.
This depends upon how you cash out the word “steadfast”, but I don’t think that the type of reliability we are looking for is a contradiction in terms. Can you think of another meaning of the word “reliability” that we are looking for, that allows me to simultaneously believe that generally intelligent systems are “dynamic machines [...] able to drift within problem solving space in response to its percepts” and that reliability doesn’t arise in generally intelligent systems by default? (Such an interpretation is more likely to be the thing I’m trying to communicate.)
I think I can see what Mark_Friedenbach is getting at here; I consider this sentence:
And I note that “any other specialised problem solver” includes both Friendly and Unfriendly AIs; this implies that Mark’s definition of human-like includes the possibility that, at any point, the AI may learn to be Unfriendly. Which would be in direct contradiction to the idea of an AI which is steadfastly Friendly. (Interestingly, if I am parsing this correctly, it does not preclude the possibility of a Friendly non-human-like intelligence...)
No, I’ve been trying for a while and can’t. I think what I mean is what you are saying. Sorry, can you try another explanation? I use “steadfast goal” in the way that Goertzel defined the term:
http://goertzel.org/GOLEM.pdf
If you literally can’t think of a meaning of the word “reliability” such that intelligent systems could be both dynamic problem-solvers (in the sense above) and “unreliable” by default, then I seriously doubt that I can communicate my point in the time available, sorry—I’m going to leave this conversation to others.
To reiterate where we are, an AI is described as steadfast by Goertzel “if, over a long period of time, it either continues to pursue the same goals it had at the start of the time period, or stops acting altogether.”[1] I took this to be a more technical specification of what you mean by “reliable”, you disagreed. I don’t see what other definition you could mean ….
[1] http://goertzel.org/GOLEM.pdf
Mark: So you think human-level intelligence by principle does not combine with goal stability. Aren’t you simply disagreeing with the orthogonality thesis, “that an artificial intelligence can have any combination of intelligence level and goal”?
To be clear I’ve been talking about human-like, which is a different distinction than human-level. Human-like intelligences operate similarly to human psychology. And it is demonstrably true that humans do not have a fixed set of fundamentally unchangeable goals, and human society even less so. For all its faults, the neoreactionaries get this part right in their critique of progressive society: the W-factor introduces a predictable drift in social values over time. And although people do tend to get “fixed in their ways”, it is rare indeed for a single person to remain absolutely rigidly so. So yes, in as far as we are talking about human-like intelligences, if they had fixed truly steadfast goals then that would be something which distinguishes them from humans.
I don’t think the orthogonality thesis is well formed. The nature of an intelligence may indeed cause it to develop certain goals in due coarse, or for its overall goal set to drift in certain, expected if not predictable ways.
Of course denying the orthogonality thesis as stated does not mean endorsing a cosmist perspective either, which would be just as ludicrous. I’m not naive enough to think that there is some hidden universal morality that any smart intelligence naturally figures out—that’s bunk IMHO. But it’s just as naive to think that the structure of an intelligence and its goal drift over time are purely orthogonal issues. In real, implementable designs (e.g. not AIXI), one informs the other.
So you disagree with the premise of the orthogonality thesis. Then you know a central concept to probe to understand the arguments put forth here. For example, check out Stuart’s Armstrong’s paper: General purpose intelligence: arguing the Orthogonality thesis
I explained in my post how the orthogonality thesis as argued by Stuart Armstrong et al presents a false choice. His argument is flawed.
I’m sorry I’m having trouble parsing what you are saying here...