The problem is that the limiting case of pure memorization is a look up table of some form. If we drop the requirement for pure memorization and instead allow world models or heuristics, then we don’t have to use a look up table.
More generally, this is important, since if AI could only memorize like the Chinese Room, GPT-3 would not be at all possible without expending vastly more energy than was actually done, so we should strongly update against the pure memorization hypothesis.
The other complaint I think Searle is making is that black box intelligence/consciousness, whether a look up table or something else, is not real intelligence/consciousness. And that’s wrong, since there are in fact black box intelligences we can’t interpret well like Evolution Strategies or Model Free RL, so the fact that a Turing machine emulates Chinese in a black box format is equivalent to learning it in a white box format.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
If some task is exactly breathing then it is plausible that starting from nothing can work if the task involves breathing. If it can’t be done then “taskness” is not sufficient.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
It’s already covered by conservation of energy, IE new feelings must pay an energy cost to feel something new, so this is double-counting.
And conservation of energy does not always hold in mathematics (though such a cosmology would allow time travel.) Thus this is a contingent fact of our universe, not something that has to be universally true.
Paying an energy cost would be a task and would operate on the intelligence side of it. Typically feelings come with cognition discernable aspects.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy conversation conservation that color conversion should be impossible.
If you had some part of the flipper that red the color of the ball and conditional on that kicked the ball then one could tell the color of the ball from its trajectory. If no such things exists then pointfulness of even talking about color might drop and somebody might think that color is “epiphenomena” that is not an actual thing that happens. A claim that the machine recolors the ball is not a claim about trajectories.
I made a bad analog in using breathing as that can sound like a trajectory term. Somebody that believes in P-zombies might ask whether a particular actor is a P-zombie. You start something that is definetely not functional and definetely is a zombie. Then you put it through and upgrade/process that makes it functional. You would expect to have produced a P-zombie. But if you actually receive a human non-p-zombie you might start suspecting that the process somehow also works in the zombie dimension. P-zombie compared to usual human is intelligence similar and conciousness dissimilar. Conflating and taking them to be synonymous with each other makes talking about it hard and/or impossible.
Edit: typo that has chances to be quite meaningful. I did not mean “turn energy to color” but rather that “energy stays stable, color is unstable”.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy conversation that color conversion should be impossible.
The key to solve this problem is that extra energy was applied somewhere else, like a color machine, and this is an open system, so conservation of energy does not hold here.
If it is open, how do we know how much energy there is supposed to be, to determine that there is extra?
If we have built the machine and therefore are quite sure that there are no surprise coloring machines then the various part placements can not be the detail where we screwed up.
Why would applying a surprise jolt of energy on the ball change its color? Imagine that you find a particular bumber before which the ball is red and after which it is blue. Why would calling this bumber a “coloring machine” explain anything? Would the blue ball leaving with a tiny bit of speed deficiency explain why the ball got colored or why it is this specific bumber rather than all the others which have come from the same assembly line?
I was of course talking about a coloring machine, though one important point is with enough energy directed in the right way, you can do things like simulate a coloring machine and then add new color.
Energy, properly applied and enough energy, can do nearly everything, like changing the color.
I will give that bumber became a superpowered arcane user that can with miniscule energy make unknown effects.
It would still be interesting that this ended up happening starting by taking out a factory-standard bumber part and setting out to build a non-magical pinball machine. That is you are not trying to make a coloring machine happen. You do not have any reason to believe any unknown high-tech spy would be interested to be gaslighting you. As long as you can be super sure that you saw it blue and super sure you saw it red and that you were not trying to make it happen, you have reason to believe you do not understand what you ended up doing.
Maybe you try to look a bit more into it and try a geiger counter on the ball. Before machine it doesn’t radiate and after machine it radiates. You could feel good and go “I invented an irradiator!” or you could try to hunt down an irradiator bumber in the pinball machine. But you still do not know how you did it.
There could be any number of properties that you could learn to check your balls for. The claim is not that you need to learn all these properties to master predicting the kinetics of the pinball. The claim is not that the new properties would be a source of infinite or absurd amount of kinetic energy. The kinetics works as predicted and is closed in regards to all these other things. Learning about new properties does not change the fact that you saw the ball previously bounce around the machine. The claim is that your mastery of kinetics can not explain the machine turning out to be a converter for property number 56.
Maybe you think that kinetics can not be closed in regards to other properties. “Kinetics is everything”. Then when you try to blueprint the pinball machine in meticilous detail you should be able to predict all the other properties. Then showing what kind of ball went in and what kind of ball came out, you should be 100% be able to determine whether it was put in your unmeddled machine or in some other machine.
But meticulousness is hard and in your non-omnisciene you can only do that to the practical limit. So you learn about new property number 12. Unassembled you test each bumber separately what kind of effect it has on the ball. Your friends ask that since you always master the latest discovered property first whether you have done it for number 12 yet. You feel confident. Then you assemble the pinball machine. And put a ball through the machine. If you are surprised whether it is a property 12 converter then you have knowledge that your mastery is not at a level of what is possible to construct and measure practically.
So claims of the form “you need to keep track of property number 9 in order to be able to practically predict what happens about practically doable measurements of property 10” do not accept unknown unknowns as an excuse.
Claims of the form “you can mixup properties 8 and 7 and it will not have a practical or observable difference” combined with not being able to be close kinetics away from them means that existence of of such properties is a settleable question.
If intelligence and conciousness are separate properties and we can contain an argument to be about intelligence only, it cannot inform us about conciousness.
If intelligence and conciousness are connected properties we can not claim that an argument about one of them is irrelevant in regards to the other.
The problem is that there are a bunch of intermediate positions between LUT and strong AI. The doubt about an LLM is whether it has a world model, when it is only designed as a word model.
That claim is unjustified and unjustifiable. Everything is fundamentally a black box until proven otherwise. And we will never find any conclusive proof. (I want to tell you to look up Hume’s problem of induction and Karl Popper’s solution, although I feel that making such a remark would be insulting your intelligence.) Our ability to imagine systems behaving in ways that are 100% predictable and our ability to test systems so as to ensure that they behave predictably does not change the fact that everything is always fundamentally a black box.
This might a key crux: I think that white box is a rare state, and interpretability is not the default case here.
Indeed I see the opposite: Black box strategies like Model Free RL work a lot better currently than white box strategies, and I think black boxes are the default scenario. You have to do a lot of work to interpret things enough to make it white box.
The problem is that the limiting case of pure memorization is a look up table of some form. If we drop the requirement for pure memorization and instead allow world models or heuristics, then we don’t have to use a look up table.
More generally, this is important, since if AI could only memorize like the Chinese Room, GPT-3 would not be at all possible without expending vastly more energy than was actually done, so we should strongly update against the pure memorization hypothesis.
The other complaint I think Searle is making is that black box intelligence/consciousness, whether a look up table or something else, is not real intelligence/consciousness. And that’s wrong, since there are in fact black box intelligences we can’t interpret well like Evolution Strategies or Model Free RL, so the fact that a Turing machine emulates Chinese in a black box format is equivalent to learning it in a white box format.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
If some task is exactly breathing then it is plausible that starting from nothing can work if the task involves breathing. If it can’t be done then “taskness” is not sufficient.
It’s already covered by conservation of energy, IE new feelings must pay an energy cost to feel something new, so this is double-counting.
And conservation of energy does not always hold in mathematics (though such a cosmology would allow time travel.) Thus this is a contingent fact of our universe, not something that has to be universally true.
Paying an energy cost would be a task and would operate on the intelligence side of it. Typically feelings come with cognition discernable aspects.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy
conversationconservation that color conversion should be impossible.If you had some part of the flipper that red the color of the ball and conditional on that kicked the ball then one could tell the color of the ball from its trajectory. If no such things exists then pointfulness of even talking about color might drop and somebody might think that color is “epiphenomena” that is not an actual thing that happens. A claim that the machine recolors the ball is not a claim about trajectories.
I made a bad analog in using breathing as that can sound like a trajectory term. Somebody that believes in P-zombies might ask whether a particular actor is a P-zombie. You start something that is definetely not functional and definetely is a zombie. Then you put it through and upgrade/process that makes it functional. You would expect to have produced a P-zombie. But if you actually receive a human non-p-zombie you might start suspecting that the process somehow also works in the zombie dimension. P-zombie compared to usual human is intelligence similar and conciousness dissimilar. Conflating and taking them to be synonymous with each other makes talking about it hard and/or impossible.
Edit: typo that has chances to be quite meaningful. I did not mean “turn energy to color” but rather that “energy stays stable, color is unstable”.
The key to solve this problem is that extra energy was applied somewhere else, like a color machine, and this is an open system, so conservation of energy does not hold here.
If it is open, how do we know how much energy there is supposed to be, to determine that there is extra?
If we have built the machine and therefore are quite sure that there are no surprise coloring machines then the various part placements can not be the detail where we screwed up.
Why would applying a surprise jolt of energy on the ball change its color? Imagine that you find a particular bumber before which the ball is red and after which it is blue. Why would calling this bumber a “coloring machine” explain anything? Would the blue ball leaving with a tiny bit of speed deficiency explain why the ball got colored or why it is this specific bumber rather than all the others which have come from the same assembly line?
I was of course talking about a coloring machine, though one important point is with enough energy directed in the right way, you can do things like simulate a coloring machine and then add new color.
Energy, properly applied and enough energy, can do nearly everything, like changing the color.
I will give that bumber became a superpowered arcane user that can with miniscule energy make unknown effects.
It would still be interesting that this ended up happening starting by taking out a factory-standard bumber part and setting out to build a non-magical pinball machine. That is you are not trying to make a coloring machine happen. You do not have any reason to believe any unknown high-tech spy would be interested to be gaslighting you. As long as you can be super sure that you saw it blue and super sure you saw it red and that you were not trying to make it happen, you have reason to believe you do not understand what you ended up doing.
Maybe you try to look a bit more into it and try a geiger counter on the ball. Before machine it doesn’t radiate and after machine it radiates. You could feel good and go “I invented an irradiator!” or you could try to hunt down an irradiator bumber in the pinball machine. But you still do not know how you did it.
There could be any number of properties that you could learn to check your balls for. The claim is not that you need to learn all these properties to master predicting the kinetics of the pinball. The claim is not that the new properties would be a source of infinite or absurd amount of kinetic energy. The kinetics works as predicted and is closed in regards to all these other things. Learning about new properties does not change the fact that you saw the ball previously bounce around the machine. The claim is that your mastery of kinetics can not explain the machine turning out to be a converter for property number 56.
Maybe you think that kinetics can not be closed in regards to other properties. “Kinetics is everything”. Then when you try to blueprint the pinball machine in meticilous detail you should be able to predict all the other properties. Then showing what kind of ball went in and what kind of ball came out, you should be 100% be able to determine whether it was put in your unmeddled machine or in some other machine.
But meticulousness is hard and in your non-omnisciene you can only do that to the practical limit. So you learn about new property number 12. Unassembled you test each bumber separately what kind of effect it has on the ball. Your friends ask that since you always master the latest discovered property first whether you have done it for number 12 yet. You feel confident. Then you assemble the pinball machine. And put a ball through the machine. If you are surprised whether it is a property 12 converter then you have knowledge that your mastery is not at a level of what is possible to construct and measure practically.
So claims of the form “you need to keep track of property number 9 in order to be able to practically predict what happens about practically doable measurements of property 10” do not accept unknown unknowns as an excuse.
Claims of the form “you can mixup properties 8 and 7 and it will not have a practical or observable difference” combined with not being able to be close kinetics away from them means that existence of of such properties is a settleable question.
If intelligence and conciousness are separate properties and we can contain an argument to be about intelligence only, it cannot inform us about conciousness.
If intelligence and conciousness are connected properties we can not claim that an argument about one of them is irrelevant in regards to the other.
The problem is that there are a bunch of intermediate positions between LUT and strong AI. The doubt about an LLM is whether it has a world model, when it is only designed as a word model.
Nothing is fundamentally a black box.
That claim is unjustified and unjustifiable. Everything is fundamentally a black box until proven otherwise. And we will never find any conclusive proof. (I want to tell you to look up Hume’s problem of induction and Karl Popper’s solution, although I feel that making such a remark would be insulting your intelligence.) Our ability to imagine systems behaving in ways that are 100% predictable and our ability to test systems so as to ensure that they behave predictably does not change the fact that everything is always fundamentally a black box.
Nothing complex is a black box , because it has components, which can potentially be understood.
Nothing artificial is a black box to the person who built it.
An LLM is , of course, complex and artificial.
What justifies that claim?
I wasn’t arguing on that basis.
This might a key crux: I think that white box is a rare state, and interpretability is not the default case here.
Indeed I see the opposite: Black box strategies like Model Free RL work a lot better currently than white box strategies, and I think black boxes are the default scenario. You have to do a lot of work to interpret things enough to make it white box.
Any system of sufficient complexity is incomprehensible to an observer of insufficient intelligence. But that’s not fundamental.