The real issue with the Chinese Room argument is that it misidentifies the reason we can’t make Giant Look Up Tables: The reason is Conservation of Energy and lightspeed means we can’t create new energy out of nothing, and this matters. Specifically, Giant Look Up Tables grow fast to learn new things. By the time you have emulated a human mind, you have now used countless observable universes to make the look up table, which you can’t do
That’s why look up tables can’t be used for intelligence past simple tasks, and why GPT can’t be a look up table.
To be fair, even if what you’re referring to above is true (I don’t believe it is—lookup table compression is a thing), it’s an implementation detail. It doesn’t matter that a naive implementation might not fit in our current observable universe; it need merely be able to exist in some universe for the argument to hold.
And in a way, this is my core problem with Searle’s argument. I believe you can fully emulate a human with both sufficiently large lookup tables, and also with pretty small lookup tables combined with some table expansion/generation code running on an organic substrate. I don’t challenge the argument based the technical feasibility of the table implementation. I challenge the argument on the basis that the author mistakenly believes that the implementation of any given table (static lookup table versus algorithmic lookup) somehow determines consciousness.
While I agree with your argument against Searle, it matters whether it’s at all feasible because if it isn’t, then the argument Searle is using has no real relation to AI today or in the future, and therefore we can’t use it to argue against the hypothesis that they lack intelligence/consciousness.
To be clear, I agree with your argument. I just want to note that physical impossibilities are being used to argue that today’s AI aren’t intelligent or conscious.
Instructions to carry out paperwork. You read a process off from the book and not a symbol.
If you are allowed to make (guided by the book) notes then you have memory that persists between “lookups”.
If you have new paper available and there are recursive instructions in the book it might be quite a while you write symbols for “internal consumption” before you produce any symbol that is put in the output slot.
It might be that the original idea was less specified but I think it points in the same direction as effective method.
Its instructions need only to be followed rigorously to succeed. In other words, it requires no ingenuity to succeed
With “you do not need to know what you are doing” meaning that “sticking to the book” is sufficient ie no ingenuity.
But brainless action still involves more stuff than just writing a single (character/phrase) to the output slot.
This is basically black box intelligence, and there’s no reason to make the assumption that black box methods cannot work. Indeed black boxes are already used for AI today.
It may be nice for it to be white box, but I see no reason for black boxes not to be intelligent or conscious.
It is about that the human knows how to read the book and doesn’t misunderstand it. If you where in the chinense room and had a book written in english and you do not know even english you do not know how to operate the room. It is a “no box” in that the human does not need to bring anything to the table (and the book does it all).
If you knew what the chinese was about it wouldn’t be an obstacle if you could not read a book written in english. But knowing english whether or not you know chinese doesn’t make a difference.
So the critical disagreement is assuming we can add enough energy to learn everything in the book with no priors, and arbitrarily large memory capacity, then it is equivalent to actually knowing Chinese, since you can store arbitrarily large memory in your head, which includes a vast but finite ruleset for Chinese. Of course to learn new languages, you will have to expend it again, which rapidly spirals into an uncontrollable energy cost, which is why Chinese Rooms can’t actually be built in real life, which is why the success of GPT refutes Searle and Gary Marcus’s thesis that they are just Chinese Rooms.
We are allowed to only look at a single page / sentence at a time which is quite a lot more possible with a finite read-head and not need to remember pages we have turned away from.
You can google to benefit from the whole internet without needing to download the whole internet. You can work a 2 TB hardrive while only having 64MiB L1 cache. You can run arbitrary python programs with a CPU core with an instruction set that is small, finite and can not be appended.
You can manage a 2-hour paperwork session with a 6-second memory buffer.
I guess the human needs to bring english in their head instead of having a literally empty head. But having english in your head is not energywise a miracle to do.
It already has all the rules for learning Chinese, so it can manage to remain a useful source of learning even if the owner passes it up for something else.
You can read the paper yourself. It doesn’t exactly say it is a LUT, and it doesn’t exactly say it isnt.
The passage quoted in the OP says
Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) “together with a book of instructions for manipulating the symbols (the program).
...But says nothing about the complexity of the “program”. Philosophybear, like so many others, jumps to the conclusion that it must be a LUT.
The problem is that the limiting case of pure memorization is a look up table of some form. If we drop the requirement for pure memorization and instead allow world models or heuristics, then we don’t have to use a look up table.
More generally, this is important, since if AI could only memorize like the Chinese Room, GPT-3 would not be at all possible without expending vastly more energy than was actually done, so we should strongly update against the pure memorization hypothesis.
The other complaint I think Searle is making is that black box intelligence/consciousness, whether a look up table or something else, is not real intelligence/consciousness. And that’s wrong, since there are in fact black box intelligences we can’t interpret well like Evolution Strategies or Model Free RL, so the fact that a Turing machine emulates Chinese in a black box format is equivalent to learning it in a white box format.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
If some task is exactly breathing then it is plausible that starting from nothing can work if the task involves breathing. If it can’t be done then “taskness” is not sufficient.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
It’s already covered by conservation of energy, IE new feelings must pay an energy cost to feel something new, so this is double-counting.
And conservation of energy does not always hold in mathematics (though such a cosmology would allow time travel.) Thus this is a contingent fact of our universe, not something that has to be universally true.
Paying an energy cost would be a task and would operate on the intelligence side of it. Typically feelings come with cognition discernable aspects.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy conversation conservation that color conversion should be impossible.
If you had some part of the flipper that red the color of the ball and conditional on that kicked the ball then one could tell the color of the ball from its trajectory. If no such things exists then pointfulness of even talking about color might drop and somebody might think that color is “epiphenomena” that is not an actual thing that happens. A claim that the machine recolors the ball is not a claim about trajectories.
I made a bad analog in using breathing as that can sound like a trajectory term. Somebody that believes in P-zombies might ask whether a particular actor is a P-zombie. You start something that is definetely not functional and definetely is a zombie. Then you put it through and upgrade/process that makes it functional. You would expect to have produced a P-zombie. But if you actually receive a human non-p-zombie you might start suspecting that the process somehow also works in the zombie dimension. P-zombie compared to usual human is intelligence similar and conciousness dissimilar. Conflating and taking them to be synonymous with each other makes talking about it hard and/or impossible.
Edit: typo that has chances to be quite meaningful. I did not mean “turn energy to color” but rather that “energy stays stable, color is unstable”.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy conversation that color conversion should be impossible.
The key to solve this problem is that extra energy was applied somewhere else, like a color machine, and this is an open system, so conservation of energy does not hold here.
If it is open, how do we know how much energy there is supposed to be, to determine that there is extra?
If we have built the machine and therefore are quite sure that there are no surprise coloring machines then the various part placements can not be the detail where we screwed up.
Why would applying a surprise jolt of energy on the ball change its color? Imagine that you find a particular bumber before which the ball is red and after which it is blue. Why would calling this bumber a “coloring machine” explain anything? Would the blue ball leaving with a tiny bit of speed deficiency explain why the ball got colored or why it is this specific bumber rather than all the others which have come from the same assembly line?
I was of course talking about a coloring machine, though one important point is with enough energy directed in the right way, you can do things like simulate a coloring machine and then add new color.
Energy, properly applied and enough energy, can do nearly everything, like changing the color.
I will give that bumber became a superpowered arcane user that can with miniscule energy make unknown effects.
It would still be interesting that this ended up happening starting by taking out a factory-standard bumber part and setting out to build a non-magical pinball machine. That is you are not trying to make a coloring machine happen. You do not have any reason to believe any unknown high-tech spy would be interested to be gaslighting you. As long as you can be super sure that you saw it blue and super sure you saw it red and that you were not trying to make it happen, you have reason to believe you do not understand what you ended up doing.
Maybe you try to look a bit more into it and try a geiger counter on the ball. Before machine it doesn’t radiate and after machine it radiates. You could feel good and go “I invented an irradiator!” or you could try to hunt down an irradiator bumber in the pinball machine. But you still do not know how you did it.
There could be any number of properties that you could learn to check your balls for. The claim is not that you need to learn all these properties to master predicting the kinetics of the pinball. The claim is not that the new properties would be a source of infinite or absurd amount of kinetic energy. The kinetics works as predicted and is closed in regards to all these other things. Learning about new properties does not change the fact that you saw the ball previously bounce around the machine. The claim is that your mastery of kinetics can not explain the machine turning out to be a converter for property number 56.
Maybe you think that kinetics can not be closed in regards to other properties. “Kinetics is everything”. Then when you try to blueprint the pinball machine in meticilous detail you should be able to predict all the other properties. Then showing what kind of ball went in and what kind of ball came out, you should be 100% be able to determine whether it was put in your unmeddled machine or in some other machine.
But meticulousness is hard and in your non-omnisciene you can only do that to the practical limit. So you learn about new property number 12. Unassembled you test each bumber separately what kind of effect it has on the ball. Your friends ask that since you always master the latest discovered property first whether you have done it for number 12 yet. You feel confident. Then you assemble the pinball machine. And put a ball through the machine. If you are surprised whether it is a property 12 converter then you have knowledge that your mastery is not at a level of what is possible to construct and measure practically.
So claims of the form “you need to keep track of property number 9 in order to be able to practically predict what happens about practically doable measurements of property 10” do not accept unknown unknowns as an excuse.
Claims of the form “you can mixup properties 8 and 7 and it will not have a practical or observable difference” combined with not being able to be close kinetics away from them means that existence of of such properties is a settleable question.
If intelligence and conciousness are separate properties and we can contain an argument to be about intelligence only, it cannot inform us about conciousness.
If intelligence and conciousness are connected properties we can not claim that an argument about one of them is irrelevant in regards to the other.
The problem is that there are a bunch of intermediate positions between LUT and strong AI. The doubt about an LLM is whether it has a world model, when it is only designed as a word model.
That claim is unjustified and unjustifiable. Everything is fundamentally a black box until proven otherwise. And we will never find any conclusive proof. (I want to tell you to look up Hume’s problem of induction and Karl Popper’s solution, although I feel that making such a remark would be insulting your intelligence.) Our ability to imagine systems behaving in ways that are 100% predictable and our ability to test systems so as to ensure that they behave predictably does not change the fact that everything is always fundamentally a black box.
This might a key crux: I think that white box is a rare state, and interpretability is not the default case here.
Indeed I see the opposite: Black box strategies like Model Free RL work a lot better currently than white box strategies, and I think black boxes are the default scenario. You have to do a lot of work to interpret things enough to make it white box.
The real issue with the Chinese Room argument is that it misidentifies the reason we can’t make Giant Look Up Tables: The reason is Conservation of Energy and lightspeed means we can’t create new energy out of nothing, and this matters. Specifically, Giant Look Up Tables grow fast to learn new things. By the time you have emulated a human mind, you have now used countless observable universes to make the look up table, which you can’t do
That’s why look up tables can’t be used for intelligence past simple tasks, and why GPT can’t be a look up table.
To be fair, even if what you’re referring to above is true (I don’t believe it is—lookup table compression is a thing), it’s an implementation detail. It doesn’t matter that a naive implementation might not fit in our current observable universe; it need merely be able to exist in some universe for the argument to hold.
And in a way, this is my core problem with Searle’s argument. I believe you can fully emulate a human with both sufficiently large lookup tables, and also with pretty small lookup tables combined with some table expansion/generation code running on an organic substrate. I don’t challenge the argument based the technical feasibility of the table implementation. I challenge the argument on the basis that the author mistakenly believes that the implementation of any given table (static lookup table versus algorithmic lookup) somehow determines consciousness.
While I agree with your argument against Searle, it matters whether it’s at all feasible because if it isn’t, then the argument Searle is using has no real relation to AI today or in the future, and therefore we can’t use it to argue against the hypothesis that they lack intelligence/consciousness.
To be clear, I agree with your argument. I just want to note that physical impossibilities are being used to argue that today’s AI aren’t intelligent or conscious.
Despite what is frequently alleged here, the Chinese Room is not specified as a look up table.
Then what is the Chinese Room, exactly, if it isn’t a look up table?
Instructions to carry out paperwork. You read a process off from the book and not a symbol.
If you are allowed to make (guided by the book) notes then you have memory that persists between “lookups”.
If you have new paper available and there are recursive instructions in the book it might be quite a while you write symbols for “internal consumption” before you produce any symbol that is put in the output slot.
It might be that the original idea was less specified but I think it points in the same direction as effective method.
With “you do not need to know what you are doing” meaning that “sticking to the book” is sufficient ie no ingenuity.
But brainless action still involves more stuff than just writing a single (character/phrase) to the output slot.
This is basically black box intelligence, and there’s no reason to make the assumption that black box methods cannot work. Indeed black boxes are already used for AI today.
It may be nice for it to be white box, but I see no reason for black boxes not to be intelligent or conscious.
It is about that the human knows how to read the book and doesn’t misunderstand it. If you where in the chinense room and had a book written in english and you do not know even english you do not know how to operate the room. It is a “no box” in that the human does not need to bring anything to the table (and the book does it all).
If you knew what the chinese was about it wouldn’t be an obstacle if you could not read a book written in english. But knowing english whether or not you know chinese doesn’t make a difference.
So the critical disagreement is assuming we can add enough energy to learn everything in the book with no priors, and arbitrarily large memory capacity, then it is equivalent to actually knowing Chinese, since you can store arbitrarily large memory in your head, which includes a vast but finite ruleset for Chinese. Of course to learn new languages, you will have to expend it again, which rapidly spirals into an uncontrollable energy cost, which is why Chinese Rooms can’t actually be built in real life, which is why the success of GPT refutes Searle and Gary Marcus’s thesis that they are just Chinese Rooms.
No, I think I am not claiming about energy usage.
We are allowed to only look at a single page / sentence at a time which is quite a lot more possible with a finite read-head and not need to remember pages we have turned away from.
You can google to benefit from the whole internet without needing to download the whole internet. You can work a 2 TB hardrive while only having 64MiB L1 cache. You can run arbitrary python programs with a CPU core with an instruction set that is small, finite and can not be appended.
You can manage a 2-hour paperwork session with a 6-second memory buffer.
I guess the human needs to bring english in their head instead of having a literally empty head. But having english in your head is not energywise a miracle to do.
Then it is equivalent to who/what knowing Chinese?
Both the human and the book.
Separately or in conjunction?
Separately.
Well, I don’t see how the Operator can speak Chinese without the Book, or vice versa.
Specifically, once it has memorized the book, it doesn’t need to use the book anymore, and can rely on it’s own memory.
So the operator can manage without the book, but the book can’t manage without the operator...?
The book can manage without the operator.
How?
It already has all the rules for learning Chinese, so it can manage to remain a useful source of learning even if the owner passes it up for something else.
So it can’t just speak Chinese.
Yes, that’s right.
You can read the paper yourself. It doesn’t exactly say it is a LUT, and it doesn’t exactly say it isnt.
The passage quoted in the OP says
...But says nothing about the complexity of the “program”. Philosophybear, like so many others, jumps to the conclusion that it must be a LUT.
The problem is that the limiting case of pure memorization is a look up table of some form. If we drop the requirement for pure memorization and instead allow world models or heuristics, then we don’t have to use a look up table.
More generally, this is important, since if AI could only memorize like the Chinese Room, GPT-3 would not be at all possible without expending vastly more energy than was actually done, so we should strongly update against the pure memorization hypothesis.
The other complaint I think Searle is making is that black box intelligence/consciousness, whether a look up table or something else, is not real intelligence/consciousness. And that’s wrong, since there are in fact black box intelligences we can’t interpret well like Evolution Strategies or Model Free RL, so the fact that a Turing machine emulates Chinese in a black box format is equivalent to learning it in a white box format.
Searle might be trying to tease apart that intelligence/consciousness. If you take a feeling and breathing human being and make it perform a complex task, you didn’t add feeling at any point but it just got passively preserved. So if you start from nothing, add performance of a complex task and end up with a feeling and breathing human being, it is a bit weird where the breathing came from.
If some task is exactly breathing then it is plausible that starting from nothing can work if the task involves breathing. If it can’t be done then “taskness” is not sufficient.
It’s already covered by conservation of energy, IE new feelings must pay an energy cost to feel something new, so this is double-counting.
And conservation of energy does not always hold in mathematics (though such a cosmology would allow time travel.) Thus this is a contingent fact of our universe, not something that has to be universally true.
Paying an energy cost would be a task and would operate on the intelligence side of it. Typically feelings come with cognition discernable aspects.
Think of a pinball machine that moves a ball in complex ways and entirely built and defined as moving operations. Then you put in a red ball and the pinball machine spits out a blue ball. This is surprising as no operation should color the ball. One can’t explain with kinetic energy
conversationconservation that color conversion should be impossible.If you had some part of the flipper that red the color of the ball and conditional on that kicked the ball then one could tell the color of the ball from its trajectory. If no such things exists then pointfulness of even talking about color might drop and somebody might think that color is “epiphenomena” that is not an actual thing that happens. A claim that the machine recolors the ball is not a claim about trajectories.
I made a bad analog in using breathing as that can sound like a trajectory term. Somebody that believes in P-zombies might ask whether a particular actor is a P-zombie. You start something that is definetely not functional and definetely is a zombie. Then you put it through and upgrade/process that makes it functional. You would expect to have produced a P-zombie. But if you actually receive a human non-p-zombie you might start suspecting that the process somehow also works in the zombie dimension. P-zombie compared to usual human is intelligence similar and conciousness dissimilar. Conflating and taking them to be synonymous with each other makes talking about it hard and/or impossible.
Edit: typo that has chances to be quite meaningful. I did not mean “turn energy to color” but rather that “energy stays stable, color is unstable”.
The key to solve this problem is that extra energy was applied somewhere else, like a color machine, and this is an open system, so conservation of energy does not hold here.
If it is open, how do we know how much energy there is supposed to be, to determine that there is extra?
If we have built the machine and therefore are quite sure that there are no surprise coloring machines then the various part placements can not be the detail where we screwed up.
Why would applying a surprise jolt of energy on the ball change its color? Imagine that you find a particular bumber before which the ball is red and after which it is blue. Why would calling this bumber a “coloring machine” explain anything? Would the blue ball leaving with a tiny bit of speed deficiency explain why the ball got colored or why it is this specific bumber rather than all the others which have come from the same assembly line?
I was of course talking about a coloring machine, though one important point is with enough energy directed in the right way, you can do things like simulate a coloring machine and then add new color.
Energy, properly applied and enough energy, can do nearly everything, like changing the color.
I will give that bumber became a superpowered arcane user that can with miniscule energy make unknown effects.
It would still be interesting that this ended up happening starting by taking out a factory-standard bumber part and setting out to build a non-magical pinball machine. That is you are not trying to make a coloring machine happen. You do not have any reason to believe any unknown high-tech spy would be interested to be gaslighting you. As long as you can be super sure that you saw it blue and super sure you saw it red and that you were not trying to make it happen, you have reason to believe you do not understand what you ended up doing.
Maybe you try to look a bit more into it and try a geiger counter on the ball. Before machine it doesn’t radiate and after machine it radiates. You could feel good and go “I invented an irradiator!” or you could try to hunt down an irradiator bumber in the pinball machine. But you still do not know how you did it.
There could be any number of properties that you could learn to check your balls for. The claim is not that you need to learn all these properties to master predicting the kinetics of the pinball. The claim is not that the new properties would be a source of infinite or absurd amount of kinetic energy. The kinetics works as predicted and is closed in regards to all these other things. Learning about new properties does not change the fact that you saw the ball previously bounce around the machine. The claim is that your mastery of kinetics can not explain the machine turning out to be a converter for property number 56.
Maybe you think that kinetics can not be closed in regards to other properties. “Kinetics is everything”. Then when you try to blueprint the pinball machine in meticilous detail you should be able to predict all the other properties. Then showing what kind of ball went in and what kind of ball came out, you should be 100% be able to determine whether it was put in your unmeddled machine or in some other machine.
But meticulousness is hard and in your non-omnisciene you can only do that to the practical limit. So you learn about new property number 12. Unassembled you test each bumber separately what kind of effect it has on the ball. Your friends ask that since you always master the latest discovered property first whether you have done it for number 12 yet. You feel confident. Then you assemble the pinball machine. And put a ball through the machine. If you are surprised whether it is a property 12 converter then you have knowledge that your mastery is not at a level of what is possible to construct and measure practically.
So claims of the form “you need to keep track of property number 9 in order to be able to practically predict what happens about practically doable measurements of property 10” do not accept unknown unknowns as an excuse.
Claims of the form “you can mixup properties 8 and 7 and it will not have a practical or observable difference” combined with not being able to be close kinetics away from them means that existence of of such properties is a settleable question.
If intelligence and conciousness are separate properties and we can contain an argument to be about intelligence only, it cannot inform us about conciousness.
If intelligence and conciousness are connected properties we can not claim that an argument about one of them is irrelevant in regards to the other.
The problem is that there are a bunch of intermediate positions between LUT and strong AI. The doubt about an LLM is whether it has a world model, when it is only designed as a word model.
Nothing is fundamentally a black box.
That claim is unjustified and unjustifiable. Everything is fundamentally a black box until proven otherwise. And we will never find any conclusive proof. (I want to tell you to look up Hume’s problem of induction and Karl Popper’s solution, although I feel that making such a remark would be insulting your intelligence.) Our ability to imagine systems behaving in ways that are 100% predictable and our ability to test systems so as to ensure that they behave predictably does not change the fact that everything is always fundamentally a black box.
Nothing complex is a black box , because it has components, which can potentially be understood.
Nothing artificial is a black box to the person who built it.
An LLM is , of course, complex and artificial.
What justifies that claim?
I wasn’t arguing on that basis.
This might a key crux: I think that white box is a rare state, and interpretability is not the default case here.
Indeed I see the opposite: Black box strategies like Model Free RL work a lot better currently than white box strategies, and I think black boxes are the default scenario. You have to do a lot of work to interpret things enough to make it white box.
Any system of sufficient complexity is incomprehensible to an observer of insufficient intelligence. But that’s not fundamental.