Taboo “know” and try to ask the question again, because I think you’re engaging in a category error when you posit that, for example, a neural network actually knows anything at all. That is, the concept of “knowledge” as it applies to a human being cannot be meaningfully compared to “knowledge” as it applies to a neural network; they aren’t the same kind of thing. A Go AI doesn’t know how to play Go; it knows the current state of the board. These are entirely different categories of things.
The closest thing I think the human brain has to the kind of “knowledge” that a neural network uses is the kind of thing we represent in our cultural narrative as, for example, a spiritual guru slapping you for thinking about doing something instead of just doing it. That is, we explicitly label this kind of thing, when it occurs in the human brain, as not-knowledge.
ETA:
You can move your arm, right? You know how to move your arms and your legs and even how to do complicated things like throw balls and walk around. But you don’t actually know how to do any of those things; if you knew how to move your arm—much less something complicated like throwing balls! - it would be a relatively simple matter for you to build an arm and connect it to somebody who was missing one.
Does this seem absurd? It’s the difference between knowing how to add and knowing how to use a calculator. Knowing how to add is sufficient information to build a simple mechanical calculator, given some additional mechanical knowledge—knowing how to use a calculator gives you no such ability.
To make my own point that may be distinct from ACP’s: the point isn’t that neural networks don’t know anything. The point is that the level of description I’m operating on when I say that phrase is so imprecise that it doesn’t allow you to make exact demands like knowing “everything the NN does” or “exactly what the NN does,” for any system other than a copy of that same neural network.
If I make the verbal chain of reasoning “the NN can know things, I can know things, therefore I can know what the NN knows,” this chain of reasoning actually fails. Even though I’m using the same English word “know” both times, the logical consequences of the word are different each time I use it. If I want to make progress here, I’ll need to taboo the word “know.”
Because I think the word “know”, as used by a human understanding a model, is standing in for a particular kind of mirror-modeling, in which we possess a reproductive model of a thing in our mind which we can use to simulate a behavior, whereas the word “know”, as used by the referent AI, is standing in for “the set of information used to inform the development of a process”.
So an AI which has been trained on a game which it lost can behave “as if it has knowledge of that game”, when in fact the only remnant of that game may be a slightly adjusted parameter, perhaps a connection weighting somewhere is 1% different than it would otherwise be.
In order to “know” what the AI knows, in the sense that it knows it, requires a complete reproduction of the AI state—that is, if you know everything the AI actually knows, as opposed to the information-state that informed the development of the AI, then all you actually know, in that case, is that this particular connection is weighted 1% different; in order to meaningfully apply this knowledge, you must simulate the AI (you must know how all the connections interaction in a holistic sense), in which case you don’t know anything, you’re just asking the AI what it would do, which is not meaningfully knowing what it knows in any useful sense.
Which is basically because it doesn’t actually know anything. Its state is an algorithm, a process; this algorithm could perhaps be dissected, broken down, simplified, and turned into knowledge of how it operates—but this is just another way of simulating and querying a part of the AI; critically, knowing how the AI operates is having knowledge that the AI itself does not actually have.
Because now we are mirror-modeling the AI, and turning what the AI is, which isn’t knowledge, into something else, which is.
I guess it seems to me that you’re claiming that the referent AI isn’t doing any mirror-modelling, but I don’t know why you’d strongly believe this. It seems false about algorithms that use Monte Carlo Tree Search as KataGo does (altho another thread indicates that smart people disagree with me about this), but even for pure neural network models, I’m not sure why one would be confident that it’s false.
Because it’s expensive, slow, and orthogonal to the purpose the AI is actually trying to accomplish.
As a programmer, I take my complicated mirror models, try to figure out how to transform them into sets of numbers, try to figure out how to use one set of those numbers to create another set of those numbers. The mirror modeling is a cognitive step I have to take before I ever start programming an algorithm; it’s helpful for creating algorithms, but useless for actually running them.
Programming languages are judged as helpful in part by how well they do at pretending to be a mirror model, and efficient by how well they completely ignore the mirror model when it comes time to compile/run. There is no program which is made more efficient by representing data internally as the objects the programmers created; efficiency gains are made in compilers by figuring out how to reduce away the unnecessary complexity the programmers created for themselves so they could more easily map their messy intuitions to cold logic.
Why would an AI introduce this step in the middle of its processing?
...but this is just another way of … querying a part of the AI...
I’ve studied Go using AI and have heard others discuss the use of AI in studying Go. Even for professional Go players, the inability for the AI to explain why it gave a higher win rate to a particular move or sequence is a problem.
Even if you could program a tertiary AI which could query the Go playing AI, analyze the calculations the Go playing AI is using to make it’s judgements, and then translate that into english (or another language) so that this tertiary AI could explain why the Go playing AI made a move, I would still disagree that even this hybrid system ‘knew’ how to play Go.
There is a definite difference between ‘calculating’ and ‘reasoning’ such that even a neural network with it’s training I think is really still just one big calculator, not a reasoner.
Taboo “know” and try to ask the question again, because I think you’re engaging in a category error when you posit that, for example, a neural network actually knows anything at all. That is, the concept of “knowledge” as it applies to a human being cannot be meaningfully compared to “knowledge” as it applies to a neural network; they aren’t the same kind of thing. A Go AI doesn’t know how to play Go; it knows the current state of the board. These are entirely different categories of things.
The closest thing I think the human brain has to the kind of “knowledge” that a neural network uses is the kind of thing we represent in our cultural narrative as, for example, a spiritual guru slapping you for thinking about doing something instead of just doing it. That is, we explicitly label this kind of thing, when it occurs in the human brain, as not-knowledge.
ETA:
You can move your arm, right? You know how to move your arms and your legs and even how to do complicated things like throw balls and walk around. But you don’t actually know how to do any of those things; if you knew how to move your arm—much less something complicated like throwing balls! - it would be a relatively simple matter for you to build an arm and connect it to somebody who was missing one.
Does this seem absurd? It’s the difference between knowing how to add and knowing how to use a calculator. Knowing how to add is sufficient information to build a simple mechanical calculator, given some additional mechanical knowledge—knowing how to use a calculator gives you no such ability.
Why do you believe that?
To make my own point that may be distinct from ACP’s: the point isn’t that neural networks don’t know anything. The point is that the level of description I’m operating on when I say that phrase is so imprecise that it doesn’t allow you to make exact demands like knowing “everything the NN does” or “exactly what the NN does,” for any system other than a copy of that same neural network.
If I make the verbal chain of reasoning “the NN can know things, I can know things, therefore I can know what the NN knows,” this chain of reasoning actually fails. Even though I’m using the same English word “know” both times, the logical consequences of the word are different each time I use it. If I want to make progress here, I’ll need to taboo the word “know.”
Because I think the word “know”, as used by a human understanding a model, is standing in for a particular kind of mirror-modeling, in which we possess a reproductive model of a thing in our mind which we can use to simulate a behavior, whereas the word “know”, as used by the referent AI, is standing in for “the set of information used to inform the development of a process”.
So an AI which has been trained on a game which it lost can behave “as if it has knowledge of that game”, when in fact the only remnant of that game may be a slightly adjusted parameter, perhaps a connection weighting somewhere is 1% different than it would otherwise be.
In order to “know” what the AI knows, in the sense that it knows it, requires a complete reproduction of the AI state—that is, if you know everything the AI actually knows, as opposed to the information-state that informed the development of the AI, then all you actually know, in that case, is that this particular connection is weighted 1% different; in order to meaningfully apply this knowledge, you must simulate the AI (you must know how all the connections interaction in a holistic sense), in which case you don’t know anything, you’re just asking the AI what it would do, which is not meaningfully knowing what it knows in any useful sense.
Which is basically because it doesn’t actually know anything. Its state is an algorithm, a process; this algorithm could perhaps be dissected, broken down, simplified, and turned into knowledge of how it operates—but this is just another way of simulating and querying a part of the AI; critically, knowing how the AI operates is having knowledge that the AI itself does not actually have.
Because now we are mirror-modeling the AI, and turning what the AI is, which isn’t knowledge, into something else, which is.
I guess it seems to me that you’re claiming that the referent AI isn’t doing any mirror-modelling, but I don’t know why you’d strongly believe this. It seems false about algorithms that use Monte Carlo Tree Search as KataGo does (altho another thread indicates that smart people disagree with me about this), but even for pure neural network models, I’m not sure why one would be confident that it’s false.
Because it’s expensive, slow, and orthogonal to the purpose the AI is actually trying to accomplish.
As a programmer, I take my complicated mirror models, try to figure out how to transform them into sets of numbers, try to figure out how to use one set of those numbers to create another set of those numbers. The mirror modeling is a cognitive step I have to take before I ever start programming an algorithm; it’s helpful for creating algorithms, but useless for actually running them.
Programming languages are judged as helpful in part by how well they do at pretending to be a mirror model, and efficient by how well they completely ignore the mirror model when it comes time to compile/run. There is no program which is made more efficient by representing data internally as the objects the programmers created; efficiency gains are made in compilers by figuring out how to reduce away the unnecessary complexity the programmers created for themselves so they could more easily map their messy intuitions to cold logic.
Why would an AI introduce this step in the middle of its processing?
I’ve studied Go using AI and have heard others discuss the use of AI in studying Go. Even for professional Go players, the inability for the AI to explain why it gave a higher win rate to a particular move or sequence is a problem.
Even if you could program a tertiary AI which could query the Go playing AI, analyze the calculations the Go playing AI is using to make it’s judgements, and then translate that into english (or another language) so that this tertiary AI could explain why the Go playing AI made a move, I would still disagree that even this hybrid system ‘knew’ how to play Go.
There is a definite difference between ‘calculating’ and ‘reasoning’ such that even a neural network with it’s training I think is really still just one big calculator, not a reasoner.