I agree that’s a more interesting question, and computational complexity theorists have done work on it which I don’t fully understand, but it also doesn’t seem as relevant for AI safety questions.
Davidmanheim
Regarding Chess agents, Vanessa pointed out that while only perfect play is optimal, informally we would consider agents to have an objective that is better served by slightly better play, for example, an agent rated 2500 ELO is better than one rated 1800, which is better than one rated 1000, etc. That means that lots of “chess minds” which are non-optimal are still somewhat rational at their goal.
I think that it’s very likely that even according to this looser definition, almost all chess moves, and therefore almost all “possible” chess bots, fail to do much to accomplish the goal.
We could check this informally by evaluating the set of possible moves in normal games would be classified as blunders, using a method such as the one used here to evaluate what proportion of actual moves made by players are blunders. Figure 1 there implies that in positions with many legal moves, a larger proportion are blunders—but this is looking at the empirical blunder rate by those good enough to be playing ranked chess. Another method would be to look at a bot that actually implements “pick a random legal move”—namely Brutus RND. It has an ELO of 255 when ranked against other amateur chess bots, and wins only occasionally against some of the worst bots; it seems hard to figure out from that what proportion of moves are good, but it’s evidently a fairly small proportion.
We earlier mentioned that it is required that the finite mapping be precomputed. If it is for arbitrary Turing machines, including those that don’t halt, we need infinite time, so the claim that we can map to arbitrary Turing machines fails. If we restrict it to those which halt, we need to check that before providing the map, which requires solving the halting problem to provide the map.
Edit to add: I’m confused why this is getting “disagree” votes—can someone explain why or how this is an incorrect logical step, or
OK, so this is helpful, but if I understood you correctly, I think it’s assuming too much about the setup. For #1, in the examples we’re discussing, the states of the object aren’t predictably changing in complex ways—just that it will change “states” in ways that can be predicted to follow a specific path, which can be mapped to some set of states. The states are arbitrary, and per the argument don’t vary in some way that does any work—and so as I argued, they can be mapped to some set of consecutive integers. But this means that the actions of the physical object are predetermined in the mapping.
And the difference between that situation and the CNS is that we know he neural circuitry is doing work—the exact features are complex and only partly understood, but the result is clearly capable of doing computation in the sense of Turing machines.
I think this was a valuable post, albeit ending up somewhat incorrect about whether LLMs would be agentic—not because they developed the capacity on their own, but because people intentionally built and are building structure around LLMs to enable agency. That said, the underlying point stands—it is very possible that LLMs could be a safe foundation for non-agentic AI, and many research groups are pursuing that today.
The blogpost this points to was an important contribution at the time, more clearly laying out extreme cases for the future. (The replies there were also particularly valuable.)
I think this post makes an important and still neglected claim that people should write their work more clearly and get it published in academia, instead of embracing the norms of the narrower community they interact with. There has been significant movement in this direction in the past 2 years, and I think this posts marks a critical change in what the community suggests and values in terms of output.
“the actual thinking-action that the mapping interprets”
I don’t think this is conceptually correct. Looking at the chess playing waterfall that Aaronson discusses, the mapping itself is doing all of the computation. The fact that the mapping ran in the past doesn’t change the fact that it’s the location of the computation, any more than the fact that it takes milliseconds for my nerve impulses to reach my fingers means that my fingers are doing the thinking in writing this essay. (Though given the typos you found, it would be convenient to blame them.)they assume ad arguendo that you can instantiate the computations we’re interested in (consciousness) in a headful of meat, and then try to show that if this is the case, many other finite collections of matter ought to be able to do the job just as well.
Yes, they assume that whatever runs the algorithm is experiencing running the algorithm from the inside. And yes, many specific finite systems can do so—namely, GPUs and CPUs, as well as the wetware in our head. But without the claim that arbitrary items can do these computations, it seems that the arguendo is saying nothing different than the conclusion—right?
Looks like I messed up cutting and pasting—thanks!
Thanks—fixed!
Yeah, perhaps refuting is too strong given that the central claim is that we can’t know what is and is not doing computation—which I think is wrong, but requires a more nuanced discussion. However, the narrow claims they made inter-alia were strong enough to refute, specifically by showing that their claims are equivalent to saying the integers are doing arbitrary computation—when making the claim itself requires the computation to take place elsewhere!
Seems worth noting that the claims of most of the philosophers being cited here is (1) - that even rocks are doing the same computation as minds.
I agree that this wasn’t intended as an introduction to the topic. For that, I will once again recommend Scott Aaronson’s excellent mini-book explaining computational complexity to philosophers.
I agree that the post isn’t a definition of what computation is—but I don’t need to be able to define fire to be able to point out something that definitely isn’t on fire! So I don’t really understand your claim. I agree that it’s objectively hard to interpret computation, but it’s not at all hard to interpret the fact that the integers are less complex and doing less complex computation than, say, an exponential-time Turing machine—and given the specific arguments being made, neither is a wall or a bag of popcorn. Which, as I just responded to the linked comment, was how I understood the position being taken by Searle, Putnam, and Johnson. (And even this ignores that one implication of the difference in complexity is that the wall / bag of popcorn / whatever is not mappable to arbitrary computations, since the number of steps required for a computation may not be finite!)
I’ve written my point more clearly here: https://www.lesswrong.com/posts/zxLbepy29tPg8qMnw/refuting-searle-s-wall-putnam-s-rock-and-johnson-s-popcorn
I think ‘we estimate… to be’
Your/Aaronson’s claim is that only the fully connected, sensibly interacting calculation matters.
Not at all. I’m not making any claim about what matters or counts here, just pointing out a confusion in the claims that were made here and by many philosophers who discussed the topic.
You disagree with Aaronson that the location of the complexity is in the interpreter, or you disagree that it matters?
In the first case, I’ll defer to him as the expert. But in the second, the complexity is an internal property of the system! (And it’s a property in a sense stronger than almost anything we talk about in philosophy; it’s not just a property of the world around us, because as Gödel and others showed, complexity is a necessary fact about the nature of mathematics!)
Yeah, something like that. See my response to Euan in the other reply to my post.
Yes, and no, it does not boil down to Chalmer’s argument. (as Aaronson makes clear in the paragraph before the one you quote, where he cites the Chalmers argument!) The argument from complexity is about the nature and complexity of systems capable of playing chess—which is why I think you need to carefully read the entire piece and think about what it says.
But as a small rejoinder, if we’re talking about playing a single game, the entire argument is ridiculous; I can write the entire “algorithm” a kilobyte of specific instructions. So it’s not that an algorithm must be capable of playing multiple counterfactual games to qualify, or that counterfactuals are required for moral weight—it’s that the argument hinges on a misunderstanding of how complex different classes of system need to be to do the things they do.
PS. Apologies that the original response comes off as combative—I really think this discussion is important, and wanted to engage to correct an important point, but have very little time to do so at the moment!
That seems like a reasonable idea. It seems not at all related to what any of the philosophers proposed.
For their proposals, it seems like the computational process is more like:
1. Extract a specific string of 1s and zeros from the sandstorm’s initial position, and another from it’s final position, with the some length as the length of the full description of the mind.
2. Calculate the bitwise sum of the initial mind state and the initial sand position.
3. Calculate the bitwise sum of the final mind state and the final sand position.
4. Take the output of state 2 and replace it with the output of state 3.
5. Declare that the sandstorm is doing something isomorphic to what the mind did. Ignore the fact that the internal process is completely unrelated, and all of the computation was done inside of the mind, and you’re just copying answers.