When you combine three blank areas of your map, the blank parts don’t cancel out. Instead, you get a part of your map that you should be even more uncertain about.
I think this makes sense. However, and I don’t know whether I obfuscated this point somewhere, I don’t think I was arguing that we should be more certain about a particular theory. Indeed, from my perspective, I was arguing against reifying a single concept (self-reflectivity) as the thing that defines whether something is conscious, before we know anything about humans, much less whether humans are even capable of self-reflection in some discontinuous way from other animals.
Rather, ‘moral value is a kludge’ and ‘consciousness is a kludge’ both make me update toward thinking the set of moral patients are smaller -- these engines don’t become less engine-y via being kludges, they just become more complicated and laden-with-arbitrary-structure.
I guess that when I said that brains are kludges, I was trying to say that their boundaries were fuzzy, rather than saying that they have well-defined boundaries but that the concept is extremely fragile, such that if you take away a single property from them they cease to be human. (I probably shouldn’t have used the term, and described it this way).
Complex structures like “tables” tend to be the type of thing that if you modify them across one or two dimensions, they belong to the same category. By contrast, a hydrogen atom is simple, and is the type of thing that if you take a property away from it, it ceases to be a hydrogen atom.
When I imagined a “consciousness engine” I visualized a simple system with clear moving parts, like a hydrogen atom. And conceptually, one of those moving parts could be a highly modular self-reflectivity component. Under this view, it might make a lot of sense that self-reflectivity is the defining component to a human, but I don’t suspect these things are actually that cleanly separable from the rest of the system.
In other words, it seems like the best model of a “table” or some other highly fuzzy concept, is not some extremely precise description of the exact properties that define a table, but rather some additive model in which each feature contributes some “tableness”, and such that no feature alone can either make something a table or prevent something from being a table. My intuitions about consciousness feel this way, but I’m not too certain about any of this.
I’d say my visualization of consciousness is less like a typical steam engine or table, and more like a Rube Goldberg machine designed by a very confused committee of terrible engineers. You can remove some parts of the machine without breaking anything, but a lot of other parts are necessary for the thing to work.
It should also be possible to design an AI that has ‘human-like consciousness’ via a much less kludge-ish process—I don’t think that much complexity is morally essential.
But chickens were built by a confused committee just like humans were, so they’ll have their own enormous intricate kludges (which may or may not be the same kind of machine as the Consciousness Machine in our heads), rather than having the really efficient small version of the consciousness-machine.
I think this makes sense. However, and I don’t know whether I obfuscated this point somewhere, I don’t think I was arguing that we should be more certain about a particular theory. Indeed, from my perspective, I was arguing against reifying a single concept (self-reflectivity) as the thing that defines whether something is conscious, before we know anything about humans, much less whether humans are even capable of self-reflection in some discontinuous way from other animals.
I guess that when I said that brains are kludges, I was trying to say that their boundaries were fuzzy, rather than saying that they have well-defined boundaries but that the concept is extremely fragile, such that if you take away a single property from them they cease to be human. (I probably shouldn’t have used the term, and described it this way).
Complex structures like “tables” tend to be the type of thing that if you modify them across one or two dimensions, they belong to the same category. By contrast, a hydrogen atom is simple, and is the type of thing that if you take a property away from it, it ceases to be a hydrogen atom.
When I imagined a “consciousness engine” I visualized a simple system with clear moving parts, like a hydrogen atom. And conceptually, one of those moving parts could be a highly modular self-reflectivity component. Under this view, it might make a lot of sense that self-reflectivity is the defining component to a human, but I don’t suspect these things are actually that cleanly separable from the rest of the system.
In other words, it seems like the best model of a “table” or some other highly fuzzy concept, is not some extremely precise description of the exact properties that define a table, but rather some additive model in which each feature contributes some “tableness”, and such that no feature alone can either make something a table or prevent something from being a table. My intuitions about consciousness feel this way, but I’m not too certain about any of this.
I’d say my visualization of consciousness is less like a typical steam engine or table, and more like a Rube Goldberg machine designed by a very confused committee of terrible engineers. You can remove some parts of the machine without breaking anything, but a lot of other parts are necessary for the thing to work.
It should also be possible to design an AI that has ‘human-like consciousness’ via a much less kludge-ish process—I don’t think that much complexity is morally essential.
But chickens were built by a confused committee just like humans were, so they’ll have their own enormous intricate kludges (which may or may not be the same kind of machine as the Consciousness Machine in our heads), rather than having the really efficient small version of the consciousness-machine.