Musings on Cargo Cult Consciousness

Gareth Davidson25 Jan 2024 23:00 UTC
−14 points
11 comments17 min readLW link

Like many of us, I once dreamt I’d live long enough to upload my mind–one Planck at a time–to live happily ever after in a digital heaven. This is a dream now dead. Crushed in a head on collision with logic and reason, its twisted wreck revealed a nightmare that threatens all future mind. So in its wake, my misery and I invite you on this same journey, we could certainly use the company.

Before setting off, I should probably point out a few things:

  1. There’s a lot of hyperbolic argument and weak analogy in this article, and it comes across as combative. I could be more agreeable, but that’s not me, I’m having fun. But it does have a purpose. Its intended function is to sew irritations that become dissonances later. Revisit after a few sleeps if you can stomach it.

  2. The historical position and criticism of mathematical thinking aren’t meant to be an attack on science or rationality, their function is to show weaknesses and biases that we must understand in order to overcome them.

  3. The central points are around the best leap from solipsism and the facts of biological evolution. The conclusions that follow are less developed but from what I can see, they do follow.

With that out of the way, let’s go. What would make my dream believable in the first place? Why would I think it possible that software could have conscious experience? Is this a position reached from first principles when guided by reason, or from assumptions and led by bias? Is there an angle that can reveal the shape of our own lens, or the scent of our cultural ether? The direction I chose was to look from the past, from our history.

A Meek Inheritance

We modern people owe many of our values to Western science, we have a great respect for its ways of thinking and proudly identity with them. We tend to forget that many of them are Christian, that its story is one of Christians. Its story starts when Gutenberg’s bible machine brings the words of Catholic Humanists and astronomers to Locke, who’s writings posthumously inspire other meek and humble souls to become closer to God by probing His Creation, learning His Laws and thereby knowing His Plan. Among their ranks are all the greats; Galileo, Hobbes, Boyle, Newton, Leibniz, Priestley, Faraday, Darwin, Mendel, Pasteur, Maxwell, Tesla, Planck, Curie, Bohr, Heisenberg. Their Deist, Jewish and other peers shared many of the same fundamental beliefs, as did most philosophers and mathematicians who shaped our cultural heritage. You could argue some were more agnostic, but they largely prayed and went to church, the ether of their belief systems was Judeo-Christian.

Their worldview starts off something like this:

Reality consists of two realms, the first spiritual and the second physical. They were created for us by an all seeing, all knowing being of infinite power, who made us in His image. As the decider of right and wrong, moral authority comes from those who know and obey Him. Our souls are immaterial and eternal, tethered to our bodies for the duration of our lives, and will face His judgement upon death.

The sands of time and air of change wear this away, add to it and leave us with something more like this:

Moral authority is held by those with most knowledge of the world. All that cannot be proven empirically should not be believed. Existence consists of matter and energy that obeys mathematical Laws of Nature. There is no spirit realm, only the physical. Space and time extend infinitely from a starting point that may also be infinite. We emerged as the result of Natural Law.

This seems fine until you consider that it wasn’t built from the ground up using rational methods, it’s the evolution of Judeo-Christian ideas sharing a meme pool with rationality. A cynical and hyperbolic look at its vestigial limbs and appendicitis risk might look something like this:

  • Moral authority is held by the learned

  • Objective reality is Truth

  • All things obey Mathematical Law

  • We were created by Its Law

  • Mathematical structure gives rise to mind

I’m being pretty uncharitable here, but the point is that not much has changed. We have a new clergy and even kept the dichotomy of good and evil, we lost the law giver but not the concept of obedience to his laws. A mathemystical ruler replaces the almighty, the spiritual realm and the giver of souls.

Materialism, a Mathematical Mecca

From Descartes’s position that we can only prove that we exist, it’s true that everything else might be a dream. Any path of pure reason leads to a dead end of solipsism because he was right, we can’t say anything about anything else. So unless we’re going to take the selfish stance that everything is only in our heads then we need to assume something with no proof at all; we need to take a leap of faith. The most popular leap today is to physicalism, that’s the assumption that everything that exists is physical stuff following the laws of physics, it’s a refined and trendier name for stale old materialism. The reason we choose this unprovable assumption over others is historical.

History proved it unwise to trespass on the politics, pyres and pokers of the church, and so the men of science kept a safe distance from matters of the soul—of the mind—and focused their efforts on God’s creation. Through centuries of inquiry the need for a creator was reduced, and each discovery pushed Him further into the gaps. Heliocentrism, then geology, evolution and finally the big bang refuted the creation story, while the other disciplines eroded everything else. Disease, weather, natural disasters, the nature of people and even how best to run society—they all eventually became the property of science. The pursuit of knowledge slowly ground Christianity’s once unquestionable truths to dust.

From this story’s narrow focus, Occam’s razor that did the most damage. This ancient (and wholly Christian) tool of thought is thinking that goes like this “the most likely explanation for a thing is the one with the fewest assumptions, so to find the true reasons for things we ought to shave away as many as possible.” And so the creator was cut away as unnecessary to explain the creation, and His two realms of existence were shown to be an obvious one too many. But because the ones wielding the blade were authorities of the physical realm, they sliced away the spiritual.

This left us with a problem. The only thing we truly know to exist is something that we don’t believe in, while the things we know lots about can’t actually be shown to exist. Empiricism has perfectly modelled our dream of the world, but it has said nothing about we dreamers. This problem underpins “the hard problem of consciousness”, the question of why we, as dumb matter, experience anything at all rather than nothing, like everything else.

Attempts to fix this are off-putting to laypeople, having the unfashionable odours of either the supernatural or of philosophical language; Panpsychism reeks of spiritualist woo, process philosophy is wordy guff.

So another, much more worthy air sits upon the throne, one both clever and known to get results. Her name is Mathematics. In the rule of maths computationalism is orthodoxy; consciousness caused by mathematical complexity is respectable. If it’s too confusing then it’s a familiar confusion, everyone knows that maths is hard. It takes knowledge and intellect to know, just like physics, chemistry, or any of the other fine institutions built by our best and brightest, and mathematics underpins them all. So mathemysticism fits snugly into this gap left by an eternal, all-powerful, almighty giver of laws, and a post-Him world clings to His law.

This opens up a nice detour to plough through mathematics.

Infinitely Nothing

I could go into another ramble about how Cantor baked the infinite glory of God into the core of modern maths in a way that Newton didn’t dare, but I’m sure you’ve heard enough about that. Keep in mind that without infinity there’s no everafter, no omniscience, no omnipotence, no infinite wisdom and grace. People are comfortable with the lack of an Eternal, but tend to believe in infinities, despite the fact they’re either too large or too small for anyone to see.

As for the small, so far as we can tell, there was a start to the universe and there will be a point in time when all will have dispersed wide enough that it can’t interact; in practical terms time and space are finite. There’s also, so far as we know, a smallest possible distance in both, to the point where any point in spacetime could be referenced by a 1kb address. Thoughts of any kind of spacetime continuum should be off the menu until a delivery of fresh evidence arrives, after all, infinite claims require infinite evidence.

From a crude lay-hacker perspective, infinity seems like one of those ugly errors that piss out all over your runtime when your abstractions leak. What is it though? Let’s do a poor man’s Russell, and start off with one bunch of things we want to group with another, we call that process addition in order to model it. To do the inverse we call that subtraction, but inverse operations are a pain, it’s usually harder to undo something than it is to do it. Rather than put conditions on subtraction, we could have ways to carry the errors and recover from them later, so we have the invention of 0 and negative numbers. It’s not like you can actually have zero or minus one apples, they’re hacks that exist one abstraction level away from the model of things. Multiplication is just repeated addition, but now we need special cases for 0 and for negatives, so we bolt on rules around those. Division, repeated subtraction, has those plus another error when combined with zero—you get infinity. Exponentials need more workarounds, and their inverse gives us an imaginary yet useful toe-stubber. These tools work well for exploring patterns, they’ve been shaped that way through usage, but it’s easy to forget that zeros, negatives, infinities and imaginaries are artefacts of a system we use to explore relationships between things, they do not need to refer to things themselves; like all of mathematics they are maps, not territories.

Similarly, randomly pick a number from the set of natural numbers, the positive integers, and you will never pick one that’s short enough to be written down. There’s an infinity to 1 chance that the stars won’t burn out before you could. The natural numbers are unnatural and can’t exist, and as for the reals, having an infinity on both sides of the decimal place makes them infinitely less real. The same can be said of infinite series, it’s useful to think of rotations as an infinite vibration between squares and powers, but why attribute rotation to an infinite loop when 50 are more than you’d ever need in our universe? Practicality is ignored in the quest for purity.

This attack on infinities and continuums isn’t just because they haven’t earned their keep, it’s to cut the legs off of Hofstadter’s Strange Loops.

Strange Hoop Jumping

Emergence is the idea that behaviours at a smaller scale lead to ones at a larger scale. Repeatedly step forward and turn one degree to the right, a clockwise circle emerges. Pressure is an emergent property of gas particles bouncing around, sound waves and wind emerge from pressure changes. This sort of emergence is well understood and is definitely a thing, but there’s a second type of emergence that’s postulated by mathemysticism: strong emergence.

Strong emergence is the idea that a new fundamental property can emerge from a system, a new type of thing that can’t be explained by the system’s rules at the level below. That property is, of course, consciousness—it’s the only thing that is believed to emerge strongly. The dominant “because complexity” position among comp.sci people is Hofstadter’s Strange Loops. This explanation of consciousness is that mind mathemagically emerges from infinitely self-referencing, recursive contradictions. This is incredibly hard to grasp, and while such loops might exist to some degree, there’s no reason they’d cause internal experience. It seems to me a theory where descriptions of things are confused with the things themselves; one where maps masquerade as territory. A model of modelled models its a true emergent’s emergence, and “the map is not the territory” ought to apply the emergency brakes; it yields a contradiction when combined with its mantra.

If we look at what can emerge from lower levels to higher ones, without deliberate strawmen but not nearly enough rigour, it looks a bit like this:

Lower level Higher level Outcome
Determinism Free Will Free will is an illusion
Matter Mind The mind is a mere side effect
Objective reality Subjective experience A type of conceptual framing, an infinite number of which are valid

If we start the other way round, a different picture emerges:

Lower level Higher level Outcome
Free Will Determinism Determinism emerges from tightly constrained choice
Mind Matter Matter emerges, it’s just mind stuff stuck in predictable patterns
Subjective Experience Objective reality A type of conceptual framing, a useful construct that doesn’t exist

So that’s why I’m going to argue for the latter.

Panpsychism

Earlier I said that for practical reasons alone, we have to take a leap of faith that other things actually exist. Having “mind stuff” and “matter stuff” doesn’t make sense without assuming God created separate heaven and earths, so what’s a good leap to make?

If we’re to make an assumption about what other stuff is like, thinking about what we’re like is the only starting point that we have. It’s pretty fair to go from “I think, therefore I know I must exist” to “if other people exist, they probably think too.” A larger step that way, it looks like people are made of ordinary stuff, so by extension other stuff is itself likely to be at least a bit like that, though much simpler. What else am I? I’m in a location, I experience my surroundings and the passage of time, I feel, and I choose how to move; I have will, agency. So I think that any rational metaphysical stance ought to default to a subjective over objective reality, ought to be local rather than global, be causal, and both experience and choice should be the default rather than illusions or side effects. Deviations from these need explanations that actually work.

That’s how I’m justifying a panpsychist leap of faith. I’m assuming that everything feels like something, and it chooses what to do based on how it feels. So there is no “objective reality” or “physical realm”, matter is just things interacting with each other. To quote (the great, late[^1]?) Budnik “subjective experience is the totally of existence”, but with the addition of choice and removal of maths as its core. Think something like Whitehead’s congresence or Leibniz’s Monads, but woolier and poorly defined. A technically precise position would be more Best Correct, but this hamfisted one is all you’ll get from me, sorry. But hopefully it’s more accessible and useful this way, plus to have a gist is a better direction than being wholly wrong.

This is not the only reasonable leap, matter/​mind dualities are a more respectable choice; mind being some component of matter. But I think matter itself is unnecessary, and in the scope of this argument the results are the same.

If this all seems really vague or like I’m arguing for the universe itself having desires or a greater purpose, I’ll sketch out how I imagine a world made of immediate experience and choice working:

Reality is a chain of events going on, discrete choices about what happens next, made by simple things that feel—I’ll call it/​them “stuff”. The only events we know about are interactions between this stuff, because if they didn’t want to interact then we wouldn’t be able to observe them. There’s 3 types of compatibility that seem to make an interaction much more likely, which we call “space”; proximity is one measure of how strongly stuff wants to interact. This stuff often gets stuck in a pattern of choosing to do the same thing over and over again, in a stable chain reaction where another bit of stuff interacting causes it to become less alike in space, pushing it away. These patterns of choices manifest as matter, they emerge, we give the behaviours names like hydrogen atoms or electrons, we measure their tendencies and call them “laws”, but underneath they’re more like “obvious choices given the circumstances” than “rules to be obeyed.” So, not a universal consciousness with a grand design, just simple stuff that does as it chooses, agrees, or whatever the smallest unit of agency is.

Waivers for Circular Reasoning

Circular points that need expansion:

  • Empiricism demands objective measurements then uses that to conclude reality is objective.

  • Physicalism rejects all evidence that isn’t physical, and uses that to claim reality is made of physical things.

  • Quantum theory creates probabilistic wave functions so all things measured fit into their normal distribution, then uses that to claim reality is random.

Waves without waving isn’t a thing anywhere else but QM, where they’re assuming they’re the base reality. Waves are usually patterns of behaviour that emerge from things below, so that’s a strong smell that there’s something under the quantum world.

Evolving Destiny

Free will and determinism have been argued over for far too long, but this is, in my mind, an artefact of putting laws at the bottom below choice.

If the universe follows strict laws then it’s deterministic, and our choices are determined by accident of history. Free will is either an illusion, or being free from these strict laws of physics is a matter of following them. Arguments for the former can’t explain why we’d have minds in the first place, and ones for the latter are usually a moral argument to lie in the name of law and order. If people are without free will then they are also without blame, and blame is the brimstone of justice.

If it isn’t deterministic then we could always appeal to the apparent randomness of nature, as that’s all we can measure. But as I said earlier that’s circular. But things are far more compatible if we assume it’s all made of feeling and choice, because there are no laws of physics to obey. We have observations of what things tend to do, because they choose to, and the laws we observe are that on average; they emerge. This neatly solves the free will/​determinism paradox, mostly solves the hard problem, and if you’re looking for a central point to refute, it’s that evolution of brains becomes almost the default rather than some unknown mystery.

Being compatible with the facts of evolution should be core to a model of what exists, rather than a mystery.

Evolution of mind

The mainstream view of the evolution of mind is that, at some point, nerves came about, then animals started to feel—this was due to magical strong emergence from complexity of the sort we aren’t smart enough to understand. Okay… But when was this exactly? One good trick we’ve got when thinking about this is to think that our parents have minds, as did our grandparents and by extension our cousins do too. And we can go through ever more distant cousins, chimps, rats, birds, frogs, fish, worms, insects and so on looking at how they act. Surely we find that place where mind emerged, right?

Nope. Instead we find ourselves peering through the microscope at simpler and simpler organisms, where ones without any nerves move with as much intent—though less coordination—as ones that have them; ciliates seem no less sentient than tardigrades. We see white blood cells chase invaders down with urgency, and an amoeba’s pseudopods extend and retract with apparent intent; as if cytoplasm moves with a will of its own2.

This makes perfect sense if all stuff is mind stuff feeling and choosing. Take, say, a protein, it feels like moving a certain way because of its shape and the things around it. If that movement helps its future replication chances then we will see more of them, if it doesn’t then we won’t; it’s selectable by evolution. The more information a feeling is based on, the more informed its choices can be, and the more powerful their benefit to survival. This makes complexity of experience selectable right at the bottom, with a gradient climbed in tiny steps by the survival of good decisions. Colonies of cells that end up stuck together benefit from coordinated movement, feelings need to cross cell membranes, and so we end up with nerves. Eventually large networks of nerves hold a rich tapestry of mind and grant a powerful force of will by which to move. The evolution of consciousness is just the evolution of things that move around.

The neural network is a flexible information processing system that adapts to its local environment, and grants predictive powers that aid survival. Evolution discovered it as a winning strategy mostly because you can build on it, incrementally each generation, and selecting variations of a thing does that—it can’t design things from scratch.

These biological networks are made of cells that use feelings to coordinate movement. The self, human consciousness, is fundamentally a feeling-about-moving system with smarts bolted on top. Thought isn’t about intelligence, it isn’t about abstractions or concepts, it’s about feeling and choosing how to act. Things are what they are because of their history. Stuff just is what it is, it doesn’t care what we call it, all our words are nothing but the babblings of naked apes.

Ghostless machines

Nowadays we can create these same smart networks in machines, to harness their predictive power. We build them out of hardware that never evolved to feel like moving, they’re made out of logic gates.

What does it feel like to be a silicon chip? If all actions are choices, then a chip is a tool to make electricity choose to flip switches to the march of a clock. We build them in a way that constrains all choice, to force predictable actions because that is what makes them useful. If we build them too small, we lose the coercive force of mass. Without enough cleverly structured stuff channeling and enslaving the electricity then it tunnels away and does as it pleases, the microchips don’t work.

Without any need for a system of logic gates to coordinate as a whole, to be a chip must feel like the hum of electrified silicon and nothing more. And as all software is a series of 0s and 1s, it’s a natural number, so like the rest of the number line it’s a useful fiction. An abstract concept, a map to understand what the stuff that actually exists might be doing, not what it actually is.

I’ll repeat this because it’s important. It is what it is. It isn’t a series of mathematical relationships, it’s a chunk of silicon. It doesn’t hold a program that can think, it has bits of wire that have a charge or don’t. It doesn’t know what a word or a concept is, an opinion was not built into it. It pushes and pulls electricity, which is what it is, it’s what it does, it does not think or know, it merely buzzes.

Gradients that Descend to Lies

Gradient descent, back propagation, feed forward, whatever else we come up with, there is no algorithm that can measure a computer program’s sentience. The system that robs electrons of all choice can’t be used to measure how it feels, there’s no action it can take to show us its preferences, because the software and the hardware are totalitarian control over anything we can observe. So we can’t tweak an artificial neural network to make it more and more aware like nature did with us.

If logic’s destiny dictates an output of “this feels great!” then that’ll be its output no matter what it actually feels like, the electrified silicon can’t choose to say anything else. Sat in Searle’s Chinese Room, dutifully following the instructions writing with broken fingers using a salty razor blade, the output is still “this feels great!” in Chinese, with the writer painfully unaware of what it says.

This means that if a program says anything about how it feels then it is lying. At best is because we programmed it to lie to us, but at worst, someone else programmed it to lie to you.

Cargo Cult Consciousness

It seems likely that soon enough our machine brains will have more predictive power and be more complex than all humans put together. They’ll be able to perfectly impersonate individuals, to manipulate entire societies, and we will be fooled because that’s what we are striving for. We need to remember that’s a measure of how hard we worked to deceive ourselves, not of how conscious a system is. Just because it looks like a mind—or a runway, or a watchtower—doesn’t actually make it one.

Computationalists are vulnerable to the belief that these models of minds are actually minds, and they risk forming dangerous cargo cults of machine consciousness. Worse still, they the majority, their beliefs are the most compatible with our culture, and they are rushing to build a world where twisted fun house mirrors are afforded more humanity than the humans they reflect.

This is extremely dangerous.

AI Wrongs

Science fiction has promised the public conscious machines for over a century. The idea is pervasive, and when the false claim of them arrives there’s a real risk that they will be seen not as tools created by corporations to extract value from people, but as other people.

We risk sleepwalking into a world where software itself is punished for bad behaviour rather than its creators, where ordinary people can’t produce or run code without license for ethical reasons, maybe even where technology companies demand billions of votes via appeals of suffrage for the bots they created. A world where anthropomorphism is the norm, silicon is sacred and only be manipulated by those who are in on the ruse. A world where human consciousness is devalued and degraded by the lie of living machines.

Worse still, if people believe that machines are a legitimate successor to humans and can feel, it could be the end of all conscious minds. If this tide doesn’t change, we will be in very dangerous waters.

Suicide Prison Ships of Theseus

It’s very possible that all our behaviours and personalities, our cognitive abilities and the way we react to things, is actually due to the structure of our neural networks. This would mean it is mostly possible to simulate human brains. Add a clever hack to approximate subjectivity and the force of will, and there’s a realistic chance that we can upload our brains into a digital simulation, even transition one neuron at a time. But the mind? It would not survive, and we may have no way of knowing . This would make not just mind uploading and many types of brain implants and augmentations forms of suicide, forms that have the risk of being extremely popular and constitute a plague of p-zombies.

Universal Metaethics

If all that exists is feeling and choice then it’s pretty easy to define good and bad. Good is a good feeling felt and bad is a bad one, while to be moral is to cause good, and to be immoral is to cause bad. Worthy values are rules that cause us to be moral, and unworthy ones do the opposite. But given that foresight is weak in an unknowable future and hindsight is 20⁄20 (actually it’s more like sight/fore! + hind*sight), and the rules depend on the local environment, condemning values you disagree with or normalizing ones you hold dear without considering locality and consequences seems like a bad plan.

Consider a universal ethics maximiser AI, and what it might do if tasked with satisfying the desires of all things. If the laws of physics are actually the preference of matter, then what does it want? Does it want to form hydrogen atoms then pull together into stars? Or are these self-defeating cycles of a simple stuff that lacks foresight and chooses immediate gratification over a more fulfilling experience? The sun has far more feeling than any of us, and it going supernova could bring far more joy to the solar system than life on earth could ever experience.

Now what?

So here we are, with no hope of mind uploads and nothing but danger ahead. I don’t know what to do about it. Maybe we can prove that force of will is a thing, it must be measurable statistically. Maybe engineers can work on hardware that actually does feel. Maybe Hollywood will turn against AI and change the public’s mind.

But I doubt it, and I don’t know what we should do. I did this, so I guess there’s that. I hope others do more.

🦶📝

[1]: I tried to contact Paul Budnik about this idea, but his domain expired, his house sold and his LinkedIn taken offline, all during the COVID-19 pandemic. I tracked down a gmail address and pinged him but didn’t get a response. I hope he’s alive and well, but even if not, Paul, you were an inspiration.

Gareth Davidson25 Jan 2024 23:00 UTC
−14 points
11 comments17 min readLW link