Programmer.
MinusGix
Karma: 108
2b/2c. I think I would say that we should want a tyranny of the present to the extent that is in our values upon reflection. If, for example, Rome still existed and took over the world, their CEV should depend on their ethics and population. I think it would still be a very good utopia, but it may also have things we dislike.
Other considerations, like nearby Everett branches… well they don’t exist in this branch? I would endorse game theoretical cooperation with them, but I’m skeptical of any more automatic cooperation than what we already have. That is, this sort of fairness is a part of our values, and CEV (if not adversarially hacked) should represent those already? I don’t think this would end up in a tyranny anything like the usual form of the word if we’re actually implementing CEV. We have values for people being able to change and adjust over time, and so those are in the CEV. There may very well be limits to how far we want humanity to change in general, but that’s perfectly allowed to be in our values. Like, as a specific example, some have said that they think global status games will be vastly important in the far future and thus a zero-sum resource. I find it decently likely that an AGI implementing CEV would discourage such, because humans wouldn’t endorse it on reflection, even if it is a plausible default outcome.Like, essentially my view is: Optimize our-branches’ humanity’s values as hard as possible, this contains desires for other people’s values to be satisfied, and thus they’re represented. Other forms of fairness to things we aren’t completely a fans of can be bargained for (locally, or acausally between branches/whatever).
So that’s my argument against the tyranny and Everett branches part. I’m less skeptical of considering whether to include the recently dead, but I also don’t have a great theory of how to weight them. Those about to be born wouldn’t have a notable effect on CEV, I’d believe.
The option you suggest in #3 is nice, though I think it runs some risks of being dominated or notably influenced by “humans in other very odd branches”, and so we’re outweighed by them despite them not locally existing. I think it is less that you want a human predicate, and more of a “human who has values compatible with this local branch”. This is part of why I advocate just bargaining between branches: if the humans in an AGI-made New Rome want us to instantiate their constructed friendly/restricted AGI Gods locally to proselytize, they can trade for it rather than that faction being automatically divvied out a star by our AGI’s CEV.
“Human who has values compatible with this local branch” feels weak as a definition, arbitrary, but I’m not sure we can do better than that. I imagine we’d even have weightings, because we likely legitimately value baby’s in special ways that don’t entail maxing out reward centers or boosting them to megaminds soon after birth, we have preferences about that. Then of course there’s minds that are sortof humanish, which is why you’d have a weighting.(This is kinda rambly, but I do think a lot of this can be avoided with just plain CEV because I think most people on reflection would end up with “reevaluate whether the deal was fair with reflection and then adjust the deal and reference class based on that”.)
I agree Grothendieck is fascinating but I mostly just see him as interesting in different ways than von Neumann. von Neumann is often focused on because his subjects are areas that are relevant to either LessWrong’s focuses or (for the cloning posts) that the subjects he was skilled at and polymath capabilities would help with alignment.
I define rationality as “more in line with your overall values”. There are problems here, because people do profess social values that they don’t really hold (in some sense), but roughly it is what they would reflect on and come up with.
Someone could value the short-term more than the long-term, but I think that most don’t. I’m unsure if this is a side-effect of Christianity-influenced morality or just a strong tendency of human thought.Locally optimal is probably the correct framing, but that it is irrational relative to whatever idealized values the individual would have. Just like how a hacky approximation of a Chess engine is irrational relative to Stockfish—they both can be roughly considered to have the same goal, just one has various heuristics and short-term thinking that hampers it. These heuristics can be essential, as it runs with less processing power, but in the human mind they can be trained and tuned.
Though I do agree that smoking isn’t always irrational: I would say smoking is irrational for the supermajority of human minds, however. The social negativity around smoking may be what influences them primarily, but I’d consider that just another fragment of being irrational— >90% of them would have a value for their health, but they are varying levels of poor at weighting the costs and the social negativity response is easier for the mind to emulate. Especially since they might see people walking around them while they’re out taking a cigarette. (Of course, the social approval is some part of a real value too; though people have preferences about which social values they give into)
An important question here is “what is the point of being ‘more real’?”. Does having a higher measure give you a better acausal bargaining position? Do you terminally value more realness? Less vulnerable to catastrophes? Wanting to make sure your values are optimized harder?
I consider these, except for the terminal sense, to be rather weak as far as motivations go.
Acausal Bargaining: Imagine a bunch of nearby universes with instances of ‘you’. They all have variations, some very similar, others with directions that seem a bit strange to the others. Still identifiably ‘you’ by a human notion of identity. Some of them became researchers, others investors, a few artists, writers, and a handful of CEOs.
You can model these as being variations on some shared utility function: where is shared, and is the individual utility function. Some of them are more social, others cynical, and so on. A believable amount of human variation that won’t necessarily converge to the same utility function on reflection (but quite close).
For a human, losing memories so that you are more real is akin to each branch chopping off the . They lose memories of a wonderful party which changed their opinion of them, they no longer remember the horrors of a war, and so on.
Everyone may do the simple ask of losing all their minor memories which has no effect on the utility function, but then if you want more bargaining power, do you continue? The hope is that this would make your coalition easier to locate, to be more visible in “logical sight”. That this increased bargaining power would thus ensure that, at the least, your important shared values are optimized harder than they could if you were a disparate group of branches.
I think this is sometimes correct, but often not.
From a simple computationalist perspective, increasing the measure of the ‘overall you’ is of little matter. The part that bargains, your rough algorithm and your utility function, is already shared: is shared among all your instances already, some of you just have considerations that pull in other directions (). This is the same core idea of the FDT explanation of why people should vote: because, despite not being clones of you, there is a group of people that share similar reasoning as you. Getting rid of your memories in the voting case does not help you!For the Acausal Bargaining case, there is presumably some value in being simpler. But, that means more likely that you should bargain ‘nearby’ to present a computationally cheaper value function ‘far away’. So, similar to forgetting, where you appear as if having some shared utility function, but without actually forgetting—and thus being able to optimize for in your local universe. As well, the bargained utility function presented far away (less logical sight to your cluster of universes) is unlikely to be the same as .
So, overall, my argument would be that forgetting does give you more realness. If at 7:59AM, a large chunk of universes decide to replace part of their algorithm with a specific coordinated one (like removing a memory) then that algorithm is instantiated across more universes. But, that from a decision-theoretic perspective, I don’t think that matters too much? You already share the important decision theoretic parts, even if the whole algorithm is not shared.
From a human perspective we may care about this as a value of wanting to ‘exist more’ in some sense. I think this is a reasonable enough value to have, but that it is oft satisfied by considering the sharing of decision methods and 99.99% of personality is enough.
My main question of whether this is useful beyond a terminal value for existing more is about quantum immortality—of which I am more uncertain about.
Beliefs and predictions that influence wants may be false or miscalibrated, but the feeling itself, the want itself, just is what it is, the same way sensations of hunger or heat just are what they are.
I think this may be part of the disconnect between me and the article. I often view the short jolt preferences (that you get from seeing an ice-cream shop) as heuristics, as effectively predictions paired with some simpler preference for “sweet things that make me feel all homey and nice”. These heuristics can be trained to know how to weigh the costs, though I agree just having a “that’s irrational” / “that’s dumb” is a poor approach to it. Other preferences, like “I prefer these people to be happy” are not short-jolts but rather thought about and endorsed values that would take quite a bit more to shift—but are also significantly influenced by beliefs too.
Other values like “I enjoy this aesthetic” seem more central to your argument than short-jolts or considered values.
This is why you could view a smoker’s preference for another cigarette as irrational: the ‘core want’ is just a simple preference for the general feel of smoking a cigarette, but the short-jolt preference has the added prediction of “and this will be good to do”. But that added prediction is false and inconsistent with everything they know. The usual statement of “you would regret this in the future”. Unfortunately, the short-jolt preference often has enough strength to get past the other preferences, which is why you want to downweight it.
So, I agree that there’s various preferences that having them is disentangled from whether you’re rational or not, but that I also think most preferences are quite entangled with predictions about reality.
“inconsistent preferences” only makes sense if you presume you’re a monolithic entity, or believe your “parts” need to all be in full agreement all the time… which I think very badly misunderstands how human brains work.
I agree that humans can’t manage this, but it does still make sense for a non-monolithic entity—You’d take there being an inconsistency as a sign that there’s a problem, which is what people tend to do, even if ti can’t be fixed.
Finally, the speed at which you communicate vibing means you’re communicating almost purely from System 1, expressing your actual felt beliefs. It makes deception both of yourself and others much harder. Its much more likely to reveal your true colors. This allows it to act as a values screening mechanism as well.
I’m personally skeptical of this. I’ve found I’m far more likely to lie than I’d endorse when vibing. Saying “sure I’d be happy to join you on X event” when it is clear with some thought that I’d end up disliking it. Or exaggerating stories because it fits with the vibe.
I view System-1 as less concerned with truth here, it is the one that is more likely to produce a fake-argument in response to a suggested problem. More likely to play social games regardless of if they make sense.
I agree that it is easy to automatically lump the two concepts together.
I think another important part of this is that there are limited methods for most consumers to coordinate against companies to lower their prices. There’s shopping elsewhere, leaving a bad review, or moral outrage. The last may have a chance of blowing up socially, such as becoming a boycott (but boycotts are often considered ineffective), or it may encourage the government to step in. In our current environment, the government often operates as the coordination method to punish companies for behaving in ways that people don’t want. In a much more libertarian society we would want this replaced with other methods, so that consumers can make it harder to put themselves in a prisoner’s dilemma or stag hunt against each other.
If we had common organizations for more mild coordination than the state interfering, then I believe this would improve the default mentality because there would be more options.
It has also led to many shifts in power between groups based on how well they exploit reality. From hunter-gatherers to agriculture, to grand armies spreading an empire, to ideologies changing the fates of entire countries, and to economic & nuclear super-powers making complex treaties.
This reply is perhaps a bit too long, oops.
Having a body that does things is part of your values and is easily described in them. I don’t see deontology or virtue ethics as giving any more fundamentally adequate solution to this (beyond the trivial ‘define a deontological rule about …‘, or ‘it is virtuous to do interesting things yourself’, but why not just do that with consequentialism?).
My attempt at interpreting what you mean is that you’re drawing a distinction between morality about world-states vs. morality about process, internal details, experiencing it, ‘yourself’. To give them names, “global”-values (you just want them Done) & “indexical”/’local”-values (preferences about your experiences, what you do, etc.) Global would be reducing suffering, avoiding heat death and whatnot. Local would be that you want to learn physics from the ground up and try to figure out XYZ interesting problem as a challenge by yourself, that you would like to write a book rather than having an AI do it for you, and so on.I would say that, yes, for Global you should/would have an amorphous blob that doesn’t necessarily care about the process. That’s your (possibly non-sentient) AGI designing a utopia while you run around doing interesting Local things. Yet I don’t see why you think only Global is naturally described in consequentialism.
I intrinsically value having solved hard problems—or rather, I value feeling like I’ve solved hard problems, which is part of overall self-respect, and I also value realness to varying degrees. That I’ve actually done the thing, rather than taken a cocktail of exotic chemicals. We could frame this in a deontological & virtue ethics sense: I have a rule about realness, I want my experiences to be real. / I find it virtuous to solve hard problems, even if in a post-singularity world.
But do I really have a rule about realness? Uh, sort-of? I’d be fine to play a simulation where I forget about the AGI world and am in some fake-scifi game world and solve hard problems. In reality, my value has a lot more edge-cases that will be explored than many deontological rules prefer. My real value isn’t really a rule, it is just sometimes easy to describe it that way. Similar to how “do not lie” or “do not kill” is usually not a true rule.
Like, we could describe my actual value here as a rule, but seems actually more alien to the human mind. My actual value for realness is some complicated function of many aspects of my life, preferences, current mood to some degree, second-order preferences, and so on. Describing that as a rule is extremely reductive.
And ‘realness’ is not adequately described as a complete virtue either. I don’t always prefer realness: if playing a first-person shooter game, I prefer that my enemies are not experiencing realistic levels of pain! So there are intricate trade-offs here as I continue to examine my own values.
Another aspect I’m objecting to mentally when I try to apply those stances is that there’s two ways of interpreting deontology & virtue ethics that I think are common on LW. You can treat them as actual philosophical alternatives to consequentialism, like following the rule “do not lie”. Or you can treat them as essentially fancy words for deontology=>”strong prior for this rule being generally correct and also a good coordination point” and virtue ethics=>”acting according to a good Virtue consistently as a coordination scheme/culture modification scheme and/or because you also think that Virtue is itself a Good”.
Like, there’s a difference between talking about something using the language commonly associated with deontology and actually practicing deontology. I think conflating the two is unfortunate.The overaching argument here is that consequentialism properly captures a human’s values, and that you can use the basic language of “I keep my word” (deontology flavored) or “I enjoy solving hard problems because they are good to solve” (virtue ethics flavored) without actually operating within those moral theories. You would have the ability to unfold these into the consequentialist statements of whatever form you prefer.
In your reply to cubefox, “respect this person’s wishes” is not a deontological rule. Well, it could be, but I expect your actual values don’t fulfill that. Just because your native internal language suggestively calls it that, doesn’t mean you should shoehorn it into the category of rule!
“play with this toy” still strikes me as natively a heuristic/approximation to the goal of “do things I enjoy”. The interlinking parts of my brain that decided to bring that forward is good at its job, but also dumb because it doesn’t do any higher order thinking. I follow that heuristic only because I expect to enjoy it—the heuristic providing that information. If I had another part of my consideration that pushed me towards considering whether that is a good plan, I might realize that I haven’t actually enjoyed playing with a teddy bear in years despite still feeling nostalgia for that. I’m not sure I see the gap between consequentialism and this. I don’t have the brain capacity to consider every impulse I get, but I do want to consider agents other than AIXI to be a consequentialist.
I think there’s a space in there for a theory of minds, but I expect it would be more mechanistic or descriptive rather than a moral theory. Ala shard theory.Or, alternatively, even if you don’t buy my view that the majority of my heuristics can be cast as approximations of consequentialist propositions, then deontology/virtue ethics are not natural theories either by your descriptions. They miss a lot of complexity even within their usual remit.
I think there’s two parts of the argument here:
Issues of expressing our values in a consequentialist form
Whether or not consequentialism is the ideal method for humans
The first I consider not a major problem. Mountain climbing is not what you can put into the slot to maximize, but you do put happiness/interest/variety/realness/etc. into that slot. This then falls back into questions of “what are our values”. Consequentialism provides an easy answer here: mountain climbing is preferable along important axes to sitting inside today. This isn’t always entirely clear to us, we don’t always think natively in terms of consequentialism, but I disagree with:
There are many reasons to do things—not everything has to be justified by consequences.
We just don’t usually think in terms of consequences, we think in terms of the emotional feeling of “going mountain climbing would be fun”. This is a heuristic, but is ultimately about consequences: that we would enjoy the outcome of mountain climbing better than the alternatives immediately available to our thoughts.
This segues into the second part. Is consequentialism what we should be considering? There’s been posts about this before, of whether our values are actually best represented in the consequentialist framework.
For mountain climbing, despite the heuristic of “I feel like mountain climbing today”, if I learned that I would actually enjoy going running for an hour then heading back home more, then I would do that instead. When I’m playing with some project, part of that is driven by in-the-moment desires, but ultimately from a sense that this would be an enjoyable route.This is part of why I view the consequentialist lens as a natural extension of most if not all of our heuristics.
An agent that really wanted to go in circles doesn’t necessarily have to stop, but for humans we do care about that.
There’s certainly a possible better language/formalization to talk about agents that are mixes of consequentialist parts and non-consequentialist parts, which would be useful for describing humans, but I also am skeptical about your arguments for non-consequentialist elements of human desires.
If I value a thing at one period of life and turn away from it later, I have not discovered something about my values. My values have changed. In the case of the teenager we call this process “maturing”. Wine maturing in a barrel is not becoming what it always was, but simply becoming, according to how the winemaker conducts the process.
Your values change according to the process of reflection—the grapes mature into wine through fun chemical reactions.
From what you wrote, it feels like you are mostly considering your ‘first-order values’. However, you have an updating process that you also have values about. Like that I wouldn’t respect simple mind control that alters my first-order values, because my values consider mind-control as disallowed. Similar to why I wouldn’t take a very potent drug even if I know my first-order values would rank the feeling very highly, because I don’t endorse that specific sort of change.I have never eaten escamoles. If I try them, what I will discover is what they are like to eat. If I like them, did I always like them? That is an unheard-falling-trees question.
Then we should split the question. Do you have a value for escamoles specifically before eating them? No. Do you have a system of thought (of updating your values) that would ~always result in liking escamoles? Well, no in full generality. You might end up with some disease that affects your tastebuds permanently. But in some reasonably large class of normal scenarios, your values would consistently update in a way that would end up liking escamoles were you to ever eat them. (But really, the value for escamoles is more instrumental of a value for [insert escamole flavor, texture, etc.] here, that the escamoles are learned to be a good instance of.)
What johnwentworth mentions would then be the question of “Would this approved process of updating my values converge to anything”; or tend to in some reasonable reference class; or at least have some guaranteed properties that aren’t freely varying. I don’t think he is arguing that the values are necessarily fixed and always persistent (I certainly don’t always handle my values according to my professed beliefs about how I should updatethem), but that they’re constrained. That the brain also models them as reasonably constrained, and that you can learn important properties of them.
Thank you!
Is there a way to get an article’s raw or original content?
My goal is mostly to put articles in some area (ex: singular learning theory) into a tool like Google’s NotebookLM to then ask quick questions about.
Google’s own conversion of HTML to text works fine for most content, excepting math. A division may turn into p ( w | D n ) = p ( D n | w ) φ ( w ) p ( D n ), becoming incorrect.I can always just grab the article’s HTML content (or use the GraphQL api for that), but HTMLified MathJax notation is very, uh, verbose. I could probably do some massaging of the data and then an LLM to translate it back into the more typical markdown $ delimited syntax, but I’m hopeful that there’s some existing method to avoid that entirely.
I’d be interested in an article looking at whether the FDA is better at regulating food safety. I do expect food is an easier area, because erring on the side of caution doesn’t really lose you much — most food products have close substitutes. If there’s some low but not extremely low risk of a chemical in a food being bad for you, then the FDA can more easily deny approval without significant consequences: Medicine has more outsized effects if you are slow to approve usage.
Yet, perhaps this has led to reduced variety in food choices? I notice less generic or lesser-known food and beverage brands relative to a decade ago, though I haven’t verified whether my that background belief is accurate. I’d be curious also for an investigation in such an article about the extent of the barriers to designing a new food product; especially food products that aren’t doing anything new, purely a mixture of ingredients already considered safe (or at least, considered allowed). Would there be more variety? Or notably cheaper food?
I see this as occurring with various pieces of Infrabayesianism, like Diffractor’s UDT posts. They’re dense enough mathematically (hitting the target) which makes them challenging to read… and then also challenging to discuss. There are fewer comments even from the people who read the entire post because they don’t feel competent enough to make useful commentary (with some truth behind that feeling); the silence also further making commentation harder. At least that’s what I’ve noticed in myself, even though I enjoy & upvote those posts.
Less attention seems natural because of specialization into cognitive niches, not everyone has read all the details of SAEs, or knows all the mathematics referenced in certain agent foundations posts. But it does still make it a problem in socially incentivizing good research.
I don’t know if there are any great solutions. More up-weighting for research-level posts? I view the distillation idea from a ~year ago as helping with drawing attention towards strong (but dense) posts, but it appeared to die down. Try to revive that more?
I draw the opposite conclusion from this: the fact that the decision theory posts seem to work on the basis of a computationalist theory of identity makes me think worse of the decision-theory posts.
Why? If I try to guess, I’d point at not often considering indexicality as a consideration, merely thinking of it as having a single utility function which simplifies coordination. (But still, a lot of decision theory doesn’t need to take into account indexicality..)
I see the decision theory posts as less as giving new intuitions, and more breaking old ones that are ill-adapted, though that’s partially framing/semantics.Can you link to some of these? I do not recall seeing anything like this here.
I’ll try to find some, but they’re more likely to be side parts of comment chains rather than posts, which does make them more challenging to search for. I doubt they’re as in-depth as we’d like, I think there is work done there, even if I do think the assumption of QM not mattering much is likely.
The basic idea is what would it give you? If the brain uses it for a random component, why can’t that be replaced with something pseudorandom? Which is fine from an angle of not seeing determinism as a problem. If the brain utilizes entangled atoms/neurons/whatever for efficiency, why can’t those be replaced with another method — possibly impractically inefficient? Does the brain functionally depend on an arbitrary precision Real for a calculation, why would it, and what would be the matter if it was cut off to N digits?
Somewhat Eliezer’s Comment Here and some of the other pieces
Does davidad’s uploading moonshot work which has more specifics about what davidad thinks is relevant to uploading
With this as also a good article to read as a reply
Scott Aaronson on Free Will About more than just FW, though he’s arguing against the LW position, but I don’t consider it a strong argument, see the comments for a bit of discussion.
Quotes and Notes on Scott Aaronson’s has more positive leaning commentary
There’s certainly more, but finding specific comments I’ve read over the years is a challenge.
Everything was determined in the initial configuration of quantum waveforms in the distant past of your lightcone. The experience of time and change is just a side-effect of your embeddedness in this giant static many-dimensional universe.”
I’m not sure I understand the distinction. Even if the true universe is a bunch of freeze-frame slices, time and change still functionally act the same. Given that I don’t remember random nonsense in my past, there’s some form of selection about which freeze-frames are constructed. Or, rather, with differing measure. Thus most of my ‘future’ measure is concentrated on freeze-frames that are consistent with what I’ve observed, as that has held true in the past.
Like, what you seem to be saying is Timeless Physics, of which I’d agree more with this statement:
An unchanging quantum mist hangs over the configuration space, not churning, not flowing. But the mist has internal structure, internal relations; and these contain time implicitly. The dynamics of physics—falling apples and rotating galaxies—is now embodied within the unchanging mist in the unchanging configuration space.
So I’d agree that computation only makes sense with some notion of time. That there has to be some way it is being stepped forward. (To me this is an argument in favor of not privileging spatial position in the common teleportation example, but we’ve seemed to move down a level to whether the brain can be implemented at all)
(bits about CEV) conceptually incoherent
I misworded what I say, sorry. I more meant that you consider it to say/imply nothing meaningful, but you can certainly still argue against it (such as arguing that it isn’t coherent).
I think it would be non-physicalist if (to slightly modify the analogy, for illustrative purposes) you say that a computer program I run on my laptop can be identified with the Python code it implements, because it is not actually what happens.
I would say the that the computer program running can be considered as an implementation of the abstract python code. I agree that this model is missing details. Such as the exact behavior of the transistor, how fast it switches, the exact positions of the atoms, etcetera. That is dependent on the mind considering it, I agree. The cosmic ray event would make so it is no longer an implementation of the abstract python program. You could expand the consideration to include more of the universe. Just as you could expand your model to consider the computer program as an implementation of the python program with some constraints: that if this specific transistor gets flipped one too many times it will fry, that there’s a slight possibility of a race condition that we didn’t consider at all in our abstract implementation, there’s a limit to the speed and heat it can operate at, a cosmic ray could come from these areas of space and hit it with 0.0x% probability thus disrupting functionality...
It still seems quite reasonable to say it is an implementation of the python program. I’m open to the argument that there isn’t a completely natural privileged point of consideration from which the computer is implementing the same pattern as another computer, and that the pattern is this python program. But as I said before, even if this is ultimately some amount of purely subjective, it still seems to capture quite a lot of the possible ideas?
Like in mathematics, I can have an abstract implementation of a sorting algorithm and prove that a python program for a more complicated algorithm (bubblesort, whatever) is equivalent. This is missing a lot of details, but that same sort of move is what I’m gesturing at.
It is merely part of a mathematical model that, as I’ve described in response to Ruby earlier, represents a very lossy compression of the underlying physical substrate
I can understand why you think that just the neurons / connections is too lossy, but I’m very skeptical of the idea that we’d need all of the amplitudes related to the brain/mind. Apriori that seems unlikely whatwith how little fundamentally turns on the specifics of QM, and those that do can all be implemented specially. As I discussed above some.
(That also reminds me of another reason why people sometimes just mentions neurons/connections which I forgot in my first reply: because they assume you’ve gotten the basic brain architecture that is shared and just need to plug in the components that vary)
I disagree that this distinction between our model and reality has been lost, merely that it has been deemed not too significant, or as something you’d study in-depth when actually performing brain uploads.
What is “the computation”? Can we try to taboo that word?
As I said in my previous comment, and earlier in this one, I’m open to the idea of computation being subjective instead of a purely natural concept. Though I’d expect that there’s not that many free variables in pinning down the meaning. As for tabooing, I think that is kind of hard, as one very simple way of viewing computation is “doing things according to rules”.
You have an expression . This is in your mind and relies on subjective interpretations of what the symbols mean. You implement that abstract program (that abstract doing-things, a chain of rules of inference, a way that things interact) into a computer. The transistors were utilized because they matched the conceptual idea of how switches should function, but they have more complexities than the abstract switch, which introduces design constraints throughout the entire chip. The chip’s ALU implements this through a bunch of transistors, which are more fundamentally made up of silicon in specific ways that regulate how electricity moves. There’s layers and layers of complexities even as it processes the specific binary representations of the two numbers and shifts them in the right way. But, despite all this, all that fundamental behavior, all the quantum effects like tunneling which restrict size and positioning, it is computing the answer. You see the result, , and are pretty confident that no differences between your simple model of the computer and reality occurred.
This is where I think arguments about subjectivity of computation can be made. Introduce a person who is talking about a different abstract concept, they encode it as binary because that’s what you do, and they have an operation that looks like multiplication and produces the same answer for that binary encoding. Then, the interpretation of that final binary output is dependent on the mind, because the mind has a different idea of what they’re computing. (But with the abstract idea being different, even if those parts match up) But I think a lot of those cases are non-natural, which is part of why I think even if computation doesn’t make sense as a fundamental thing or a completely natural concept, it still covers a wide area of concern and is a useful tool. Similar to how the distinction of values and beliefs is a useful tool even when strictly discussing humans, but even moreso. So then, the two calculators are implementing the same abstract algorithm in their silicon, and then we fall back to two questions 1) is the mind within the edge-cases such that it is not entirely meaningful to talk about an abstract program that it is implementing 2) okay, but even if they share the same computation, what does that imply. I think there could and should be more discussion of the complications around computation, with the easy to confuse interaction between levels of ‘completely abstract idea’ (platonism?), ‘abstract idea represented in the mind’ (what I’m talking about with abstract; subjective), ‘the physical way that all the parts of this structure behave’ (excessive detail but as accurate as possible; objective), ‘the way these rules do a specific abstract idea’ (chosen because of abstract ideas like a transistor is chosen because it functions like a switch, and the computer program is compiled in such a way because it matches the textual code you wrote which matches the abstract idea in your own mind; objective in that it is behaving in such a way, possibly subjective interpretation of the implications of that behavior).
We could also view computation through the lens of Turing Machines, but then that raises the argument of “what about all these quantum shenanigans, those are not computable by a turing machine”. I’d say that finite approximations get you almost all of what you want. Then there’s the objection of “turing machines aren’t available as a fundamental thing”, which is true, and “turing machines assume a privileged encoding”, which is part of what I was trying to discuss above.
(I got kinda rambly in this last section, hopefully I haven’t left any facets of the conversation with a branch I forgot to jump back to in order to complete)
the lack of argumentation or discussion of this particular assumption throughout the history of the site means it’s highly questionable to say that assuming it is “reasonable enough”
While discussion on personal identity has mostly not received a single overarching post focusing solely on arguing all the details, it has been discussed to varying degrees of possible contention points. Thou Art Physics which focuses on getting the idea that you are made up of physics into your head, Identity Isn’t in Specific Atoms which tries to dissolve the common intuition of the specific basic atoms mattering, Timeless Identity which is a culmination of various elements of those posts into the idea that even if you duplicate a person they both are still ‘you’. There is also more, some of which you’ve linked, but I consider it strange to say that there’s a lack of discussion. The sequence that the posts I’ve linked are a part of have other discussions, though I agree that they are often from the position of arguing against a baseline of dualism, but I believe they have many points that are relevant to an argument for computationalism. I think there is a lack of discussion about the very specific points you have a tendency to raise, but as I’ll discuss, I find myself confused about their relevancy to varying degrees.
There’s also the facet of decision theory posting that LW enjoys, which encourage this class of view. With decision problems like Newcomb’s Paradox or Parfit’s hitchhiker emphasizing the focus of “you can be instantiated inside a simulation to predict your actions, and you should act like that you — roughly — control their actions because of the similarity of your computational implementations”. Of course, this works even without assuming the simulations are conscious, but I do think it has led to clearer consideration because it helps break past people’s intuitions. Those intuitions are not made for the scenarios that we face, or will potentially have to face.
Bensinger yet again replied in a manner that seemed to indicate he thought he was arguing against a dualist who thought there was a little ghost inside the machine, an invisible homunculus that violated physicalism
Because most often the people suggesting such are dualists, or have a lot of the similar ideas even if they are discussed in an “I am uncertain” manner. I agree Rob could’ve given a better reply, but it was a reasonable assumption. (I personally found Andesolde’s argument confused, with the later parts having a focus on first-person subjective experience that I think is not really useful to consider. There is uncertainties in there, but besides the idea that the mind could be importantly quantum in some way, didn’t seem that relevant.)
That’s perfectly fine, but “souls don’t exist and thus consciousness and identity must function on top of a physical substrate” is very different from “the identity of a being is given by the abstract classical computation performed by a particular (and reified) subset of the brain’s electronic circuit,” and the latter has never been given compelling explanations or evidence.
I agree it hasn’t been argued in depth — but there has definitely been arguments about the extent QM affects the brain. Of which, the usual conclusion was that the effect is minor, and/or that we had no evidence for believing it necessary. I would need a decently strong argument that QM is in some way computationally essential.
the entire brain structure in favor of (a slightly augmented version of) its connectome, and the entire chemical make-up of it in favor of its electrical connections.
More than just the electrical signals matter, this is understood by most. There’s plenty of uncertainty about the level of detail needed to simulate/model the brain. Computationalism doesn’t imply that only the electrical signals matter, it implies that whatever makes up the computation matters, which can be done via tiny molecules & electrons, water pipes, or circuitry. Simplifying a full molecular simulation to the functional implications of it is just one example of how far we can simplify, which I believe should extend pretty far.
“your mind is a pattern instantiated in matter”
I agree that people shouldn’t assume that just neurons/connections are enough, but I doubt that is a strongly held belief; nor is it a required sub-belief of computationalism. You assume too much about Bensinger’s reply when he didn’t respond, especially as he was responding to subargument in the whole chain.
As well, the quoted sentence by Herd is very general — allowing both the neuron connections and molecular behavior. (There’s also the fact that people often handwave over the specifics of what part of the brain you’re extracting, because they’re talking about the general idea through some specific example that people often think about. Such as a worm’s neurons.)For example, for two calculators, wouldn’t you agree with a description of them as having the same ‘pattern’ even if all the atoms aren’t in the same position relative to a table? You agree-reacted on one of dirk’s comments:
https://www.lesswrong.com/posts/zPM5r3RjossttDrpw/when-is-a-mind-me?commentId=wziGLYTwM4Nb9gd6E I disagree that your mind is “a pattern instantiated in matter.” Your mind is the matter. It’s precisely the assumption that the mind is separable from the matter that I would characterize as non-physicalist.
Would the idea that a calculator has some pattern, some logical rules that it is implementing via matter, thus be non-physicalist about calculators? A brain follows the rules of reality, with many implications about how certain molecules constrain movement, how these neuron spikes cause hunger, etcetera. There is a logical/computational core to this that can be reimplemented.
The basic concept of computation at issue here is a feature of the map you could use to approximate reality (i.e., the territory) . It is merely part of a mathematical model that, as I’ve described in response to Ruby earlier, represents a very lossy compression of the underlying physical substrate
Why shouldn’t we decide based on a model/category? Just as there’s presumably edge-cases to what counts as a ‘human’ or ‘person’. There very well may be strange setups which we can’t reasonably determine to our liking whether we consider it computably implementing a person, a chihuahah, or the weather of Jupiter.
We could try to develop a theory of identity down to the last atom, still operating on a model but at least an extremely specific model, which would presumably force us to narrow in on confusing edge-cases. This would be interesting to do once we have the technology, though I expect there to be edge-cases no matter what, where our values aren’t perfectly defined, which might mean preserving option value. I’m also skeptical that most methods present a very lossy compression even if we assume classical circuits. Why would it? (Or, if you’re going to raise the idea of only getting some specific sub-class of neuron information, then sure, that probably isn’t enough, but I don’t care about that)From this angle where you believe that computation is not fundamental or entirely well-defined, you can simplify the computationalist proposal as “merely” applying in a very large class of cases. Teleporters have no effect on personal identity due to similarity in atomic makeup up to some small allowance for noise (whether simple noise, or because we can’t exactly copy all the quantum parts; I don’t care if my lip atoms are slightly adjusted). Cloning does not have a strictly defined “you” and “not-you”. Awakening from cryogenics counts as a continuation of you. A simulation implementing all the atomic interactions of your mind is very very likely to be you, and a simulation that has simplified many aspects of that down is also still very likely to be you.
Though there are definitely people who believe that the universe can fundamentally be considered computation, which I find plausible, especially due to a lack of other lenses that aren’t just “reality is”. Of which, your objection does not work without further argumentation with them.
Going back to the calculator example, you would need to provide argumentation for why the essential parts of the brain can’t be implemented computationally.
(You link https://www.lesswrong.com/posts/zPM5r3RjossttDrpw/when-is-a-mind-me#5DqgcLuuTobiKqZAe ])
What I value about me is the pattern of beliefs, memories, and values.The attempted mind-reading of others is (justifiably) seen as rude in conversations over the Internet, but I must nonetheless express very serious skepticism about this claim, as it’s currently written. For one, I do not believe that “beliefs” and “values” ultimately make sense as distinct, coherent concepts that carve reality at the joints. This topic has been talked about before on LW a number of times, but I still fully endorse Charlie Steiner’s distillation of it in his excellently-written Reducing Goodhart sequence
Concepts can still be useful categorizations even if they aren’t hard and fast. Beliefs are often distinct from values in humans. They are vague and intertwine with each other, a belief forming a piece of value that doesn’t fade away even once the belief is proven false, a value endorsing a belief for no reason… They are still not one and the same. I also don’t see what this has relevance to in the statement. I agree with what they said. I value my pattern of beliefs, memories, and values. I don’t care about my specific spatial position for identity (except insofar as I don’t want to be in a star), or if I’m solely in baseline reality. They are vague and intertwine with each other, but they do behave differently. Your objections to CEV also seem to me to follow a similar pattern as this, where you go “this does not have a perfect foundational backing” to thus imply “it has no meaning, and there’s nothing to be said about it”. The consideration of path-dependency in CEV has been raised before, and it is an area that would be great to understand more. My values would say that I meta-value my beliefs to be closer to the truth. There are ambiguities in this area. What about beliefs affecting my values? There’s more uncertainty in that region of what I wish to allow.
In any case, the rather abstract “beliefs, memories and values” you solely purport to value fit the category of professed ego-syntonic morals much more so than the category of what actually motivates and generates human behavior, as Steven Byrnes explained in an expectedly outstanding way:
I’d need a whole extra long comment to respond to all the various other parts of your comment chain. Such as indexicality, or the part which does the lines of saying “professed values are not real”. Which seems decently false, overly cynical, and also not what Byrnes’ linked post tries to imply. I’d say, professed values are often what you tend towards, but that your basic drives are often strong enough to stall out methods like “spend long hours solving some problem” due to many small opportunities. If you were given a big button to do something you profess to value, then you’d press it.
This also raises the question of: Why should I care that the human motivational system has certain basic drives driving it forward? Give me a big button and I’d alter my basic drives to be more in-line with my professed values. The basic drives are short-sighted. (Well, I’d prefer to wait until superintelligent help, because there’s lots of ways to mess that up) Of course, that I don’t have the big button has practical implications, but I’m primarily arguing against the cynical denial of having any other values than what these basic drives allow.
(I don’t entirely like my comment, it could be better. I’d suggest breaking the parent question-post up into a dozen smaller questions if you want discussion, as the many facets could have long comments dedicated to each. Which is part of why there’s no single post! You’re touching on everything from theory of how the universe works, to how much the preferences we say are real, to whether our models of reality are useful enough for theories of identity, indexicality, whether it makes sense to talk about a logical pattern, etc. Then there’s things like andesolde’s posts that you cite, but I’m not sure I rely on, where I’d have various objections to their idea of reality as subjective-first. I’ll probably find more I dislike about my comment, or realize that I could have worded or explained better once I come around to reading back over it with fresh eyes.)
it fits with that definition
Ah, I rewrote my comment a few times and lost what I was referencing. I originally was referencing the geometric meaning (as an alternate to your statistical definition), two vectors at a right angle from each other.
But the statistical understanding works from what I can tell? You have your initial space with extreme uncertainty, and the orthogonality thesis simply states that (intelligence, goals) are not related — you can pair some intelligence with any goal. They are independent of each other at this most basic level. This is the orthogonality thesis. Then, in practice, you condition your probability distribution over that space with your more specific knowledge about what minds will be created, and how they’ll be created. You can consider this as giving you a new space, moving probability around. As an absurd example: if height/weight of creatures were uncorrelated in principal, but then we update on “this is an athletic human”, then in that new distribution they are correlated! This is what I was trying to get at with my R^2 example, but apologies that I was unclear since I was still coming at it from a frame of normal geometry. (Think, each axis is an independent normal distribution but then you condition on some knowledge that restricts them such that they become correlated)
I agree that it is an informal argument and that pinning it down to very detailed specifics isn’t necessary or helpful at this low-level, I’m merely attempting to explain why orthogonality works. It is a statement about the basic state of minds before we consider details, and they are orthogonal there; because it is an argumentative response to assumptions about “smart → not dumb goals”.
I’m confused, why does that make the term no longer useful? There’s still a large distinction between companies focusing on developing AGI (OpenAI, Anthropic, etc.) vs those focusing on more ‘mundane’ advancements (Stability, Black Forest, the majority of ML research results). Though I do disagree that it was only used to distinguish them from narrow AI. Perhaps that was what it was originally, but it quickly turned into the roughly “general intelligence like a smart human” approximate meaning we have today.
I agree ‘AGI’ has become an increasingly vague term, but that’s because it is a useful distinction and so certain groups use it to hype. I don’t think abandoning a term because it is getting weakened is a great idea.
We should talk more about specific cognitive capabilities, but that isn’t stopped by us using the term AGI, it is stopped by not having people analyzing whether X is an important capability for risk or capability for stopping risk.