If you aim as if there were no external factors at that range (especially bullet drop!) you will definitely miss both. The factors aren’t all random with symmetric distributions having a mode at the aim point.
JBlack
Karma: 2,117
This looks like a false dichotomy. There are far more philosophies than this, both implicit and explicitly stated, on the nature of existence and suffering.
I expect that for pretty much everyone there is a level of suffering that they would be willing to endure for the rest of their lives. Essentially everyone that hasn’t yet killed themselves is evidence of this, and those that do express intending to kill themselves very often report that continuing to live seems unbearable in some sense or other—which seems to indicate a greater than average degree of suffering.
Likewise I expect that for pretty much everyone, there exists a level of suffering beyond which they’d rather die if they knew that the suffering was was going to persist for the rest of their life. They may say now that they’d rather endure it, but there is plenty of evidence that people routinely underestimate their reactions in such circumstances.
So my expectation is that even at its simplest, it’s a scale. Then this is confounded by all sorts of other considerations such as whether they feel moral obligations to continue enduring suffering (especially if other people are depending upon them in some way), how they would like to be perceived in the future (for both outcomes not just one), whether they want to be a person who endures suffering, and so on.
There’s a very plausible sense in which you may not actually get a choice to not exist.
In pretty much any sort of larger-than-immediately-visible universe, there are parts of the world (timelines, wavefunction sections, distant copies in an infinite universe, Tegmark ensembles, etc) in which you exist and have the same epistemic state as immediately prior to this choice, but weren’t offered the choice. Some of those versions of you are going to suffer for billions of years regardless of you choosing to no longer exist in this fragment of the world.
Granted, there’s nothing you can do about them—you can only choose your response in worlds where you get the choice.
From the wider point of view it may or may not change things. For example suppose you knew (or the superintelligence told you as follow-up information) that in worlds having an essentially identical “you” in them, 10% will be unconditionally tortured for billions of years, and 90% are offered the question (with a 2% chance of hell and 98% chance of utopia). The superintelligence knows that in most timelines leading to hellworlds there is no care for consent while utopias do, which is why conditional on consent the chance is only 2% rather than the overall 11.8%.
If you are the sort of person to choose “nonexistence” then 10% of versions of you go to hell and 90% die. If you choose “live” then in total 11.8% of you go to hell and 88.2% to utopia. The marginal numbers are the same, but you no longer get the option to completely save all versions of you from torture.
Is it still worthwhile for those who can to choose death? This is not rhetorical, it is a question that only you can answer for yourself. Certainly those in the 1.8% would regret being a “choose life” decider and joining the 10% who never got a choice.
I don’t see how they’re “the exact opposite way”. The usual rules of English grammar make this a statement that those those who are born in the United State but belong to families of accredited diplomatic personnel are foreigners, i.e. aliens.
Perhaps you read the statement disjunctively as “foreigners, [or] aliens, [or those] who belong [...]”? That would require inserting extra words to maintain correct grammatical structure, and also be a circular reference since the statement is intended to define those who are considered citizens and those who are considered non-citizens (i.e. foreigners, aliens).
By the nature of the experiment you know that the people on Mars will have direct, personal experience of continuity of identity across the teleport. By definition, their beliefs will be correct.
In 99.9999999999999999999999999999% of world measure no version of you is alive on Earth to say any different. In 0.0000000000000000000000000001% of world measure there is a version of you who is convinced that teleportation does not preserve personal identity, but that’s excusable because extremely unlikely things actually happening can make even rational people have incorrect world models. Even in that radical outlier world, there are 10 people on Mars who know, personally, that the Earth person is wrong.
In my exposure to mathematical literature, almost all sequences have values for which the term “countable” is inapplicable since they’re not sets. Even in the cases where the values themselves are sets, it was almost always used to mean a sequence with countable domain (i.e. length) and not one in which all elements of the codomain (values) are countable. It’s usually in the sense of “countably infinite” as opposed to “finite”, rather than opposed to “uncountably infinite”.
ChatGPT is just bad at mathematical reasoning.
I don’t think you would get many (or even any) takers among people who have median dates for ASI before the end of 2028.
Many people, and particularly people with short median timelines, have a low estimate of probability of civilization continuing to function in the event of emergence of ASI within the next few decades. That is, the second dot point in the last section “the probability of me paying you if you win was the same as the probability of you paying me if I win” does not hold.
Even without that, suppose that things go very well and ASI exists in 2027. It doesn’t do anything drastic and just quietly carries out increasingly hard tasks through 2028 and 2029 and is finally recognized as having been ASI all along in 2030. By this time everyone knows that it could have automated everything back in 2027, but Metaculus doesn’t resolve until 2030 so you win despite being very wrong about timelines.
Other not-very-unlikely scenarios include Metaculus being shut down before 2029 for any reason whatsoever (violating increasingly broad online gambling laws, otherwise failing as a viable organization, etc.), or that specific question being removed or reworded more tightly.
So the bet isn’t actually decided just by ASI timelines, but is one in which the short-timelines side of the bet only wins in the case of additional conjunctions with many clauses.
Operationalizing bets where at least one side believe that there is a significant probability of the end of civilization if they should win is already difficult. Tying one side of the bet but not the other to the continued existence of a very specific organization just makes it worse.
Yes, and (for certain mainstream interpretations) nothing in quantum mechanics is probabilistic at all: the only uncertainty is indexical.
My description “better capabilities than average adult human in almost all respects”, differs from “would be capable of running most people’s lives better than they could”. You appear to be taking these as synonymous.
The economically useful question is more along the lines of “what fraction of time taken on tasks could a business expect to be able to delegate to these agents for free vs a median human that they have to employ at socially acceptable wages” (taking into account supervision needs and other overheads in each case).
My guess is currently “more than half, probably not yet 80%”. There are still plenty of tasks that a supervised 120 IQ human can do that current models can’t. I do not think there will remain many tasks that a 100 IQ human can do with supervision that a current AI model cannot with the same degree of supervision, after adjusting processes to suit the differing strengths and weakness of each.
Your test does not measure what you think it does. There are people smarter than me who I could not and would not trust to make decisions about me (or my computer) in my life. So no. (Also note, I am very much not of average capability, and likewise for most participants on LessWrong)
I am certain that you also would not take a random person in the world of median capability and get them to do 90% of the things you do with your computer for you, even for free. Not without a lot of screening and extensive training and probably not even then.
However, it would not take much better reliability for other people to create economically valuable niches for AIs with such capability. It would take quite a long time, but even with zero increases in capability I think AI would be eventually be a major economic factor replacing human labour. Not quite transformative, but close.
In my reading, I agree that the “Slow” scenario is pretty much the slowest it could be, since it posits an AI winter starting right now and nothing beyond making better use of what we already have.
Your “Fast” scenario is comparable with my “median” scenario: we do continue to make progress, but at a slower rate than the last two years. We don’t get AGI capable of being transformative in the next 3 years, despite going from somewhat comparable to a small child in late 2022 (though better in some narrow ways than an adult human) to better capabilities than average adult human in almost all respects in late 2024 (and better in some important capabilities than 99.9% of humans).
My “Fast” scenario is one in which internal deployment of AI models coming into existence in early-to-mid 2025 allow researchers to make large algorithmic and training improvements in the next generation (by late 2025) which definitely qualify as AGI. Those then assist to accelerate the pace of research with better understanding of how intelligence arises leading to major algorithmic and training improvements and indisputably superhuman ASI in 2026.
This Fast scenario’s ASI may not be economically transformative by then, because human economies are slow to move. I wouldn’t bet on 2027 being anything like 2026 in such a scenario, though.
I do have faster scenarios in mind too, but far more speculative. E.g. ones in which the models we’re seeing now are already heavily sandbagging and actually superhuman, or in which other organizations have such models privately.
The largest part of my second part is “If consciousness is possible at all for simulated beings, it seems likely that it’s not some “special sauce” that they can apply separately to some entities and not to otherwise identical entities, but a property of the structure of the entities themselves.” This mostly isn’t about simulators and their motivations, but about the nature of consciousness in simulated entities in general.
On the other hand your argument is about simulators and their motivations, in that you believe they largely both can and will apply “special sauce” to simulated entities that are the most extreme in some human-obvious way and almost never to the others.
I don’t think we have any qualitative disagreements, just about what fraction of classes of simulated entities may or may not have consciousness.
There is no correct mathematical treatment, since this is a disagreement about models of reality. Your prior could be correct if reality is one way, though I think it’s very unlikely.
I will point out though that for your reasoning to be correct, you must literally have Main Character Syndrome, believing that the vast majority of other apparently conscious humans in such worlds as ours are actually NPCs with no consciousness.
I’m not sure why you think that simulators will be sparse with conscious entities. If consciousness is possible at all for simulated beings, it seems likely that it’s not some “special sauce” that they can apply separately to some entities and not to otherwise identical entities, but a property of the structure of the entities themselves. So in my view, an exceptionally tall human won’t be given “special sauce” to make them An Observer, but all sufficiently non-brain-damaged simulated humans will be observers (or none of them).
It might be different if the medically and behaviourally similar (within simulation) “extremest” and “other” humans are not actually structurally similar (in the system underlying the simulation), but are actually very different types of entities that are just designed to appear almost identical from examination within the simulation. There may well be such types of simulations, but that seems like a highly complex additional hypothesis, not the default.
In my opinion, your trilemma definitely does not hold. “Free will” is not a monosemantic term, but one that encompasses a range of different meanings both when used by different people and even the same person in different contexts.
is false, because the term is meaningful, but used with different meanings in different contexts;
is false, because you likely have free will in some of those senses and do not in others, and it may be unknown or unknowable in yet more;
is false for the same reason as 2.
For example: your mention of “blame” is a fairly common cluster of moral or pragmatic concepts attached to discussions of free will, but is largely divorced from any metaphysical aspects of free will.
Whether or not a sapient agent metaphysically could have acted differently in that specific moment is irrelevant to whether it is moral or useful to assign blame to that agent for the act (in such discussions, usually an act that harms others). Even under the most hardcore determinism and assuming immutable agents, they can be classified into those that would and those that wouldn’t have performed that act and so there is definitely some sort of distinction to be made. Whether you want to call it “blame” or not in such a world is a matter of opinion.
However, sapient agents such as humans in the real world are not immutable and can observe how such agents (possibly including themselves) are treated when they carry out certain acts, and can incorporate that into future decisions. This feeds into moral and pragmatic considerations regardless of the metaphysical nature of free will.
There are likewise many other concepts tied into such “free will” discussions that could be separated out instead of just lumping them all together under the same term.
You make the assumption that half of all simulated observers are distinctively unique in an objectively measurable property within simulated worlds having on the order of billions of entities in the same class. Presumably you also mean a property that requires very few bits to specify—such as, if you asked a bunch of people for their lists of such properties that someone could be “most extreme” in, and entropy-code the results, then the property in question would be in the list and correspond to very few bits (say, 5 or fewer).
That seems like a massive overestimate, and is responsible for essentially all of your posterior probability ratio.
I give this hypothesis very much lower weight.
How long is a piece of string?
No, I do not believe that it has been solved for the context in which it was presented.
What we have is likely adequate for current AI capabilities, with problems like this for which solutions exist in the training data. Potential solutions far beyond the training data are currently not accessible to our AI systems.
The parable of wishes is intended to apply to superhuman AI systems that can easily access solutions radically outside such human context.
There are in general simple algorithms for determining S in polynomial time, since it’s just a system of linear equations as in the post. Humans came up with those algorithms, and smart LLMs may be able to recognize the problem type and apply a suitable algorithm in chain-of-thought (with some probability of success).
However, average humans don’t know any linear algebra and almost certainly won’t be able to solve more than a trivial-sized problem instance. Most struggle with the very much simpler “Lights Out” puzzle.
Why doesn’t it work to train on all the 1-hot input vectors using an architecture that suitably encodes Z_2 dot product and the only variable weights are those for the vector representing S? Does B not get to choose the inputs they will train with?Edit: Mentally swapped A with B in one place while reading.
Building every possible universe seems like a very direct way of purposefully creating one of the biggest possible S-risks. There are almost certainly vastly more dystopias of unimaginable suffering than there are of anything like a utopia.
So to me this seems like not just “a bad idea” but actively evil.