Dach

Karma: 92

Dach 18 Oct 2022 4:56 UTC
12 points
0
on: Decision theory does not imply that we get to have nice things
But, sure, if you’re somehow magically unhackable and very good at keeping the paperclipper boxed until you fully understand it, then there’s a chance you can trade, and you have the privilege of facing the next host of obstacles.
Now’s your chance to figure out what the next few obstacles are without my giving you spoilers first. Feel free to post your list under spoiler tags in the comment section.
Ideas:
1. Someone else definitely builds and deploys an UFAI before you finish studying Clippy. (This would almost always happen?)
2. Clippy figures out that it’s in a prisoner’s dilemma with the other cobbled-together UFAI humanity builds, wherein each UFAI is given the option to shake hands with Humanity or pass 100% of the universe to whichever UFAI Humanity eventually otherwise deploys. Clippy makes some models, does some decision theory, predicts that if it defects and handshakes other UFAIs are more likely to defect too based on their models, and decides to not trade. The multiverse contains twice as many paperclips.
3. The fact that you’re going to forfeit half of the universe to Clippy leaks. You lose, but you get the rare novelty Game Over screen as compensation?
Interlocutor: Well, maybe we can train the infant paperclipper in games of increasing complexity, so that it’s never quite sure whether it’s in reality yet. The paperclipper will then be uncertain about whether we humans are simulating it, and will want to cater to our preferences to some extent.
Me: Uh, yeah, your paperclipper will be able to tell when it is finally in the real world.
Interlocutor: What? How?
Me: I suggest maybe spending five minutes thinking about your own answer to that question before I give mine.
Ideas:
1. It could just act friendly for enough time to be sure it’s not in a simulation on the grounds that a civilization that could simulate what it was doing on its computers wouldn’t simulation-fakeout it for non-exotic reasons. Imagine Clippy mulling over its galaxy-sized supercomputing cluster and being like “Hm, I’m not sure if I’m still in those crude simulations those stupid monkeys put me in or I’m in the real world.”
2. I would be surprised if we’re able to build a simulation (before we build AGI) that I couldn’t discern as a simulation 99.99% of the time. Simulation technology just won’t advance fast enough.

Dach 4 Dec 2021 18:50 UTC
4 points
on: Biology-Inspired AGI Timelines: The Trick That Never Works
Is Humbali right that generic uncertainty about maybe being wrong, without other extra premises, should increase the entropy of one’s probability distribution over AGI, thereby moving out its median further away in time?
Writing my response in advance of reading the answer, for fun.
One thought is that this argument fails to give accurate updates to other people. Almost 100% of people would give AGI medians much further away than what I think is reasonable, and if this method wants to be a generally useful method for getting better guesses by recognizing your uncertainty then it needs to push them towards shorter timelines, to whatever degree I trust short timelines.
In fact, this argument seems to only be useful for people whose AGI timelines are shorter than whatever the true timeline ends up being. If this were a real comment I would say this revealed behavior was unsurprising because the argument was generated to argue someone towards longer timelines and thus I couldn’t trust it to give reality-aligned answers.
It strikes me that such a system probably doesn’t exist? At the very least, I don’t know how to turn my “generic uncertainty about maybe being wrong, without other extra premises” into anything. You need to actually exert intelligence, actually study the subject matter, to get better probability distributions. Suppose I have a random number generator that I think gives 0 10% of the time and 1 90% of the time. I can’t improve this without exerting my intelligence- I can’t just shift towards 50% 0 and 50% 1 with no further evidence. That would rely on the assumption that my uncertainty signals I’m biased away from the prior of 50% 0 and 50% 1, which is completely arbitrary.
Note that if you have reason to think your guess is biased away from the prior, you can just shift in the direction of the prior. In this case, if you think you’re too confident relative to a random distribution over all years, which basically means you think your timeline is too short, you can shift in the direction of a random distribution over all years.
In this context, you can’t get better AGI estimates by just flattening over years. You need to actually leverage intelligence to discern reality. There are no magical “be smarter” functions that take the form of flattening your probability distribution at the end.

Dach 2 Dec 2021 20:16 UTC
2 points
in reply to: AnthonyC’s comment on: How common are abiogenesis events?
If I try to answer that question as written, I’d say that any time I see a probability estimate with on-the-order-of-hundreds of zeroes, when I know that event actually happened (at least) once in Earth’s past light cone, I’m going to assume there is an error in the model that generated the estimate, whether I know what it is or not.
I would agree for pretty much any other topic. This is an event required for people to be around to observe it. Imagine a universe in which abiogenesis events really were absurdly rare- unlikely to ever occur in a given observable universe sized area. Every observer in this universe would still look back and see an abiogenesis event occurring in their past! Having observed exactly one event is completely required and provides no evidence. This is essentially the weak anthropic principle.
The fact that we observe only exactly one event is thus bayesian evidence in the direction of rare abiogenesis. You bring up the point that the fact we observe only exactly one event in earth’s history isn’t that strong of evidence because e.g. events being concealed by significantly more advanced competitors. I certainly don’t disagree- that was just unfortunate overemphasis on my part.
So what I way trying to point to is that if a catalytic cycle of many (much smaller) RNA strands was sufficient for an abiogenesis event, that could lower the probability estimate enough to make such events more likely by enough that there could have been multiple even just on Earth without straining credulity, and the world today would likely look basically the same either way since the more-competitive biochemistry would have long since reach fixation (and/or the lineages could have merged in some analog of later endosymbiosis events).
I was thinking of systems more complex than the random coalescence of nucleotides when I wrote my post, but I didn’t know how to productively model that. Someone else had a similar argument and I responded that I slightly shifted away from rare-abiogenesis because I wasn’t thinking of complicated groups of cross-catalyzing RNA in particular, which are dumb enough to have no modern analogues but not so dumb as to not be competitive against single-strand solutions which require the random coalescence of another 100 base pairs, or whatever it is. It’s unclear to what degree this should affect my model, given a lack of understanding of how such systems work.

Dach 30 Nov 2021 1:41 UTC
2 points
in reply to: AnthonyC’s comment on: How common are abiogenesis events?
I’m not aware of an argument that there was only on abiogenesis event on Earth, just the observation that all known surviving lineages come from a universal common ancestor fairly early on. In principle that would be compatible with any number of initial events. It’s just that once a given lineage evolved enough adaptions/improvements, it would spread and take over, and then no new lineage would be able to compete/get started.
Your observation is an argument for only one abiogenesis event, and your claim that one would spread and take over and no new lineages would be able to compete is a counterargument. I think mentally using this wording is helpful for properly probabilistic thinking.
I didn’t mean for the opening comments of my question to be interpreted as justification for the structure of my vague estimate. Said estimate (although plausibly off by several hundred orders of magnitude) says abiogenesis events are cosmically unlikely, and this notion was used in the presentation of the question.

Dach 27 Nov 2021 21:32 UTC
1 point
in reply to: YimbyGeorge’s comment on: How common are abiogenesis events?
While cool, I didn’t expect indefinite self-replication to be hard under these circumstances. The enzymes work by combining two halves of the other enzyme- i.e. they are not self-replicating using materials we would expect to ever naturally occur, they are self-replicating using bisected versions of themselves.
I’ve slightly downgraded my estimate for the minimum viable genome size for self-replicating RNA because I wasn’t thinking about complicated groups of cross-catalyzing RNA.

Dach 19 Oct 2021 13:17 UTC
21 points
in reply to: vV_Vv’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)
Scott: if you believe that people have auras that can implant demons into your mind then you’re clearly insane and you should seek medical help.
Also Scott: beware this charismatic Vassar guy, he can give you psychosis!
These so obviously aren’t the same thing- what’s your point here? If just general nonsense snark, I would be more inclined to appreciate it if it weren’t masquerading as an actual argument.
People do not have auras that implant demons into your mind, and alleging so is… I wish I could be more measured somehow. But it’s insane and you should probably seek medical help. On the other hand, people who are really charismatic can in fact manipulate others in really damaging ways, especially when combined with drugs etc. These are both simultaneously true, and their relationship is superficial.
Scott 2015: Universal love, said the cactus person
Scott 2016: acritically signal boosts Aella talking about her inordinate drug use.
Scott 2018: promotes a scamcoin by Aella and Vinay Gupta, a differently sane tech entrepreneur-cum-spiritual guru, who apparently burned his brain during a “collaborative celebration” session.
Personally, when I read the cactus person thing I thought it was a joke about how using drugs to seek “enlightenment” was dumb, and aside from that it was just entertainment? That Aella thing is a single link in a sea of 40 from 5 years ago, so I don’t care. I don’t know who Vinay Gupta is- from reading Scott’s comments on that thread I get the impression he also didn’t really know who he was?
I’ll add a fourth silly piece of evidence to this list for laughs. In Unsong, the prominent villain known as the Drug Lord is evil and brainwashes people. Must be some sort of hidden message about Michael Vassar, huh? He warned us in advance!

Dach 19 Oct 2021 10:56 UTC
1 point
in reply to: acylhalide’s comment on: Humans are the universal economic bottleneck
Why is this limit unique? Why can’t we be working on “distribution inefficiencies and similar” for the next 100 years?
In the case of real GDP per capita per hour worked, this limit is exactly unique- “distribution inefficiencies and similar” doesn’t apply. Indeed, this is tautologically true as you say. Think about what it would look like for an increase in real GDP per capita per hour worked to not have the form of “Something allowed for more work to be done per hour per person”. It wouldn’t look like anything- that doesn’t make any sense.
I would completely ignore my comment on “distribution inefficiencies and similar” until you know what I mean by this. To explain my comment, real GDP per capita per hour worked is not the same as the nebulous “prosperity” I was referring to, which also contains some level of preference for how material goods are distributed.
Maybe I’ll ask this, does your statement regarding universal bottleneck apply explicitly to humans? Or does it also apply to apes and bacteria and AI?
Because we particularly care about how much work humans do, and how wealthy they are. We do not really care about the work hours or prosperity of bacteria. Economic productivity is measured relative to how much money people have and how much they must work to get it. Just read my previous comment and/or the post again- this would seem to be a really basic sort of confusion that I can’t fix for you.

Dach 19 Oct 2021 6:47 UTC
1 point
in reply to: MikkW’s comment on: Humans are the universal economic bottleneck
Self-driving technology is advancing and will soon(ish) allow us to move cars without humans being directly involved, except in terms of maintenance and management. This will be a major boon because it will partially remove humans from the equation- the bottleneck is partially removed. This has no real bearing on the title statement- I even remark about this in my post.
The “universality” here is trivial- here is a copy-paste of part of my response to a similar comment:
For everyone to become richer without working harder, we must develop technologies that allow more work to be done per man-hour. Aside from working out distribution inefficiencies and similar, this is the unique limit on prosperity. This is what I mean by “humans are the universal bottleneck”- we only have so many man-hours, so any growth is going to be of the form “With the same amount of hours, we do more”.
Imagine if every area of economic activity was automated- humans were fully removed. This would look very sci-fi: think of von Neumann probes. In this situation there is no practical limit- the probes will expand and convert our entire light cone. Assuming constant population, per capita wealth would approach 50 billion stars, I guess.

Dach 19 Oct 2021 6:33 UTC
1 point
in reply to: acylhalide’s comment on: Humans are the universal economic bottleneck
For everyone to become richer without working harder, we must develop technologies that allow more work to be done per man-hour. Aside from working out distribution inefficiencies and similar, this is the unique limit on prosperity. This is what I mean by “humans are the universal bottleneck”- we only have so many man-hours, and any growth is going to be of the form “With the same amount of hours, we do more”.
Some segments of the economy have not had as much growth in the above department. For example, houses are assembled manually- all major parts must be done by hand, many with the assistance of only hand-held tools. Because we require shelter and the choice way of getting that shelter is owning a manually built house, this is a drag on the economy.
Domains such as this which have not been as revolutionized by automation have scaling costs due to how directly they are pegged to the price of labor- in particular, houses of constant size cost the same general fraction of our income (if not more) despite real GDP per capita having grown significantly over time, because richer workers demand more pay. Why aren’t houses cheap? This is one of many reasons, and doing some quick googling on how many man-hours are required to build a house, the main factor in many areas of the united states.
Is this way of thinking “new”? Surely not to humanity, but hopefully to the reader.

Dach 18 Oct 2021 14:34 UTC
1 point
in reply to: Yair Halberstadt’s comment on: Humans are the universal economic bottleneck
Interesting, thank you.
Is the quadrupling of drag and octupling of rolling resistance related to the assumption that drag is proportional to the surface area of the side on which the drag is produced, and that rolling resistance is proportional to weight? Either way, cost would still decrease due to larger and more complex engines, as rolling resistance per kg would not change.
Of course, railway sizes are fixed, so there is little to be done. I was just speculating where the relative efficiency of cargo ships comes from. I made an edit at the end of the post which contains a very rough approximation of how large savings on wages are in the case of cargo container ships.

Dach 18 Oct 2021 13:24 UTC
3 points
in reply to: Yair Halberstadt’s comment on: Humans are the universal economic bottleneck
I would expect fuel efficiency to be related to the size and complexity of the engine. Producing some amount of force is going to require the same amount of fuel assuming energy loss due to resistance/friction is the same, and the engine is the same.

If true, we could e.g. have absurdly large trains on lots of rails? I would expect energy loss due to rubbing on rails and changing elevation to be similar to energy loss due to rubbing on water.

Dach 17 Oct 2021 1:59 UTC
2 points
in reply to: cousin_it’s comment on: What Do GDP Growth Curves Really Mean?
This seems wrong. Imagine some country doesn’t have unobtainium, a mineral which is rare and also not particularly useful. You can’t get it at any price. Then it finds some, and soon enough many citizens have unobtainium paper holders. Does this mean GDP has grown by a factor of infinity? Hell no, most people would gladly exchange their paper holders for something more useful but also previously obtainable.
Think about it this way. Suppose we have some device that was moderately valuable which everyone needed to own exactly one of, and it costs $100 per year. I don’t know- maybe all humanity has some heart disease and we need to have this medical device or we die. How much “better” does this situation get if we figure out how to cure this problem for essentially free? Say someone develops a pill that costs $.01 and it lasts for a year. In some sense, this is 10,000 times better. No one’s lives are even approximately 10,000 times better, though- people are only $99.99 richer per year.
Ignoring diminishing marginal utility, this can be attributed to the fact that people’s budgets are not completely saturated with such innovations- housing isn’t sufficiently decoupled from things we can’t dramatically improve, like the skill of workers who operate power tools, zoning laws, and the surface area of cities.
If everything got cheaper and better at the same rate as computers, I think it would be fair to assign some ridiculous multiplier to the increase in productivity over the past 60 years. Houses would cost cents and particle colliders would circle the solar system.
Isn’t it the same for computers then? You’d exchange an iphone for a house in an instant, and you can’t exchange your iphone for a hundred houses. So it’s just not that valuable to you, and real GDP hasn’t grown by a factor of hundreds.
This example seems bad? Few of us are wealthy enough to purchase a hundred houses. More directly, I would pay up to all of my discretionary income (i.e. not required to sustain my life or support my income) for access to computers and internet.
The situation is not such that the degree to which computers are valuable to me is appropriately represented in my decision making- computers are extremely cheap, and I am wealthy enough to not have to make any sort of compromise to own one.
I agree that real GDP or rather human productivity in the intuitively relevant sense hasn’t grown by a factor of 100 since 1960.

Dach 5 Oct 2021 23:16 UTC
1 point
on: Dissolving the Experience Machine Objection
Is that your real disagreement with the experience machine?
I think if you accept the premise that the machine somehow magically truly simulates perfectly and indistinguishably from actual reality, in such a way that there is absolutely no way of knowing the difference between the simulation and the outside universe, then the simulated universe is essentially isomorphic to reality, and we should be fully indifferent. I’m not sure it even makes sense to say either universe is more “real”, since they’re literally identical in every way that matters (for the differences we can’t observe even in theory, I appeal to Newton’s flaming laser sword). Our intuitions here should be closer to stepping into an identical parallel universe, rather than entering a simulation.
The experience machine does not need to be an exact simulation of the entire observable universe, it merely needs to be a passably high quality simulation of your brain and how it would change. You can’t see the fundamental particles around you, or the evolution of the universal wavefunction. A superintelligence would thus not need to perfectly fake reality to perfectly deceive you and leave you with the same level of confidence that you’re in a simulation as you have right now.
In fact, this is the original construction of the argument- it didn’t involve perfectly simulating the universe, it just involved deceiving the subject to such a degree that they wouldn’t be smart enough to figure out which was real.
However, I think it’s not actually possible to have such a perfect experience machine, which would explain our intuition for not wanting to step inside. First, if this machine simulates reality using our knowledge of physics at the time, it’s entirely possible that there are huge parts of physics you would never be able to find out about inside the machine, since you can never be 100% sure whether you really know the Theory of Everything. Second, this machine would have to be smaller than the universe in some sense, since it’s part of the universe. As a result, the simulation would probably have to cut corners or reduce the size of the simulated universe substantially to compensate.
This can’t “explain” our intuitions, because the fact that physically simulating the entire universe in exact detail is impossible is evidently not what’s causing those intuitions. People who disagree with the experience machine include those who have no idea what a “wave function” is, those who think the earth is flat and the entire universe is in a small crystal dome, leading physicists, philosophers who think building the experience machine as described is possible, and I allege many of the readers of this post before they knew that simulating the universe in exact detail was physically impossible.
Can it explain your disagreement? That’s something for you to think about yourself.
I personally would not step in, because I am at least mildly robust to attempts to hijack my senses. Whenever the map could become disentangled from the territory due to the plots of mad philosophers, I will try to avoid this.

Dach 3 Nov 2020 1:37 UTC
1 point
in reply to: Scott Garrabrant’s comment on: What are good election betting opportunities?
I can confirm that this still works. Sum of the price of all Nos is $14.77, payoff is $15.

Dach 20 Oct 2020 4:46 UTC
1 point
on: As a Washed Up Former Data Scientist and Machine Learning Researcher What Direction Should I Go In Now?
So, I guess the question boils down to, how seriously should I consider switching into the field of AI Alignment, and if not, what else should I do instead?
I think you should at least take the question seriously. You should consider becoming in involved in AI Alignment to the extent that you think doing so will be the highest value strategy, accounting for opportunity costs. An estimate for this could be derived using the interplay between your answers to the following basic considerations:
- What are your goals?
- What are the most promising methods for pursuing your various goals?
  - What resources do you have, and how effective would investing those resources be, on a method by method and goal by goal basis?
An example set of (short and incomplete) answers which would lead you to conclude “I should switch to the field of AI Alignment” is:
- In addition to “normal” every day goals, I want extreme things. I care about the broad scope future of Humanity to a degree even roughly corresponding to what basic math suggests.
- The best way to achieve my extreme goals is to break the current limitations on intelligence (brains need to fit in very small boxes, synapses fire in slow motion, you need to train each individual human, training time for individual humans is bad, I/O speed is complete trash, etc.) and to safely exploit this.
  - I have the personal aptitude to work on this problem. It would be better for me to devote my personal time to this project than to pay other people to work on this project for me, for example. My normal ever day goals can also be effectively pursued by moving to the field.
Like should I avoid working on AI at all and just do something fun like game design, or is it still a good idea to push forward ML despite the risks?
If you’re not doing bleeding edge research (and no one doing bleeding edge research is reading your papers), your personal negative impact on AI Alignment efforts can be more effectively offset by making more money and then donating e.g. $500 to MIRI (or related) than changing career.
And if switching to AI Alignment should be done, can it be a career or will I need to find something else to pay the bills with as well?
AI Alignment is considered by many to be literally the most important problem in the world. If you can significantly contribute to AI Alignment, you will be able to find someone to give you money.
If you can’t significantly personally contribute to AI Alignment but still think the problem is important, I would advise advancing some other career and donating money to alignment efforts, starting a youtube channel and spreading awareness of the problem, etc.
I am neither familiar with you nor an alignment researcher, so I will eschew giving specific career advice.

Dach 15 Oct 2020 10:58 UTC
1 point
in reply to: Said Achmiz’s comment on: Industrial literacy
You were welcome to write an actual response, and I definitely would have read it. I was merely announcing my advanced intent to not respond in detail to any following comments, and explaining why in brief, conservative terms. This is seemingly strictly better- it gives you new information which you can use to decide whether or not you want to respond. If I was being intentionally mean, I would have allowed you to write a detailed comment and never responded, potentially wasting your time.
If your idea of rudeness is constructed in this (admittedly inconvenient) way, I apologize.

Dach 15 Oct 2020 9:45 UTC
−6 points
in reply to: Said Achmiz’s comment on: Industrial literacy
Whether the future “matters more than today” is not a question of impersonal fact. Things, as you no doubt know, do not ‘matter’ intransitively; they matter to someone. So the question is, does “the future” (however construed) matter to me more than “today” (likewise, however construed) does? Does “the future” matter to my hypothetical friend Alice more than today does, or to her neighbor Bob? Etc.
And any of these people are fully within their right to answer in the negative.
Eh… We can draw conclusions about the values of individuals based on the ways in which they seem to act in the limit of additional time and information, the origins of humanity (selection for inclusive genetic fitness), by constructing thought experiments to solicit revealed beliefs, etc.
Other agents are allowed to claim that they have more insight than you into certain preferences of yours- they often do. Consider the special cases in which you can prove that the stated preferences of some humans allow you to siphon infinite money off of them. Also consider the special cases in which someone says something completely incoherent- “I prefer two things to one another under all conditions”, or some such. We know that they’re wrong. They can refuse to admit they’re wrong, but they can’t properly do that without giving us all of their money or in some sense putting their fingers in their ears.
These special cases are just special cases. In general, values are highly entangled with concrete physical information. You may say that you want to put your hand on that (unbeknownst to you) searing plate, but we can also know that you’re wrong. You don’t want to do that, and you’d agree if only you knew that the plate was searing hot.
They are fully within their right to answer in the negative, but they’re not allowed to decide that they’re correct. There is a correct answer to what they value, and they don’t necessarily have perfect insight into that.
Note that you’re making a non-trivial claim here. In past discussions, on Less Wrong and in adjacent spaces, it has been pointed out that our ability to predict future consequences of our actions drops off rapidly as our time horizon recedes into the distance. It is not obvious to me that I am in any particularly favorable position to affect the course of the distant future in any but the most general ways (such as contributing to, or helping to avert, human extinction—and even there, many actions I might feasibly take could plausibly affect the likelihood of my desired outcome in either the one direction or the other).
You don’t need to be able to predict the future with omniscient accuracy to realize that you are in an unusually important position for affecting the future.
If it’s not obvious, here we go: You’re an above average intelligence person living in the small period directly before Humanity is expected (By top experts- and with good cause) to develop artificial general intelligence. This technology will allow us to break the key scarcities of civilization:
1. Allowing vastly more efficient conversion of matter into agency through the fabrication of computer hardware. This process will, given the advent of artificial general intelligence, soon far surpass the efficiency with which we can construct Human agency. Humans take a very long time to make, and you must train each individual Human- you can’t directly copy Human software, and the indirect copying is very, very slow.
2. Allowing agents with intelligence vastly above that of the most intelligent humans (whose brains must all fit in a container of relatively limited size) in all strategically relevant regards- speed, quality, modularity, I/O speed, multitasking ability, adaptability, transparency, etc.
3. Allowing us to build agents able to access a much more direct method of recursively improving their own intelligence by buying or fabricating new hardware and directly improving their own code, triggering an extremely exploitable direct feedback loop.
The initial conditions of the first agent(s) we deploy which possesses these radical and simultaneously new options which will, on account of the overwhelming importance of these limitations on the existing state of affairs, precisely and “solely” determine the future.
This is a pretty popular opinion among the popular rationalist writers- I pass the torch on to them.
Sorry, no. There is a categorical difference between bringing a person into existence and affecting a person’s future life, contingent on them being brought into existence. It of course makes sense to speak of doing the latter sort of thing “for” the person-to-be, but such isn’t the case for the former sort of thing.
I was aware of the difference. The point (Which I directly stated at the end- convenient!) is that “It does make sense to say that you’re doing things ‘for’ people who don’t exist.” If this doesn’t directly address your point, the proper response to make would have been “Ok, I think you misunderstood what I was saying.” I think that I did misunderstand what you were saying, so disregard.
Aside from that, I still think that saying bringing someone into existence “for” them makes sense. I think you saying the thing doesn’t “make sense” is unfairly dismissive and overly argumentative. If someone said that they weren’t going to have an abortion “for” their baby, (or, if you disagree with me about the lines of what constitutes a “person”) that they were stopping some pain relieving experimental drug that was damaging their fertility “for” their future children, you’d receive all of the information they meant to convey about their motivations. It would definitely make sense. You might disagree with that reasoning, but it’s coherent. They have an emotional connection with their as of yet not locally instantiated children.
I personally do happen to disagree with this reasoning for reasons I will explain later- but it does make sense.
To the contrary: your point hinges on this. You may of course discuss or not discuss what you like, but by avoiding this topic, you avoid one of the critical considerations in your whole edifice of reasoning. Your conclusion is unsupportable without committing to a position on this question.
It isn’t, and I just told you that it isn’t. You should have tried to understand why I was saying that before arguing with me- I’m the person who made the comment in the first place, and I just directly told you that you were misinterpreting me.
My point is: “It’s that we should be spending most of our effort on planning for the long term future.” See later for an elaboration.
Quite so—but surely this undermines your thesis, rather than supporting it?
No- I’m not actually arguing for the specific act of ensuring that future humans exist. I think that all humans already exist, perhaps in infinite supply, and I thus see (tentatively) zero value in bringing about future humans in and of itself. My first comment was using a rhetorical flair that was intended to convey my general strategy for planning for the future; I’m more interested in solving the AI alignment problem (and otherwise avoiding human extinction/s-risks) than I am about current politically popular long term planning efforts and the problems that they address, such as climate change and conservation efforts.
I think that we should be interested in manipulating the relative ratios (complicated stuff) of future humans, which means that we should still be interested in “ensuring the existence” (read: manipulating the ratios of different types of) of “Future Humanity”, a nebulous phrase meant to convey the sort of outcome that I want to see to the value achievement dilemma. Personally, I think that the most promising plan for this is engineering an aligned AGI and supporting it throughout its recursive self improvement process.
Your response was kindof sour, so I’m not going to continue this conversation.

Dach 15 Oct 2020 4:56 UTC
2 points
in reply to: Said Achmiz’s comment on: Industrial literacy
Why, exactly, is this our only job (or, indeed, our job at all)? Surely it’s possible to value present-day things, people, etc.?
The space that you can affect is your light cone, and your goals can be “simplified” to “applying your values over the space that you can affect”, therefore your goal is to apply your values over your light cone. It’s you’re “only job”.
There is, of course, a specific notion that I intended to evoke by using this rephrasing: the idea that your values apply strongly over humanity’s vast future. It’s possible to value present-day things, people, and so on- and I do. However… whenever I hear that fact in response to my suggestions that the future is large and it matters more than today, I interpret it as playing defense for their preexisting strategies. Everyone was aware of this before the person said it, and it doesn’t address the central point- it’s...
“There are 4 * 10^20 stars out there. You’re in a prime position to make sure they’re used for something valuable to you- as in, you’re currently experiencing the top 10^-30% most influential hours of human experience because of your early position in human history, etc. Are you going to change your plans and leverage your unique position?”
“No, I think I’ll spend most of my effort doing the things I was already going to do.”
Really- Is that your final answer? What position would you need to be in to decide that planning for the long term future is worth most of your effort?
Seeing as how future humanity (with capital letters or otherwise) does not, in fact, currently exist, it makes very little sense to say that ensuring their existence is something that we would be doing “for” them.
“Seeing as how a couple’s baby does not yet exist, it makes very little sense to say that saving money for their clothes and crib is something that they would be doing ‘for’ them.” No, wait, that’s ridiculous- It does make sense to say that you’re doing things “for” people who don’t exist.
We could rephrase these things in terms of doing them for yourself- “you’re only saving for their clothes and crib because you want them to get what they want”. But, what are we gaining from this rephrasing? The thing you want is for them to get what they want/need. It seems fair to say that you’re doing it for them.
There’s some more complicated discussion to be had on the specific merits of making sure that people exist, but I’m not (currently) interested in having that discussion. My point isn’t really related to that- it’s that we should be spending most of our effort on planning for the long term future.
Also, in the context of artificial intelligence research, it’s an open question as to what the border of “Future Humanity” is. “Existing humans” and “Future Humanity” probably have significant overlap, or so the people at MIRI, DeepMind, OpenAI, FHI, etc. tend to argue- and I agree.

Dach 11 Oct 2020 1:27 UTC
1 point
in reply to: Walker Vargas’s comment on: Not Even Evidence
This doesn’t require faster than light signaling. If you and the copy are sent way with identical letters, that you open after crossing each other’s event horizons. You learn want was packed with your clone when you open your letter. Which lets you predict what your clone will find.
Nothing here would require the event of your clone seeing the letter to affect you. You are affected by the initial set up.
Another example would be if you learn a star that has crossed your cosmic event horizon was 100 solar masses, it’s fair to infer that it will become a black hole and not a white dwarf.
If you can send a probe to a location, radiation, gravitational waves, etc. from that location will also (in normal conditions) be intercepting you, allowing you to theoretically make pretty solid inferences about certain future phenomena at that location. However, we let the probe fall out of our cosmological horizon- information is reaching it that couldn’t/can’t have reached the other probes, or even the starting position of that probe.
In this setup, you’re gaining information about arbitrary phenomena. If you send a probe out beyond your cosmological horizon, there’s no way to infer the results of, for example, non-entangled quantum experiments.
I think we may eventually determine the complete list of rules and starting conditions for the universe/multiverse/etc. Using our theory of everything and (likely) unobtainable amounts of computing power, we could (perhaps) uniquely locate our branch of the universal wave function (or similar) and draw conclusions about the outcomes of distant quantum experiments (and similar). That’s a serious maybe- I expect that a complete theory of everything would predict infinitely many different instances of us in a way that doesn’t allow for uniquely locating ourselves.
However… this type of reasoning doesn’t look anything like that. If SSA/SSSA require us to have a complete working theory of everything in order to be usable, that’s still invalidating for my current purposes.
For the record, I ran into a more complicated problem which turns out to be incoherent for similar reasons- namely, information can only propagate in specific ways, and it turns out that SSA/SSA allows you to draw conclusions about what your reference class looks like in ways that defy the ways in which information can propagate.
You are affected by the initial set up. If the clone counterfactually saw something else, this wouldn’t affect you according to SIA.
This specific hypothetical doesn’t directly apply to the SIA- it relies on adjusting the relative frequencies of different types of observers in your reference class, which isn’t possible using SIA. SIA still suffers from the similar problem of allowing you to draw conclusions about what the space of all possible observers looks like.

Dach 6 Oct 2020 22:01 UTC
1 point
in reply to: ike’s comment on: Not Even Evidence
I don’t understand why you’re calling a prior “inference”. Priors come prior to inferences, that’s the point.
SIA is not isomorphic to “Assign priors based on Kolmogorov Complexity”. If what you mean by SIA is something more along the lines of “Constantly update on all computable hypotheses ranked by Kolmogorov Complexity”, then our definitions have desynced.
Also, remember: you need to select your priors based on inferences in real life. You’re a neural network that developed from scatted particles- your priors need to have actually entered into your brain at some point.
Regardless of whether your probabilities entered through your brain under the name of a “prior” or an “update”, the presence of that information still needs to work within our physical models and their conclusions about the ways in which information can propagate.
SIA has you reason as if you were randomly selected from the set of all possible observers. This is what I mean by SIA, and is a distinct idea. If you’re using SIA to gesture to the types of conclusions that you’d draw using Solomonoff Induction, I claim definition mismatch.
It clearly is unnecessary—nothing in your examples requires there to be tiling, you should give an example with a single clone being produced, complete with the priors SIA gives as well as your theory, along with posteriors after Bayesian updating.
I specifically listed the point of the tiling in the paragraph that mentions tiling:
for you to agree that the fact you don’t see a pink pop-up appear provides strong justified evidence that none of the probes saw <event x>
The point of that the tiling is, as I have said (including in the post), to manipulate the relative frequencies of actually existent observers strongly enough to invalidate SSA/SSSA in detail.
I don’t see any such implications. You need to simplify and more fully specify your model and example.
There’s phenomena which your brain could not yet have been impacted by, based on the physical ways in which information propagates. If you think you’re randomly drawn from the set of all possible observers, you can draw conclusions about what the set of all possible observers looks like, which is problematic.
I don’t see any such implications. You need to simplify and more fully specify your model and example.
Just to reiterate, my post isn’t particularly about SIA. I showed the problem with SSA/SSSA- the example was specified for doing something else.