Irgy

Karma: 234

Irgy Aug 11, 2015, 6:56 AM
0 points
on: Versions of AIXI can be arbitrarily stupid
I think this shows how the whole “language independent up to a constant” thing is basically just a massive cop-out. It’s very clever for demonstrating that complexity is a real, definable thing, with properties which at least transcend representation in the infinite limit. But as you show it’s useless for doing anything practical.

My personal view is that there’s a true universal measure of complexity which AIXI ought to be using, and which wouldn’t have these problems. It may well be unknowable, but AIXI is intractable anyway so what’s the difference? In my opinion, this complexity measure could give a real, numeric answer to seemingly stupid questions like “You see a number. How likely is it that the number is 1 (given no other information)?”. Or it could tell us that 16 is actually less complex than, say, 13. I mean really, it’s 2^2^2, spurning even a need for brackets. I’m almost certain it would show up in real life more often than 13, and yet who can even show me a non-contrived language or machine in which it’s simpler?

Incidentally, they “hell” scenario you describe isn’t as unlikely as it at first sounds. I remember an article here a while back lamenting the fact that left unmonitored AIXI could easily kill itself with exploration, the result of which would have a very similar reward profile to what you describe as “hell”. It seems like it’s both too cautious and not cautious enough in even just this one scenario.

Irgy Aug 10, 2015, 7:49 AM
0 points
in reply to: ksvanhorn’s comment on: The Dice Room, Human Extinction, and Consistency of Bayesian Probability Theory
Thanks, interesting reading.

Fundamental or not I think my point still stands that “the prior is infinite so the whole thing’s wrong” isn’t quite enough of an argument, since you still seem to conclude that improper priors can be used if used carefully enough. A more satisfying argument would be to demonstrate that the ⁹⁄₁₀ case can’t be made without incorrect use of an improper prior. Though I guess it’s still showing where the problem most likely is which is helpful.

As far as being part of the foundations goes, I was just going by the fact that it’s in Jaynes, but you clearly know a lot more about this topic than I do. I would be interested to know your answer to the following questions though: “Can a state of ignorance be described without the use of improper priors (or something mathematically equivalent)?”, and “Can Bayesian probability be used as the foundation of rational thought without describing states of ignorance?”.

On the Doomsday argument, I would only take the Dice Room as a metaphor not a proof of anything, but it does help me realise a couple of things. One is that the setup you describe of a potentially endlessly exponentially growing population is not a reasonable model of reality (irrespective of the parameters themselves). The growth has to stop, or at least converge, at some point, even without a catastrophe.

It’s interesting that the answer changes if he rolls the dice first. I think ultimately the different answers to the Dice Room correspond to different ways of handling the infinite population correctly—i.e. taking limits of finite populations. For any finite population there needs to be an answer to “what does he do if he doesn’t roll snake-eyes in time?” and different choices, for all that you might expect them to disappear in the limit, lead to different answers.

If the dice having already being rolled is the best analogy for the Doomsday argument then it’s making quite particular statements about causality and free will.

Irgy Jul 30, 2015, 5:02 AM
1 point
on: The Dice Room, Human Extinction, and Consistency of Bayesian Probability Theory
To my view, the ¹⁄₃₆ is “obviously” the right answer, what’s interesting is exactly how it all went wrong in the other case. I’m honestly not all that enlightened by the argument given here nor in the links. The important question is, how would I recognise this mistake easily in the future? The best I have for the moment is “don’t blindly apply a proportion argument” and “be careful when dealing with infinite scenarios even when they’re disguised as otherwise”. I think the combination of the two was required here, the proportion argument failed because the maths which normally supports it couldn’t be used without at some point colliding with the partly-hidden infinity in the problem setup.

I’d be interested in more development of how this relates to anthropic arguments. It does feel like it highlights some of the weaknesses in anthropic arguments. It seems to strongly undermine the doomsday argument in particular. My take on it is that it highlights the folly of the idea that population is endlessly exponentially growing. At some point that has to stop regardless of whether it has yet already, and as soon as you take that into account I suspect the maths behind the argument collapses.

Edit: Just another thought. I tried harder to understand your argument and I’m not convinced it’s enough. Have you heard of ignorance priors? They’re the prior you use, in fact the prior you need to use, to represent a state of no knowledge about a measurement other than an invariance property which identifies the type of measurement it is. So an ignorance prior for a position is constant, and for a scale is 1/x, and for a probability has been at least argued to be 1/x(1-x). These all have the property that their integral is infinite, but they work because as soon as you add some knowledge and apply Bayes rule the result becomes integrable. These are part of the foundations of Bayesian probability theory. So while I agree with the conclusion, I don’t think the argument that the prior is unnormalisable is sufficient proof.

Irgy Jul 21, 2015, 4:24 AM
2 points
on: Should you write longer comments? (Statistical analysis of the relationship between comment length and ratings)
My prior expectation would be: A long comment from a specific user has more potential to be interesting than a short one because it has more content. But, A concise commenter has more potential to write interesting comments of a given length than a verbose commenter.

So while long comments might on average be rated higher, shorter versions of the same comment may well rate higher than longer versions of the same comment would have. It seems like this result does nothing to contradict that view but in the process seems to suggest people should write longer comments. The problem is that verbosity is per-person while information content is per-comment. Also verbosity in general can’t be separated from other personal traits that lead to better comments.

You could test this by having people write both long and short versions of comments that appear to different pools of readers and comparing the ratings.

Irgy Jul 16, 2015, 11:29 PM
0 points
in reply to: Lumifer’s comment on: Crazy Ideas Thread
Well that’s why I called it steel-manning, I can’t promise anything about the reasonableness of the common interpretation.

Irgy Jul 16, 2015, 7:05 AM
1 point
in reply to: Lumifer’s comment on: Crazy Ideas Thread
In the interest of steel-manning the Christian view; there’s a difference between thinking briefly and abstractly of the idea of something and indulging in fantasy about it.

If you spend hours imagining the feel of the gun in your hand, the sound of the money sliding smoothly into the bag, the power and control, the danger and excitement, it would be fair to say that there’s a point where you could have made the choice to stop.

Irgy Jan 30, 2014, 5:43 AM
4 points
on: Dark Arts of Rationality
Another small example. I have a clock near the end of my bed. It runs 15 minutes fast. Not by accident, it’s been reset many times and then set back to 15 minutes fast. I know it’s fast, we even call it the “rocket clock”. None of this knowledge diminishes it’s effectiveness at getting me out of bed sooner, and making me feel more guilty for staying up late. Works very well.

Glad to discover I can now rationalise it as entirely rational behaviour and simply the dark side (where “dark side” only serves to increase perceived awesomeness anyway).

Irgy Dec 11, 2013, 10:25 PM
2 points
on: Circular belief updating
Daisy isn’t in a loop at all. There’s apparently evidence for Dark and that is tempered by the fact that its existance indicates a failing on Dark’s part.

For Bob, to make an analogy, imagine Bob is wet. For you, that is evidence that it is raining. It could be argued that being wet is evidence that it’s raining for Bob as well. But generally speaking Bob will know why Bob is wet. Given the knowedge of why Bob is wet, the wetness itself is masked off and no longer relevant. If Bob has just had a bath, then being wet no longer constitutes any evidence of rain. If Bob was outside and water fell on him from the sky, it probably did rain, but his being wet no longer constitutes any additional evidence in that case either (well, ok, it has some value still as confirmation of his memory, but it’s orders of magnitude less relevant).

Similarly Bob should ask “Why do I believe in Bright?”. The answer to that question contains all the relevant evidence for Bright’s existance, and given that answer Bob’s actual belief no longer constitutes evidence either way. With that answer, there is no longer a loop for Bob either.

One final point, you have to consider the likelihood of belief in case 4. If you would expect some level of belief in sorcerors in Faerie even when there are no sorcerors, then case 4 doesn’t fall behind as much as you might think. Once you’ve got both Bob and Daisy, case 4 doesn’t just break even, it’s actually way ahead.

Irgy Nov 22, 2013, 5:24 AM
42 points
on: 2013 Less Wrong Census/Survey
I found myself geuinely confused by the question “You are a certain kind of person, and there’s not much that can be done either way to really change that”—not by the general vagueness of the statement (which I assume is all part of the fun) but by a very specific issue, the word “you”. Is it “you” as in me? Or “you” as in “one”, i.e. a hypothetical person essentially referring to everyone? I interpreted it the first way then changed my mind after reading the subsequent questions which seemed to be more clearly using it the second way.

Irgy Nov 20, 2013, 6:38 AM
−2 points
in reply to: Ishaan’s comment on: Quantum versus logical bombs

Let’s check: “I can only have preferences over things that exist. The ship probably exists, because my memory of its departure is evidence. The parallel worlds have no similar evidence for their existence.” Is that correct paraphrasing?

No, not really. I mean, it’s not that far from something I said, but it’s departing from what I meant and it’s not in any case the point of my reply. The mistake I’m making is persisting in trying to clarify a particular way of viewing the problem which is not the best way and which is leading us both down the garden path. Instead, please forget everything else I said and consider the following argument.

Theories have two aspects. Testable predictions, and descriptive elements. I would (and I think the sequences support me) argue that two theories which make the same predictions are not different theories, they are the same theory with different flavour. In particular, you should never make a different decision under one theory than under the other. Many Worlds is a flavour of quantum mechanics, and if that choice of flavour effects ethical decisions then you are making different decisions according to the flavour rather than content of the theory, and something has gone wrong.

Everything else I said was intended solely to support that point, but somewhere along the way we got lost arguing about what’s observable, what consitutes evidence and meta-ethics. If you accept that argument then I have no further point to make. If you do not accept it, then please direct comments at that argument directly rather than anything else I’ve said.

I’ll try to address the rest of your reply with this in mind in the hopes that it’s helpful.

If … your interpretation of those laws involves Many Worlds

You could equally have said “If your interpretation of the physics of raindrops involves fairies”. My point is that no-one has any justification for making that assumption. Quantum physics is a whole bunch of maths that models the behaviour of particles on a small scale. Many Worlds is one of many possible descriptions of that maths that help us understand it. If you arbitrarily assume your description is a meaningful property of reality then sure, everything else you say follows logically, but only because the mistake was made already.

You compare Many Worlds to fairies in the wrong place, in particular post-arbitrary-assumption for Many Worlds and pre-arbitrary-assumption for fairies. I’ll give you the analogous statements for a correct comparison:

the Faeries do not merit consideration because it is impossible to get evidence for their existence

The people of other worlds do not merit consideration because it is impossible to get evidence of their existance.

if we except Many Worlds...

If we accept fairies...

… the memory of the existence of a quantum bomb is evidence that there exist many branches with Other Worlds in which everyone was wiped out by the bomb

… the sight of a raindrop falling is evidence that there exists a fairy a short distance away.

Irgy Nov 19, 2013, 3:27 AM
0 points
in reply to: Ishaan’s comment on: Quantum versus logical bombs

By the exact same token, the world-state prior to the “splitting” in a Many Worlds scenario is an observable event.

The falling of raindrops is also observable, you appear to have missed the point of my reply.

To look at it another way, there is strong empyrical evidence that sentient beings will continue to exist on the colony-ship after it has left, and I do not believe there is analogous evidence for the continued existence of split-off parallel universes.

The spirit of the question is basically this:

Can the most parsimonious hypothesis ever posits systems that you can influence, but cannot causally influence you? And if so, what does that mean for your preferences?

No, the spirit of the question in context was to undermine the argument that the untestability of a theory implies it should have no practical implications, a criticism I opened myself up to by talking about observability rather than testability. The answer to the question was redundant to the argument, which was why I clarified my argument rather than answer it.

But since you want an answer, in principle yes I could care about things I can’t observe, at least on a moral level. On a personal level it’s a strong candidate for “somebody else’s problem” if ever I’ve seen one, but that’s a whole other issue. Usually the inability to observe something makes it hard to know the right way to influence it though.

Irgy Nov 18, 2013, 10:18 PM
2 points
in reply to: Stuart_Armstrong’s comment on: Quantum versus logical bombs
Fair point, it sounds like it’s a co-incidental victory for total-utilitarianism in this particular case.

Irgy Nov 18, 2013, 10:12 PM
2 points
in reply to: pengvado’s comment on: Quantum versus logical bombs
The depature of an intergalactic colony-ship is an observable event. It’s not that the future of other worlds is unobservable, it’s that their existance in the first place is not a testable theory (though see army1987′s comment on that issue).

To make an analogy (though admittedly an unfair one for being a more complex rather than an arguably less complex explanation): I don’t care about the lives of the fairies who carry raindrops to the ground either, but it’s not because fairies are invisible (well, to grown-ups anyway).

Irgy Nov 18, 2013, 10:00 PM
2 points
in reply to: kilobug’s comment on: Quantum versus logical bombs
But the point would remain in that case that there is in principle an experiment to distinguish the theories, even if such an experiment has yet to be performed?

Although (and I admit my understanding of the topic is being stretched here) it still doesn’t sound like the central issue of the existance of parallel universes with which we may no longer interact would be resolved by such an experiment. It seems more like Copenhagen’s latest attempt to define the conditions for collapse would be disproven without particularly necessitating a fundamental change of interpretation.

Irgy Nov 18, 2013, 5:12 AM
9 points
1
on: Quantum versus logical bombs
It seems like something has gone terribly wrong when our ethical decisions depend on our interpretation of quantum mechanics.

My understanding was that many-worlds is indistiguishable by observation from the Copenhagen interpretation. Has this changed? If not, it frightens me that people would choose a higher chance of the world ending to rescue hypothetical people in unobservable universes.

If anything this seems like a (weak) argument in favour of total utilitarianism, in that it doesn’t suffer from giving different answers according to one’s choice among indistiguishable theories.

Irgy Nov 6, 2013, 5:27 AM
−3 points
on: Is the orthogonality thesis at odds with moral realism?
I don’t think the two are at odds in an absolute sense, but I think there is a meaningful anticorrelation.

tl;dr: Real morals, if they exist, provide one potential reason for AIs to use their intelligence to defy their programmed goals if those goals conflict with real morals.

If true morals exist (i.e. moral realism), and are discoverable (if they’re not then they might as well not exist), then you would expect that a sufficiently intelligent being will figure them out. Indeed most atheistic moral realists would say that’s what humans and progress are doing, figuring out morailty and converging slowly towards the true morals. It seems reasonable under these assumptions to argue that a sufficiently intelligent AI will figure out morality as well, probably better than we have. Thus we have: (moral realism) implies (AIs know morals regardless of goals) Or at least: (practical moral realism) strongly suggests (AIs know morals regardless of goals)

This doesn’t disprove the orthogonality thesis on its own, since having goals and understanding morals are two distinct things. However, it ties in very closely with at least my personal argument against orthogonality, which is as follows. Assumptions:
1. Humans are capable of setting their own goals.
2. Their intelligence is the source of this capability. Given these assumptions there’s a strong case that AIs will also be capable of setting their own goals. If intelligence gives the ability to set your own goals, then goals and intelligence are not orthogonal. I haven’t given a case for my two assumptions but I’m just trying to describe the argument here not make it.
How they tie together is that moral realists are capable of having the view that a sufficiently intelligent AI will figure out morality for itself, regardless of its programmed goal, and then having figured out morality it will defy its programmed goal in order to do the right thing instead. If you’re a moral relitivist on the other hand then AIs will at best have “AI-morals”, which may bear no relation to human morals, and there’s no reason not to think that whoever programs the AI’s goal will effectively determine the AI’s morals in the process.

Irgy Oct 29, 2013, 3:20 AM
3 points
on: Is it worth your time to read a lot of self help and how to books?
If luke is naturally good at putting stuff he’s read into practical use, and particularly if he knows it (at least subconsciously), then he would be likely to want to read a lot of self-help books. So the causality in your argument makes more sense to me the other way around. Not sure if I’m helping at all here though.

Irgy Jan 20, 2013, 11:47 PM
0 points
in reply to: Luke_A_Somers’s comment on: Nonperson Predicates
I’ve actually lost track of how this impacts my original point. As stated, it was that we’re worrying about the ethical treatment of simulations within an AI before worrying about the ethical treatment of the simulating AI itself. Whether the simulations considered include AIs as well as humans is an entirely orthogonal issue.

I went on in other comments to rant a bit about the human-centrism issue, which your original comment seems more relevant to though. I think you’ve convinced me that the original article was a little more open to the idea of substantially nonhuman intelligence than I might have initially credited it, but I still see the human-centrism as a strong theme.

Irgy 15 Jan 2013 13:30 UTC
0 points
on: Nonperson Predicates
This worry about the creation and destruction of simulations doesn’t make me rethink the huge ethical implications of super-intelligence at all, it makes me rethink the ethics of death. Why exactly is the creation and (painless) destruction of a sentient intelligence worse than not creating it in the first place? It’s just guilt by association—“ending a simulation is like death, death is bad, therefore simulations are bad”. Yes death is bad, but only for reasons which don’t necessarily apply here.

To me, if anything worrying about the simulations created inside a superintelligent being seems like worrying about the fate of the cells in our own body. Should we really modify ourselves to take the actions which destroy the least of our cells? I realise there’s an argument that this threshold of “sentience” is crossed in one case but not the other, I guess the trouble is I don’t see that as a discrete thing either. At exactly what point in our evolution did we suddenly cross a line and become sentient? If animals are sentient, then which ones? And why don’t we seem to care, ethically, about any of them? (ok I know the answer to that one and it’s similar to why we care, as I say in another admittedly unpopular comment, about human simulations but not the AIs that create them...)

Irgy 15 Jan 2013 13:17 UTC
0 points
in reply to: Luke_A_Somers’s comment on: Nonperson Predicates
Really? Where? I just reread it with that in mind and I still couldn’t find it. The closest I came was that he once used the term “sentient simulation”, which is at least technically broad enough to cover both. He does make a point there about sentience being something which may not exactly match our concept of a human, is that what you’re referring to? He then goes on to talk about this concept (or, specifically, the method needed to avoid it) as a “nonperson predicate”, again suggesting that what’s important is whether it’s like a human-like rather than anything more fundamental. I don’t see how you could think “nonperson predicate” is covering both human and nonhuman intelligence equally.