eva_

Karma: 237

eva_Nov 21, 2022, 4:13 AM
7 points
0
in reply to: Shmi’s comment on: Decision Theory but also Ghosts
https://www.lesswrong.com/tag/functional-decision-theory argues for choosing as if you’re choosing the policy you’d follow in some situation before you learnt any of the relevant infortmation. In many games, having a policy of making certain choicese (that others could perhaps predict, and adjust their own choices accordingly) gets you better outcomes then just always doing what seems like a good idea ta the time. For example if someone credibly threatens you might be better off paying them to go away, but before you got the threat you would’ve prefered to commit yourself to never pay up so that people don’t threaten you in the first place.
A problem with arguments of the form “I expect that predictably not paying up will cause them not to threaten me” is that at the time you recieve the threat, you now know that argument to be wrong. They’ve proven to be somebody who still threatens you even though you do FDT, at which point you can simultaneously prove that refusing the threat doesn’t work and so you should pay up (because you’ve already seen the threat) and that you shouldn’t pay up for whatever FDT logic you were using before. Behaviour of agents who can prove a contradiction that is directly relevant to their decision function seems undefined. There needs to be some logical structure that lets you pick which information causes your choice, despite having enough in total to derive contradictions.
My alternative solution is that you aren’t convinced by the information you see, that they’ve actually already threatened you. It’s also possible you’re still inside their imagination as they decide whether to issue the threat. Whenever something is conditional on your actions in an epistemic state without being conditional on that epistemic state actually being valid (such as if someone predicts how you’d respond to a hypothetical threat before they issue it, knowing you’ll know it’s too late to stop when you get it) then there’s a ghost being lied to and you should think maybe you’re that ghost to justify ignoring the threat, rather than try to make decisions during a logically impossible situation.

eva_Nov 21, 2022, 2:20 AM
1 point
0
in reply to: hairyfigment’s comment on: Decision Theory but also Ghosts
I totally agree we can be coherently uncertain about logical facts, like whether P=NP. FDT has bigger problems then that.
When writing this I tried actually doing the thing where you predict a distribution, and only 21% of LessWrong users were persuaded they might be imaginary and being imagined by me, which is pretty low accuracy considering they were in fact imaginary and being imagined by me. Insisting that the experience of qualia can’t be doubted did come up a few times, but not as aggressively as you’re pushing it here. I tried to cover it in the “highly detailed internal subjective experience” counterargument, and in my introduction, but I could have been stronger on that.
I agree that the same argument on philosophers or average people would be much less successful even then that, but that’s a fact about them, not about the theory.

eva_Nov 21, 2022, 12:00 AM
1 point
0
in reply to: hairyfigment’s comment on: Decision Theory but also Ghosts
If you think you might not have qualia, then by definition you don’t have qualia.
What? just a tiny bit of doubt and your entire subjective conscious experience evaporates completely? I can’t see any mechanism that would do that, it seems like you can be real and have any set of beliefs or be fictional and have any set of beliefs. Something something map-territory distinction?
This just seems like a restatement of the idea that we should act as if we were choosing the output of a computation.
Yes, it is a variant of that idea, with different justifications that I think are more resilient. The ghosts of FDT agents still make the correct choices, they just have incoherent beliefs while they do it.

eva_Nov 20, 2022, 11:26 PM
3 points
0
in reply to: hairyfigment’s comment on: Decision Theory but also Ghosts
Actually, I don’t assume that, I’m totally ok with believing ghosts don’t have qualia. All I need is that they first-order believe they have qualia, because then I can’t take my own first-order belief I have qualia as proof I’m not a ghost. I can still be uncertain about my ghostliness because I’m uncertain in the accuracy of my own belief I have qualia, in explicit contradiction of ‘cogito ergo sum’. The only reason ghosts possibly having qualia is a problem is that then maybe I have to care about how they feel.

eva_Nov 20, 2022, 8:55 PM
3 points
−2
in reply to: Wuschel Schulz’s comment on: Decision Theory but also Ghosts
A valid complaint. I know the answer must be something like “coherent utility functions can only consist of preferences about reality” because if you are motivated by unreal rewards you’ll only ever get unreal rewards, but that argument needs to be convincing to the ghost too, whose got more confidence in her own reality. I know that e.g. in Bomb ghost-theory agents choose the bomb even if they think the predictor will simulate them a painful death, because they consider the small amount of money at much greater measure for their real selves to be worth it, but I’m not sure how they get to that position.

eva_Nov 15, 2022, 2:34 AM
3 points
0
in reply to: Chris_Leong’s comment on: Two New Newcomb Variants
They can though? Bomb box 5, incentivise box 1 or 2, bet on box 3 or 4. Since FDT’s strategy puts rewarding cooperative hosts above money or grenades, she picks the box that rewards the defector and thus incentivises all 4 to defect from that equilibrium. (I thought over her strategy for quite a bit and there are probably still problems with it but this isn’t one of them)

eva_Nov 15, 2022, 12:51 AM
5 points
1
on: Moral contagion heuristic
How much of this effect is from morality being causally contagious (associating with Evil people turns you Evil) vs. morality being evidientarily contagious (Evil people are more likely to choose to associate with Evil people)?
I’d expect that, all else being equal, organisations secretly run in evil ways will be more willing to secretly accept money from other evil people, for many reasons including that they’ve got higher expectation of how normal that sort of behaviour is. It seems harder to imagine how a good organisation choosing to take dirty money would corrupt itself in the process if it was being reasonably diligent. Even if the moral contagion argument is wrong from inside hypothetical-good-MITs perspective, and so they should take the money, from everyone elses perspective it’s still information we can update on.
If taking bad money for good causes is first-order good, because you’re doing good things with it, but other donors can notice and it lowers their confidence in how good you are (since bad causes are more willing to take bad money), then you might lose other support sufficient to make it not worthwhile. There’s probably some sort of signalling equilibria here, which is completely destroyed by the whole concept of accepting the money in secret. Hopefully actually good organisations wouldn’t do that sort of deontology violation and would just make their donor lists public?

eva_Nov 15, 2022, 12:45 AM
1 point
0
in reply to: Dacyn’s comment on: Two New Newcomb Variants
If the hosts move first logically, then TDT will lead to the same outcomes as CDT, since it’s in each host’s interest to precommit to incentivising the human to pick their own box
It’s in the hosts interests to do that if they think the player is CDT, but it’s not in their interests to commit to doing that. They don’t lose anything by retaining the ability to select a better strategy later after reading the players mind.

eva_Nov 14, 2022, 11:57 PM
1 point
0
in reply to: Chris_Leong’s comment on: Two New Newcomb Variants
Not whichever is lighter, one of whichever pair is heavier. Yes, I claim an EDT agent upon learning the rules, if they have a way to blind themself but not to force a commitment, will do this plan. They need to maximise the amount of incentive that the hosts have to put money in boxes, but to whatever extent they actually observe the money in boxes they will then expect to get the best outcome by making the best choice given their information. The only middle ground I could find is pairing the boxes, and picking one from the heavier pair. I’d be very happy to learn if I was wrong and there’s a better plan or if this doesn’t work for some reason.

eva_Nov 14, 2022, 11:39 PM
1 point
0
in reply to: Chris_Leong’s comment on: Two New Newcomb Variants
EDT isn’t precommiting to anything here, she does her opinion of the best choice at every step. That’s still a valid complaint that it’s unfair to give her a blindfold though. If CDT found out about the rules of the game before the hosts made their predictions, he’d make himself an explosive collar that he can’t take off and that automatically kills him unless he chooses the box with the least money, and get the same outcome as FDT, and EDT would do that as well. For the blindfold strategy EDT only needs to learn the rules before she sees whats in the boxes, and the technology required is much easier. I mostly wrote it this way because I think it’s a much more interesting way to get stronger behaviour than the commitment-collar trick.

eva_Nov 14, 2022, 8:42 PM
1 point
0
in reply to: Dacyn’s comment on: Two New Newcomb Variants
The hosts aren’t competing with the human, only each other, so even if the hosts move first logically they have no reason or opportunity to try to dissuade the player from whatever they’d do otherwise. FDT is underdefined in zero-sum symmetrical strategy games against psychological twins, since it can foresee a draw no matter what, but choosing optimal strategy to get to the draw still seems better than playing dumb strategies on purpose and then still drawing anyway.
Why do you think they should be $100 and $200? Maybe you could try simulating it?
What happens if FDT tries to force all the incentives into one box? If the hosts know exactly what every other host will predict, what happens to their zero-sum competition and their incentive to coordinate with FDT?

eva_Nov 12, 2022, 2:56 AM
1 point
0
in reply to: Victor Novikov’s comment on: Adversarial Priors: Not Paying People to Lie to You
Yes, that’s the intended point, and probably a better way of phrasing it. I am concluding against the initial assertion, and claiming that it does make sense to trust people in some situations even though you’re implementing a strategy that isn’t completely immune to exploitation.

eva_Nov 10, 2022, 4:05 AM
5 points
4
in reply to: gwern’s comment on: Adversarial Priors: Not Paying People to Lie to You
I don’t consider the randomized response technique lying, it’s mutually understood that their answer means “either I support X or both coins came up heads” or “either I support Y or both coins came up tails”. There’s no deception because you’re not forming a false belief and you both know the precise meaning of what is communicated.
I don’t consider penetration testing lying, you know that penetration testers exist and have hired them. It’s a permitted part of the cooperative system, in a way that actual scam artists aren’t.
What’s a word that means “antisocially decieve someone in a way that harms them and benefits you” such that everyone agrees it’s a bad thing for people to be incentivised to do? I want to be using that word but don’t know what it is.

eva_Nov 10, 2022, 3:01 AM
4 points
0
in reply to: ShowMeTheProbability’s comment on: Adversarial Priors: Not Paying People to Lie to You
Not sure what’s unclear here? I mean that you’d generally prefer not to have incentive structures where you need true information from other people and they can benefit at your loss by giving you false information. Paying someone to lie to you means creating an incentive for them to actually decieve you, not merely giving them money to speak falsehoods.

eva_Nov 9, 2022, 11:32 PM
4 points
2
on: A caveat to the Orthogonality Thesis
A sufficiently strong world model can answer the question “What would a very smart very good person think about X?” and then you can just pipe that to the decision output, but that won’t get you higher intelligence than what was present in the training material.
Shouldn’t human goals have to be in the within human intelligence part, since humans have them? Or are we considering exactly human intelligence AI unsafe? Do you imagine a slightly dumber version of yourself failing to actualise your goals from not having good strategies, or failing to even embed them due to having a world model that lacks definitions of objects you care about?
Corrigibility has to be in the reachable part of the goals because a well-trained dog genuinely wants to do what you want it to do, even if it doesn’t always understand, and even if following the command will get it less food than otherwise and this is knowable to the dog. You clearly don’t need human intelligence to describe the terminal goal “Do what the humans want me to do”, although it’s not clear the goal will stay there as intelligence rises above human intelligence.

eva_Nov 9, 2022, 10:49 PM
9 points
−7
in reply to: Yitz’s comment on: Is there any discussion on avoiding being Dutch-booked or otherwise taken advantage of one’s bounded rationality by refusing to engage?
You’re far more likely to be a background character than the protagonist in any given story, so a theory claiming you’re the most important person in a universe with an enormous number of people has an enormous rareness penalty to overcome before you should believe it instead of that you’re just insane or being lied to. Being in a utilitarian high-leverage position for the lives of billions can be overcome by reasonable evidence, but for the lives of 3^^^^3 people the rareness penalty is basically impossible to overcome. Even if the story is true, most of the observers will be witnessing it from the position of tied-to-the-track, not holding the lever, so if you’d assign low prior expectation to being in the tied-to-the-track part of the story, you should assign an enormous factor lower of being in the decision-making part of it.

eva_Nov 7, 2022, 10:10 AM
8 points
1
on: Is there any discussion on avoiding being Dutch-booked or otherwise taken advantage of one’s bounded rationality by refusing to engage?
You can dodge it by having a bounded utilityfunction, or if you’re utilitarian and good a function that is at most linear in anthropic experience.
If the mugger says “give me your wallet and I’ll cause you 3^^^^3 units of personal happiness” you can argue that’s impossible because your personal happiness doesn’t go that high.
If the mugger says “give me your wallet and I’ll cause 1 unit of happiness to 3^^^^3 people who you altruistically care about” you can say that, in the possible world where he’s telling the truth, there are 3^^^^3 + 1 people only one of which gets the offer and the others get the payout, so on priors it’s at least 1/3^^^^3 against for you to experience recieving an offer, and you should consider it proportionally unlikely.
I don’t think people realise how much astronomically more likely it is to truthfully be told “God created this paradise for you and your enormous circle of friends to reward an alien for giving him his wallet with zero valid reasoning whatsoever” than to be truthfully asked by that same Deity for your stuff in exchange for the distant unobservable happiness of countless strangers.
More generally, you can avoid most flavours of adversarial muggings with 2 rules: first don’t make any trade that an ideal agent wouldn’t make (because that’s always some kind of money pump), and second don’t make any trade that looks dumb. Not making trades can cost you in terms of missed opportunities, but you can’t adversarially exploit the trading strategy of a rock with “no deal” written on it.

eva_Nov 6, 2022, 5:37 AM
6 points
0
on: Open & Welcome Thread—November 2022
Hello, I read much of the background material over the past few years but am a new account. Not entirely sure what linked me here first but 70% guess is HPMoR. compsci / mathematics background. I have mostly joined due to having ideas slightly too big for my own brain to check and wanting feedback, wanting to infect the community that I get a lot of my memes from with my own original memes, having now read enough to feel like LessWrong is occasionally incorrect about things where I can help, and to improve my writing quality in ways that generalise to explaining things to other people.

eva_Nov 6, 2022, 2:47 AM
5 points
1
in reply to: Dagon’s comment on: Unpricable Information and Certificate Hell
If you instead say “evidence of”, this makes more sense
Accepted and changed, I’m only claiming some information/entanglement, not absolute proof.
It applies to all transactions (because all transactions are fundamentally about trust)
Would it be clearer to say “markets with perfect information”? The problem I’m trying to describe can only occur with incomplete information / imperfect trust, but doesn’t require so little information and trust that transactions become impossible in general. There’s a wide middleground of imperfect trust where all of real life happens, and we still do business anyway.
And this is where you lose me. Failure to add value is not an externality. Good competition (offering a more attractive transaction) is not a market failure.
It sure looks like an externality when generally terrible things can happen as a result. I agree that being able to offer a better product is good, and being able to incentivise that is good if it can lead to more better products, but it does also have this side problem that can be harmful enough to be worth considering.
Signaling is a competitive/adversarial game.
Yeah, I know this idea isn’t completely original / exists inside broader frameworks already, but I wanted to highlight it more specifically and I haven’t found anything identical to this before. Thanks for the feedback.

eva_Nov 3, 2022, 12:30 AM
5 points
0
in reply to: Dagon’s comment on: Is there a good way to award a fixed prize in a prediction contest?
Weighted by credence means you’re scored on Probability*Prediction, which isn’t a fair rule.
If I sincerely believe its 60:40 between two options, and I write that down, I expect 36+16=52 payout, but if I write 100:0 I expect 60+0=60 payout, putting more credence on higher probability outcomes gets me more money even in excess of my true beliefs.