dogiv

Karma: 173

dogiv Jun 23, 2017, 3:50 PM
0 points
in reply to: Screwtape’s comment on: Open thread, June. 19 - June. 25, 2017
Are you talking about a local game in NY or a correspondence thing?

dogiv Jun 23, 2017, 2:29 PM
0 points
in reply to: turchin’s comment on: Open thread, June. 19 - June. 25, 2017
I like the first idea. But can we really guarantee that after changing its source code to give itself maximum utility, it will stop all other actions? If it has access to its own source code, what ensures that its utility is “maximum” when it can change the limit arbitrarily? And if all possible actions have the same expected utility, an optimizer could output any solution—”no action” would be the trivial one but it’s not the only one.

An AI that has achieved all of its goals might still be dangerous, since it would presumably lose all high-level executive function (its optimization behavior) but have no incentive to turn off any sub-programs that are still running.

Both proposals have the possible failure mode that the AI will discover or guess that this mechanism exists, and then it will only care about making sure it gets activated—which might mean doing bad enough things that humans are forced to open the box and shut it down.

dogiv Jun 22, 2017, 2:06 PM
2 points
in reply to: Viliam’s comment on: Open thread, June. 19 - June. 25, 2017
It seems like the ideal leisure activities, then, should combine the social games with games against nature. Sports do this to some extent, but the “game against nature” part is mostly physical rather than intellectual.

Maybe we could improve on that. I’m envisioning some sort of combination of programming and lacrosse, where the field reconfigures itself according to the players’ instructions with a 10-second delay...

But more realistically, certain sports are more strategic and intellectual than others. I’ve seen both tennis and fencing mentioned as sports that involve quick strategic thinking and predicting your opponent, although they lack the team element that lets you build coordination skills. Maybe some kind of group fencing would be good… or doubles tennis?

dogiv Jun 22, 2017, 1:46 PM
2 points
in reply to: turchin’s comment on: Existential risk from AI without an intelligence explosion
AI is good at well-defined strategy games, but (so far) bad at understanding and integrating real-world constraints. I suspect that there are already significant efforts to use narrow AI to help humans with strategic planning, but that these remain secret. For an AGI to defeat that sort of human-computer combination would require considerably superhuman capabilities, which means without an intelligence explosion it would take a great deal of time and resources.

dogiv Jun 21, 2017, 7:37 PM
7 points
in reply to: Lumifer’s comment on: S-risks: Why they are the worst existential risks, and how to prevent them
More like driving to the store and driving into the brick wall of the store are adjacent in design space.

dogiv Jun 21, 2017, 7:35 PM
3 points
in reply to: cousin_it’s comment on: S-risks: Why they are the worst existential risks, and how to prevent them
Yes, many people intuitively feel that a universe of pleasure and a universe of pain add to a net negative. But I suspect that’s just a result of experiencing (and avoiding) lots of sources of extreme pain in our lives, while sources of pleasure tend to be diffuse and relatively rare. The human experience of pleasure is conjunctive because in order to survive and reproduce you must fairly reliably avoid all types of extreme pain. But in a pleasure-maximizing environment, removing pain will be a given.

It’s also true that our brains tend to adapt to pleasure over time, but that seems simple to modify once physiological constraints are removed.

dogiv Jun 21, 2017, 1:18 PM
−1 points
in reply to: cousin_it’s comment on: S-risks: Why they are the worst existential risks, and how to prevent them
Human disutility includes more than just pain too. Destruction of the humanity (the flat plain you describe) carries a great deal of negative utility for me, even if I disappear without feeling any pain at all. There’s more disutility if all life is destroyed, and more if the universe as a whole is destroyed… I don’t think there’s any fundamental asymmetry. Pain and pleasure are the most immediate ways of affecting value, and probably the ones that can be achieved most efficiently in computronium, so external states probably don’t come into play much at all if you take a purely utilitarian view.

dogiv Jun 15, 2017, 8:02 PM
0 points
in reply to: Lumifer’s comment on: Bet or update: fixing the will-to-wager assumption
I’m not sure what you mean here by risk aversion. If it’s not loss aversion, and it’s not due to decreasing marginal value, what is left?

Would you rather have $5 than a 50% chance of getting $4 and a 50% chance of getting $7? That, to me, sounds like the kind of risk aversion you’re describing, but I can’t think of a reason to want that.

dogiv Jun 15, 2017, 4:22 PM
0 points
in reply to: Lumifer’s comment on: Bet or update: fixing the will-to-wager assumption
You will not bet on just one side, you mean. You already said you’ll take both bets because of the guaranteed win. But unless your credence is quite precisely 50%, you could increase your expected value over that status quo (guaranteed $1) by choosing NOT to take one of the bets. If you still take both, or if you now decide to take neither, it seems clear that loss aversion is the reason (unless the amounts are so large that decreasing marginal value has a significant effect).

dogiv Jun 15, 2017, 12:09 AM
0 points
in reply to: Lumifer’s comment on: Bet or update: fixing the will-to-wager assumption
True, you’re sure to make money if you take both bets. But if you think the probability is 51% on odd rather than 50%, you make a better expected value by only taking one side.

dogiv Jun 14, 2017, 9:17 PM
0 points
in reply to: Lumifer’s comment on: Bet or update: fixing the will-to-wager assumption
Let’s reverse this and see if it makes more sense. Say I give you a die that looks normal, but you have no evidence about whether it’s fair. Then I offer you a two-sided bet: I’ll bet $101 to your $100 that it comes up odd. I’ll also offer $101 to your $100 that it comes up even. Assuming that transaction costs are small, you would take both bets, right?

If you had even a small reason to believe that the die was weighted towards even numbers, on the other hand, you would take one of those bets but not the other. So if you take both, you are exhibiting a probability estimate of exactly 50%, even though it is “uncertain” in the sense that it would not to make evidence to move that estimate.

dogiv Jun 13, 2017, 8:10 PM
1 point
in reply to: MaryCh’s comment on: Stupid Questions June 2017
Gasoline is an excellent example of this behavior. It consists of a mixture of many different non-polar hydrocarbons with varying densities, some of which would be gaseous outside of solution. It stays mixed indefinitely (assuming you don’t let the volatile parts escape) because separation would require a reduction in entropy.

dogiv Jun 6, 2017, 2:56 AM
4 points
in reply to: Stuart_Armstrong’s comment on: Futarchy, Xrisks, and near misses
It seems like there’s also an issue with risk aversion. In regular betting markets there are enough bets that you can win some and lose some, and the risks can average out. But if you bet substantially on x-risks, you will get only one low-probability payout. Even if you assume you’ll actually get that one (relatively large) payout, the marginal value will be greatly decreased. To avoid that problem, people will only be willing to bet small amounts on x-risks. The people betting against them, though, would be willing to make a variety of large bets (each with low payoff) and thereby carry almost no risk.

dogiv Apr 3, 2017, 2:06 PM
0 points
in reply to: denimalpaca’s comment on: Naturally solved problems that are easy to verify but that would be hard to compute
I guess where we disagree is in our view of how a simulation would be imperfect. You’re envisioning something much closer to a perfect simulation, where slightly incorrect boundary conditions would cause errors to propagate into the region that is perfectly simulated. I consider it more likely that if a simulation has any interference at all (such as rewinding to fix noticeable problems) it will be filled with approximations everywhere. In that case the boundary condition errors aren’t so relevant. Whether we see an error would depend mainly on whether there are any (which, like I said, is equivalent to asking whether we are “in” a simulation) and whether we have any mechanism by which to detect them.

dogiv Mar 31, 2017, 5:08 PM
0 points
in reply to: denimalpaca’s comment on: Naturally solved problems that are easy to verify but that would be hard to compute
If it is the case that we are in a “perfect” simulation, I would consider that no different than being in a non-simulation. The concept of being “in a simulation” is useful only insofar as it predicts some future observation. Given the various multiverses that are likely to exist, any perfect simulation an agent might run is probably just duplicating a naturally-occurring mathematical object which, depending on your definitions, already “exists” in baseline reality.

The key question, then, is not whether some simulation of us exists (nearly guaranteed) but how likely we are to encounter an imperfection or interference that would differentiate the simulation from the stand-alone “perfect” universe. Once that happens, we are tied in to the world one level up and should be able to interact with it.

There’s not much evidence about the likelihood of a simulation being imperfect. Maybe imperfect simulations are more common than perfect ones because they’re more computationally tractable, but that’s not a lot to go on.

dogiv Mar 28, 2017, 7:54 PM
3 points
on: Elon Musk launches Neuralink, a venture to merge the human brain with AI
Does anybody think this will actually help with existential risk? I suspect the goal of “keeping up” or preventing irrelevance after the onset of AGI is pretty much a lost cause. But maybe if it makes people smarter it will help us solve the control problem in time.

dogiv Mar 28, 2017, 4:00 PM
3 points
in reply to: lifelonglearner’s comment on: Open thread, Mar. 27 - Apr. 02, 2017
I just tried this out for a project I’m doing at work, and I’m finding it very useful—it forces me to think about possible failure modes explicitly and then come up with specific solutions for them, which I guess I normally avoid doing.

dogiv Mar 27, 2017, 9:42 PM
0 points
in reply to: Lumifer’s comment on: Act into Uncertainty
Encrypting/obscuring it does help a little bit, but doesn’t eliminate the problem, so it’s not just that.

dogiv Mar 27, 2017, 4:16 PM
2 points
in reply to: Viliam’s comment on: Act into Uncertainty
I agree with that… personally I have tried several times to start a private journal, and every time I basically end up failing to write down any important thoughts because I am inhibited by the mental image of how someone else might interpret what I write—even though in fact no one will read it. Subconsciously it seems much more “defensible” to write nothing at all, and therefore effectively leave my thoughts unexamined, than to commit to having thought something that might be socially unacceptable.

dogiv Mar 24, 2017, 2:26 PM
0 points
in reply to: madhatter’s comment on: Making equilibrium CDT into FDT in one+ easy step
I’ve been trying to understand the differences between TDT, UDT, and FDT, but they are not clearly laid out in any one place. The blog post that went along with the FDT paper sheds a little bit of light on it—it says that FDT is a generalization of UDT intended to capture the shared aspects of several different versions of UDT while leaving out the philosophical assumptions that typically go along with it.

That post also describes the key difference between TDT and UDT by saying that TDT “makes the mistake of conditioning on observations” which I think is a reference to Gary Drescher’s objection that in some cases TDT would make you decide as if you can choose the output of a pre-defined mathematical operation that is not part of your decision algorithm. I am still working on understanding Wei Dai’s UDT solution to that problem, but presumably FDT solves it in the same way.