Having a top-level domain doesn’t make an entity a country. Lots of indisputably non-countries have top-level domains. Nobody thinks the Bailiwick of Guernsey is a country, and yet .gg exists.
asr
Karma: 1,676
To do that it’s going to need a decent sense of probability and expected utility. Problem is, OpenCog (and SOAR, too, when I saw it) is still based in a fundamentally certainty-based way of looking at AI tasks, rather than one focused on probability and optimization.
I don’t see why this follows. It might be that mildly smart random search, plus a theorem prover with a fixed timeout, plus a benchmark, delivers a steady stream of useful optimizations. The probabilistic reasoning and utility calculation might be implicit in the design of the “self-improvement-finding submodule”, rather than an explicit part of the overall architecture. I don’t claim this is particularly likely, but neither does undecidability seem like the fundamental limitation here.
But it would have a very hard time strengthening its core logic, as Rice’s Theorem would interfere: proving that certain improvements are improvements (or, even, that the optimized program performs the same task as the original source code) would be impossible.
This seems like the wrong conclusion to draw. Rice’s theorem (and other undecidability results) imply that there exist optimizations that are safe but cannot be proven to be safe. It doesn’t follow that most optimizations are hard to prove. One imagines that software could do what humans do—hunt around in the space of optimizations until one looks plausible, try to find a proof, and then if it takes too long, try another. This won’t necessarily enumerate the set of provable optimizations (much less the set of all enumerations), but it will produce some.
You might look into all the work that’s been done with Functional MRI analysis of the brain—your post reminds me of that. The general technique of “watch the brain and see which regions have activity correlated with various mental states” is a well known technique, and well enough known that all sorts of limitations and statistical difficulties have been pointed out (see wikipedia for citations.)
In other words, even if this is completely correct, it doesn’t disprove relativity. Rather, it disproves either relativity or most versions of utilitarianism—pick one.
It seems like all it shows is that we ought to keep our utility functions Lorentz-invariant. Or, more generally, when we talk about consequentialist ethics, we should only consider consequences that don’t depend on aspects of the observer that we consider irrelevant.
I’m curious if anyone has made substantial effort to reach a ‘flow’ state in tasks outside of coding, like reading or doing math etc etc., and what they learned. Are there easy tricks? Is it possible? Is flow just a buzzword that doesn’t really mean anything?
I find reading is just about the easiest activity to get into that state with. I routinely get so absorbed in a book that I forget to move. And I think that’s the experience of most readers. It’s a little harder with programming actually, since there are all these pauses while I wait for things to compile or run, and all these times when I have to switch to a web browser to look something up. With reading, you can just keep turning pages.
The canonical example is that of a child who wants to steal a cookie. That child gets its morality mainly from its parents. The child strongly suspects that if it asks, all parents will indeed confirm that stealing cookies is wrong. So it decides not to ask, and happily steals the cookie.
I find this example confusing. I think what it shows is that children (humans?) aren’t very moral. The reason the child steals instead of asking isn’t anything to do with the child’s subjective moral uncertainty—it’s that the penalty for stealing-before-asking is lower than stealing-after-asking, and the difference in penalty is enough to make “take the cookie and ask forgiveness if caught” better than “ask permission”.
I suspect this is related to our strong belief in being risk-averse when handing out penalties. If I think there’s a 50% chance my child misbehaved, the penalty won’t be 50% of the penalty if they were caught red-handed. Often, if there’s substantial uncertainty about guilt, the penalty is basically zero—perhaps a warning. Here, the misbehavior is “doing a thing you knew was wrong;” even if the child knows the answer in advance, when the child explicitly asks and is refused, the parent gets new evidence about the child’s state of mind, and this is the evidence that really matters.
I suspect this applies to the legal system and society more broadly as well—because we don’t hand out partial penalties for possible guilt, we encourage people to misbehave in ways that are deniable.
Without talking about utility functions, we can’t talk about expected utility maximization, so we can’t define what it means to be ideally rational in the instrumental sense
I like this explanation of why utility-maximization matters for Eliezer’s overarching argument. I hadn’t noticed that before.
But it seems like utility functions are an unnecessarily strong assumption here. If I understand right, expected utility maximization and related theorems imply that if you have a complete preference over outcomes, and have probabilities that tell you how decisions influence outcomes, you have implicit preferences over decisions.
But even if you have only partial information about outcomes and partial preferences, you still have some induced ordering of the possible actions. We lose the ability to show that there is always an optimal ‘rational’ decision, but we can still talk about instances of irrational decision-making.
I appreciate you writing this way—speaking for myself, I’m perfectly happy with a short opening claim and then the subtleties and evidence emerges in the following comments. A dialogue can be a better way to illuminate a topic than a long comprehensive essay.
High frequency stock trading.
The attack that people are worrying about involves control of a majority of mining power, not control of a majority of mining output. So the seized bitcoins are irrelevant. The way the attack works is that the attacker would generate a forged chain of bitcoin blocks showing nonsense transactions or randomly dropping transactions that already happened. Because they control a majority of mining power, this forged chain would be the longest chain, and therefor a correct bitcoin implementation would try to follow it, with bad effects. This in turn would break the existing bitcoin network.
The government almost certainly has enough compute power to mount this attack if they want.
I didn’t down-vote, but was tempted to. The original post seemed content-free. It felt like an attempt to start a dispute about definitions and not a very interesting one.
It had an additional flaw, which is that it presented its idea in isolation, without any context on what the author was thinking, or what sort of response the author wanted. It didn’t feel like it raised a question or answered a question, and so it doesn’t really contribute to any discussion.
The only reasons I can think of are your #1 and #2. But I think both are perfectly good reasons to vote...
Think about the continuum between what we have now and the free market (where you can control exactly where your money goes), and it becomes fairly clear that the only points which have a good reason to be used are the two extreme ends. If you advocate a point in the middle, you’ll have a hard time justifying the choice of that particular point, as opposed to one further up or down.
I don’t follow your argument here. We have some function that maps from “levels of individual control” to happiness outcomes. We want to find the maximum of this function. It might be that the endpoints are the max, or it might be that the max is in the middle.
Yes, it might be that there is no good justification for any particular precise value. But that seems both unsurprising and irrelevant. If you think that our utility function here is smooth, then sufficiently near the max, small changes in the level of social control would result in negligible changes in outcome. Once we’re near enough the maximum, it’s hard to tune precisely. What follows from this?
Eliezer thinks the phrase ‘worst case analysis’ should refer to the ‘omega’ case.
“Worst case analysis” is a standard term of art in computer science, that shows up as early as second-semester programming, and Eliezer will be better understood if he uses the standard term in the standard way.
A computer scientist would not describe the “omega” case as random—if the input is correlated with the random number source in a way that is detectable by the algorithm, they’re by definition not random.
Yes. Perhaps we might say, this is what middle school or high school science should be.
Likewise direct demonstrations are the sort of thing I wish science museums focused on more clearly. Often they have 75% of it, but the story of “this experiment shows X” gets lost in the “whoa, cool”. I’m in favor of neat stuff, but I wish they explained better what insight the viewer should have.
Juries have a lot of “professional supervision.” In the Common Law system, the judge restricts who can serve on the jury, determines the relevant law, tells the jury what specific question of fact they are deciding, controls the evidence shown to the jury, does the sentencing, and more. My impression is that the non-Common Law systems that use juries give them even less discretion. So when we have citizen-volunteers, we get good results only by very carefully hemming them in with professionals.
You can’t supervise the executive in the same way. By definition, the executive is the part of the government in control of the coercive apparatus. If the nominal executives aren’t able to give orders to the military without the approval of some other body, then the nominal executives aren’t really in charge; they’re just constitutional decoration, like the modern British monarchy, or the Presidium of the USSR.
I found this post hard to follow. It would be more intelligible if you gave a clearer explanation of what problem you are trying to solve. Why exactly is it bad to have the same people look for problems and fix them? Why is it bad to have a legislature that can revise and amend statutes during the voting process?
I also don’t really understand what sort of comment or feedback you are expecting here. Do you want us to discuss whether this lottery-and-many-committees structure is in general a good idea? Do you want us to critique through the details of your scheme?
The scheme seems to have certain advantages and certain disadvantages. I am personally quite skeptical; I would need to see lottery-based administration work at a small scale before I tried it on anything larger than a village.
What sort of evidence would convince you that this was, on balance, a bad idea?
I basically agree, but I think the point is stronger if framed differently:
Some defects in an argument are decisive, and others are minor. In casual arguments, people who nitpick are often unclear both to themselves and to others whether their objections are to minor correctable details, or seriously undermine the claim in question.
My impression is that mathematicians, philosophers, and scientists are conscious of this distinction and routinely say things like “the paper is a little sloppy in stating the conclusions that were proved, but this can be fixed easily” or “there’s a gap in the argument here and I think it’s a really serious problem.” Outside professional communities, I don’t see people make distinctions between fatal and superficial flaws in somebody else’s argument.
In summary: I think your post is a good one but with minor correctable flaws.
Doing an audit to catch all vulnerabilities is monstrously hard. But finding some vulnerabilities is a perfectly straightforward technical problem.
It happens routinely that people develop new and improved vulnerability detectors that can quickly find vulnerabilities in existing codebases. I would be unsurprised if better optimization engines in general lead to better vulnerability detectors.