Thanks for the suggestion, Patrick. I’ve now adapted Tsvi’s formal argument to the reframed “5-equals-10 problem” and added it into the last section of my writeup.
Quinn
Karma: 285
Because the length of Scott’s Moloch post greatly exceeds my working memory (to the extent that I had trouble remembering what the point was by the end) I made these notes. I hope this is the right place to share them.
Notes on Moloch (ancient god of child sacrifice)
http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
Intro—no real content.
Moloch as coordination failure: everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status.
10 examples: Prisoner’s Dilemma, dollar auctions, fish-farming story (tragedy of the commons), Malthusian trap, ruthless/exploitative Capitalist markets, the two-income trap, agriculture, arms races, cancer, political race to the bottom (lowering taxes to attract business)
4 partial examples: inefficient education, inefficient science, government corruption (corporate welfare), Congress (representatives voting against good of nation for good of constituency)
Existing systems are created by incentive structures, not agents, e.g. Las Vegas caused by a known bias in human reward circuitry, not optimization for human values.
But sometimes we move uphill anyway. Possible explanations:
Excess resources / we are in the dream time and can afford non-competitive behavior.
Physical limitations to what can be sacrificed
Economic competition actually producing positive utility for consumers (but this is fragile)
Coordination, e.g. via governments, guilds, friendships, etc.
Technology/ingenuity creates new opportunities to fall into such traps. Technology overcomes physical limitations, consumes excess resources. Automation further decouples economic activity from human values. Technology can improve coordination, but can also exacerbate existing conflicts by giving all sides more power.
AGI opens up whole new worlds of traps: Yudkowsky’s paperclipper, Hanson’s subsistence-level ems, Bostrom’s Disneyland with no children.
6 & 7. Gnon—basically the god of the conservative scarcity mindset. Nick Land advocates compliance; Nyan wants to capture Gnon and build a walled garden. Scott warns that Moloch is far more terrifying than Gnon and will kill both of them anyway.
8 & 9. So we have to kill this Moloch guy, by lifting a better God to Heaven (Elua).
Cool.
For those who don’t want to wait until April 7th, Udacity is scheduled to launch their own Intro to Data Science on Feb 5th (this Wednesday). I expect it to be easier/shallower than the Hopkins/Coursera sequence, but it wins on actionability.
For a countable language L and theory T (say, with no finite models), I believe the standard interpretation of “space of all models” is “space of all models with the natural numbers as the underlying set”. The latter is a set with cardinality continuum (it clearly can’t be larger, but it also can’t be smaller, as any non-identity permutation of the naturals gives a non-identity isomorphism between different models).
Moreover this space of models has a natural topology, with basic open sets {M: M models phi} for L-sentences phi, so it makes sense to talk about (Borel) probability measures on this space, and the measures of such sets. (I believe this topology is Polish, actually making the space Borel isomorphic to the real numbers.)
Note that by Lowenheim-Skolem, any model of T admits a countable elementary substructure, so to the extent that we only care about models up to some reasonable equivalence, countable models (hence ones isomorphic to models on the naturals) are enough to capture the relevant behavior. (In particular, as pengvado points out, the Christiano et al paper only really cares about the complete theories realized by models, so models on the naturals suffice.)
For the sake of the absent gods do not start with an IDE or anything that hides the compilation step.
Could you say more about why this is important for beginning programmers?
While you’re certainly technically correct, it’s an easy/common mistake for people to focus on the “save all you can” part, overlooking “gain all you can” opportunities. The EA movement is notable for proactively trying to counter this mistake, and apparently so is John Wesley.
For the same reasons you outline above, I’m okay with fighting this hypothetical target.
If I must dignify the hypothesis with a strategy: my “buy” and “sell” prices for such a bet correspond to the inner and outer measures of the target, respectively.
In other words, “what is the measure of an unmeasurable set?”. The question is wrong.
Several popular comments say something to the effect of “I was too arrogant to just get with the program and cooperate with the other humans”.
The biggest of my own arrogant mistakes was not taking CS/programming very seriously while in college because I was dead set on becoming a mathematician, and writing code was “boring” compared to math. Further arrogance: I wasn’t phased by the disparity between the number of graduating Ph. D.’s and the number of academic jobs.
I found out in grad school that my level of talent in mathematics, while somewhat rare, was certainly not so exceedingly rare that real world considerations would not apply to me.
I’ve since changed my attitude, and I’m working on fixing this mistake.
GEB will take you from superficial knowledge to full grok.
A word of caution: there is a risk when reading popular science/math books like GEB of coming away feeling like one understands something at a higher level than one actually does, particularly if one hasn’t already studied the subject formally.
If one has formally studied incompleteness before, it’s easy to wave away standard primitive recursive derivations (e.g. the proof predicate) as tedious and trivial and beside the main point, but having this attitude the first time around could be dangerous.
I read GEB years ago, and recall liking it quite a bit, though I disagree with Louie Helm’s endorsement of GEB as a course reference, at least not without supplementation from a “standard” source like these notes (or any formal logic textbook).
Thanks!
Martin Gardner’s Mathematical Games column from Scientific American Volume 242, Number 6, June, 1980. Paywalled here:June:1980).
EDIT: escape characters
Robin Hanson has advocated this point of view.
I find the argument quite unconvincing; Hanson seems to be making the mistake of conflating “life worth living” with “not committing suicide” that is well addressed in MTGandP’s reply (and grandchildren).
The 52-karma top comment of the Virtual Employment thread has been deleted. I gather that it said something about copywriting, with online skill tests for prospective applicants.
Can anyone provide a bit more information about this apparently quite valuable comment?
Awesome, thanks!
Thanks for the nice comment. I listed the PD post first, as it is probably the most readable of the three, written more like an article than like notes.
Thanks for looking! I’ll try to get my hands on a physical copy, as the Google Books version has highly distracting page omissions.
Unless I’m going insane, Luke_A_Somers’ comment below is incorrect. If you defect and your opponent times out, you still get 1 point and they get 0, which is marginally better than both getting 1 point in the case of mutual defection.
That was the purpose my (sleep 9). I figured anyone who was going to eval me twice against anything other than CooperateBot is going to figure out that I’m a jerk and defect against me, so I’m only getting one point from them anyway. Might as well lower their score by 1.
My assumption might not have been correct though. In the original scoreboard, T (who times out against me) actually does cooperate with K, who defected against everybody!
The bizarre-looking always-false conditional was a long shot at detecting simulation. I heard an idea at a LW meetup (maybe from So8res?) that players might remove sleeps from each other’s code before calling eval. I figured I might as well fake such players out if there were any (there were not).
Yes, the “here’s why Quinn is wrong about CooperateBot being a troll submission” comments were valuable, so I don’t regret provoking them. Presumably if my comment had said “players” instead of “trolls” from the outset, it would have been ignored for being inoffensive and content-free.
But provoking a few good comments was a happy accident from my perspective. I will avoid casual use of the word “troll” on this site, unless I have a specific purpose in mind.
Thanks for pointing that out. My feeling is still “well yes, that’s technically true, but it still seems unnatural, and explosion is still the odd axiom out”.
Coq, for example, allows empty case expressions (for empty types), and I expect that other languages which double as proof assistants would support them as well… for the very purpose of satisfying explosion. General purpose languages like Haskell (and I just checked OCaml too) can seemingly overlook explosion/empty cases with few if any practical problems.