I got the impression Eliezer’s claiming that a dangerous superintelligence is merely sufficient for nanotech.
How would you save us with nanotech? It had better be good given all the hardware progress you just caused!
Gram Stone
Karma: 160
A genuine congratulations for learning the rare skill of spotting and writing valid proofs.
Graham’s Number I see as ridiculous, apparently one of the answers to his original problem could be as low as a single digit number, why have power towers on power towers then?
Graham’s number is an upper bound on the exact solution to a Ramsey-type problem. Ramsey numbers and related generalizations are notorious for being very easy to define and yet very expensive to compute with brute-force search, and many of the most significant results in Ramsey theory are proofs of extraordinarily large upper bounds on Ramsey numbers. Graham would have proved a smaller bound if he could.
(In fact, as I understand it, the popular Graham’s number is slightly larger than the published result, but the published result is only slightly smaller in relative terms, for a lot more work.)
Now that we clarified up-thread that Eliezer’s position is not that there was a giant algorithmic innovation in between chimps and humans, but rather that there was some innovation in between dinosaurs and some primate or bird that allowed the primate/bird lines to scale better
Where was this clarified...? My Eliezer-model says “There were in fact innovations that arose in the primate and bird lines which allowed the primate and bird lines to scale better, but the primate line still didn’t scale that well, so we should expect to discover algorithmic innovations, if not giant ones, during hominin evolution, and one or more of these was the core of overlapping somethingness that handles chipping handaxes but also generalizes to building spaceships.”
If we’re looking for an innovation in birds and primates, there’s some evidence of ‘hardware’ innovation rather than ‘software.’
For a speculative software innovation hypothesis, there seem to be cognitive adaptations that arose in the LCA of all anthropoids for mid-level visual representations e.g. glossiness, above the level of, say, lines at a particular orientation, and below the level of natural objects, which seem like an easy way to exapt into categories, then just stack more layers for generalization and semantic memory. These probably reduce cost and error by allowing the reliable identification of foraging targets at a distance. There seem to be corresponding cognitive adaptations for auditory representations that strongly correlate with foraging targets, e.g. calls of birds that also target the fruits of angiosperm trees. Maybe birds and primates were under similar selection for mid-level visual representations that easily exapt into categories, etc.
If information is ‘transmitted’ by modified environments and conspecifics biasing individual search, marginal fitness returns on individual learning ability increase, while from the outside it looks just like ‘cultural ’evolution.″
If I take the number of years since the emergence of Homo erectus (2 million years) and divide that by the number of years since the origin of life (3.77 billion years), and multiply that by the number of years since the founding of the field of artificial intelligence (65 years), I get a little under twelve days. This seems to at least not directly contradict my model of Eliezer saying “Yes, there will be an AGI capable of establishing an erectus-level civilization twelve days before there is an AGI capable of establishing a human-level one, or possibly an hour before, if reality is again more extreme along the Eliezer-Hanson axis than Eliezer. But it makes little difference whether it’s an hour or twelve days, given anything like current setups.” Also insert boilerplate “essentially constant human brain architectures, no recursive self-improvement, evolutionary difficulty curves bound above human difficulty curves, etc.” for more despair.
I guess even though I don’t disagree that knowledge accumulation has been a bottleneck for humans dominating all other species, I don’t see any strong reason to think that knowledge accumulation will be a bottleneck for an AGI dominating humans, since the limits to human knowledge accumulation seem mostly biological. Humans seem to get less plastic with age, mortality among other things forces us to specialize our labor, we have to sleep, we lack serial depth, we don’t even approach the physical limits on speed, we can’t run multiple instances of our own source, we have no previous example of an industrial civilization to observe, I could go on: a list of biological fetters that either wouldn’t apply to an AGI or that an AGI could emulate inside of a single mind instead of across a civilization. I am deeply impressed by what has come out of the bare minimum of human innovative ability plus cultural accumulation. You say “The engine is slow,” I say “The engine hasn’t stalled, and look how easy it is to speed up!”I’m not sure I like using the word ‘discontinuous’ to describe any real person’s position on plausible investment-output curves any longer; people seem to think it means “intermediate value theorem doesn’t apply,” (which seems reasonable) when usually hard/fast takeoff proponents really mean “intermediate value theorem still applies but the curve can be almost arbitrarily steep on certain subintervals.”
I have an alternative hypothesis about how consciousness evolved. I’m not especially confident in it.
In my view, a large part of the cognitive demands on hominins consists of learning skills and norms from other hominins. One of a few questions I always ask when trying to figure out why humans have a particular cognitive trait is “How could this have made it cheaper (faster, easier, more likely, etc.) to learn skills and/or norms from other hominins?” I think the core cognitive traits in question originally evolved to model the internal state of conspecifics, and make inferences about task performances, and were exapted for other purposes later.
I consider imitation learning a good candidate among cognitive abilities that hominins may have evolved since the last common ancestor with chimpanzees, since as I understand it, chimps are quite bad at imitation learning. So the first step may have been hominins obtaining the ability to see another hominin performing a skill as another hominin performing a skill, in a richer way than chimps, like “That-hominin is knapping, that-hominin is striking the core at this angle.” (Not to imply that language has emerged yet, verbal descriptions of thoughts just correspond well to the contents of those thoughts; consider this hypothesis silent on the evolution of language at the moment.) Then perhaps recursive representations about skill performance, like “This-hominin feels like this part of the task is easy, and this part is hard.” I’m not very committed on whether self-representations or other-representations came first. Then higher-order things like, “This-hominin finds it easier to learn a task when parts of the task are performed more slowly, so when this-hominin performs this task in front of that-hominin-to-be-taught, this-hominin should exaggerate this part, or that part of the task.” And then, “This-hominin-that-teaches-me is exaggerating this part of the task,” which implicitly involves representing all those lower order thoughts that lead to the other hominin choosing to exaggerate the task, and so on. This is just one example of how these sorts of cognitive traits could improve learning efficiency, in sink and source.
Once hominins encounter cooperative contexts that require norms to generate a profit, there is selection for these aforementioned general imitation learning mechanisms to be exapted for learning norms, which could result in metarepresentations of internal state relevant to norms, like emotional distress, among other things. I also think this mechanism is a large part of how evolution implements moral nativism in humans. Recursive metarepresentations of one’s own emotional distress can be informative when learning norms as well. Insofar as one’s own internal state is informative about the True Norms, evolution can constrain moral search space by providing introspective access to that internal state. On this view, this is pretty much what I think suffering is, where the internal state is physical or emotional distress.
I think this account allows for more or less conscious agents, since for every object-level representation, there can be a new metarepresentation, so as minds become richer, so does consciousness. I don’t mean to imply that full-blown episodic memory, autobiographical narrative, and so on falls right out of a scheme like this. But it also seems to predict that mostly just hominins are conscious, and maybe some other primates to a limited degree, and maybe some other animals that we’ll find have convergently evolved consciousness, maybe elephants or dolphins or magpies, but also probably not in a way that allows them to implement suffering.
I don’t feel that I need to invoke the evolution of language for any of this to occur; I find I don’t feel the need to invoke language for most explicanda in human evolution, actually. I think consciousness preceded the ability to make verbal reports about consciousness.
I also don’t mean to imply that dividing as opposed to making pies is a small fraction of the task demands that hominins faced historically, but I also don’t think it’s the largest fraction.
Your explanation does double-duty, with its assumptions at least, and kind of explains how human cooperation is stable where it wouldn’t be by default. I admit that I don’t provide an alternative explanation, but I also feel like it’s outside of the scope of the conversation and I do have alternative explanations in mind that I could shore up if pressed.
Your argument has a Holocene, sedentary, urban flavor, but I think it applies just as well to Pleistocene, nomadic cultures; I think of it as an argument about population size and ‘cognitive capital’ as such, not only about infrastructure or even technology. Although my confidence is tempered by mutually compatible explanations and taphonomic bias, my current models of behavioral modernity and Neanderthal extinction essentially rely on a demographic argument like the one made here. I don’t think this comment would be as compelling without a reminder that almost everyone explains these phenomena via interspecies differences in individual cognitive adaptation, as opposed to demography.
On behavioral modernity, I am a gradualist; I do not think there was a sudden pulse of innovation. Signs of modernity, like a wider resource base, public symbol use, and certain technologies appear and then disappear, before becoming a permanent fixture later in the record. The most striking example of this would be ancient Australia, where humans did not reliably demonstrate all features of behavioral modernity for the first 25,000 years of residence even though these features were present in contemporaneous cultures in different geographic locations; Australia also happens to be one of the last places to which humans migrated. The idea here is that reduced population size, and thus density, affects the fidelity and bandwidth of social learning in lots of ways (e.g. decreased redundancy and specialization), resulting in a less sophisticated behavioral repertoire, as well as reduced selection for public symbol use (since there are no frequently encountered outgroups to which to signal).
On Neanderthal extinction, I think this is a collapse in the sense of your post, by a failure to reliably transmit cognitive capital. The life history of hominins is similar to a case of growth-dependent economics, where large energy and time investments are made under the expectation of future economic productivity. While I think environmental effects started the decay, temperate specialists as the Neanderthals were, I think they went extinct ultimately because these effects started a process that gradually invalidated the preconditions for reliable cultural transmission in that species.
For those wondering about the literature, although Kahneman and Tversky coined no term for it, Kahneman & Tversky (1981) describes counterfactual-closeness and some of its affective consequences. This paper appears to be the origin of the missed flight example. Roese (1997) is a good early review on counterfactual thinking with a section on contrast effects, of which closeness effects are arguably an instance.
Succubi/incubi and the alien abduction phenomenon point to hypnagogia, and evo-psych explanations of anthropomorphic cognition are often washed down with arguments that anthropomorphism causes good enough decisions while being technically completely false; there’s an old comment by JenniferRM talking about how surprisingly useful albeit wrong it would be to model pathogens as evil spirits.
An attempt at problem #1; seems like there must be a shorter proof.
The proof idea is “If I flip a light switch an even number of times, then it must be in the same state that I found it in when I’m finished switching.”
Theorem. Let e a path graph on ertices with a vertex oloring uch that if hen Let s bichromatic Then s odd.
Proof. By the definition of a path graph, there exists a sequence ndexing An edge s bichromatic iff A subgraph f s a state iff its terminal vertices are each incident with exactly one bichromatic edge or equal to a terminal vertex of The color of a state is the color of its vertices. There exists a subsequence of ontaining the least term of each state; the index of a state is equal to the index of its least term in this subsequence.
Note that none of the states with even indexes are the same color as any of the states with odd indexes; hence all of the states with even indexes are the same color, and all of the states with odd indexes are the same color.
For each state there exists a subsequence of orresponding to the vertices of and the least term of each subsequence is either r some hat is the greatest term in a bichromatic edge. Thus the number of states in
By contradiction, suppose that s even. Then the number of states is odd, and the first and last states are the same color, so the terminal vertices of re the same color, contrary to our assumption that they are different colors. Thus ust be odd.
:::
I see that New Zealand is also a major wood exporter. In case of an energy crisis, wood gas could serve as a renewable alternative to other energy sources. Wood gas can be used to power unmodified cars and generators. Empirically this worked during the Second World War and works today in North Korea. Also, FEMA once released some plans for building an emergency wood gasifier.
Yeah it’s fixed.
The Lahav and Mioduser link in section 14 is broken for me. Maybe it’s just paywalled?
Just taking the question at face value, I would like to choose to lift weights for policy selection reasons. If I eat chocolate, the non-Boltzmann brain versions will eat it too, and I personally care a lot more about non-Boltzmann brain versions of me. Not sure how to square that mathematically with infinite versions of me existing and all, but I was already confused about that.
The theme here seems similar to Stuart’s past writing claiming that a lot of anthropic problems implicitly turn on preference. Seems like the answer to your decision problem easily depends on how much you care about Boltzmann brain versions of yourself.
The closest thing to this I’ve seen in the literature is processing fluency, but to my knowledge that research doesn’t really address the willpower depletion-like features that you’ve highlighted here.
It’s also a useful analogy for aspects of group epistemics, like avoiding double counting as messages pass through the social network.
Fake Causality contains an intuitive explanation of double-counting of evidence.
Re: proof calibration; there are a couple textbooks on proofwriting. I personally used Velleman’s How to Prove It, but another option is Hammack’s Book of Proof, which I haven’t read but appears to cover the same material at approximately equal length. For comparison, Halmos introduces first-order logic on pages 6 and 7 of Naive Set Theory, whereas Velleman spends about 60 pages on the same material.
It doesn’t fit my model of how mathematics works technically or socially that you can really get very confident but wrong about your math knowledge without a lot of self-deception. Exercises provide instant feedback. And according to Terence Tao’s model, students don’t spend most of their education learning whether or not a proof is valid at all, so much as learning how to evaluate longer proofs more quickly without as much conscious thought.
Part of that process is understanding formal things, part of it is understanding how mathematicians’ specialized natural language are shorthand for formal things. E.g. my friend was confused when he read an exercise telling him to prove that a set was “the smallest set” with this property (and perhaps obviously the author didn’t unpack this). What this means formally when expanded is “Prove that this set is a subset of every set with this property.” AFAICT, there’s no way to figure out what this means formally without someone telling you, or (this is unlikely) inventing the formal version yourself because you need it and realizing that ‘smallest set’ is good shorthand and this is probably what was meant. Textbooks are good for fixing this because the authors know that textbooks are where most students will learn how to talk like a mathematician without spelling everything out. I find ProofWiki very useful for having everything spelled out the way I would like it and consistently when I don’t know what the author is trying to say.
Finally, I have a rationalist/adjacent friend who tutored me enough to get to the point where I could verify my own proofs; I haven’t talked to them in a while, but I could try to get in touch and see if they would be interested in checking your proofs. Last time I talked to them, they expressed that the main bottleneck on the number of students they had was students’ willingness to study.
Re: category-theory-first approaches; I find that most people think this is a bad idea because most people need to see concrete examples before category theory clicks for them, otherwise it’s too general, but a few people feel differently and have published introductory textbooks on category theory that assume less background knowledge than the standard textbooks. If you’re interested, you could try Awodey’s Category Theory (free), or Lawvere’s Conceptual Mathematics. After getting some more basics under your belt, you could give either of those a shot, just in case you’re the sort of person who learns faster by seeing the general rule before the concrete examples. (These people exist, but I think it’s easy to fall into the trap of wishing that you were that sort of person and banging your head against the general rule when you really just need to pick up the concrete examples first. One should update if first-principles approaches are not working.)
I’d be very interested in reading about EverQuest as an exemplar of Fun Theory, if you’re willing to share.
I just want to register that there are things I feel like I couldn’t have realistically learned about my partner in an amount of time short enough to precede the event horizon, and that my partner and I have changed each other in ways past the horizon that could be due to limerence but could also just be good old-fashioned positive personal change for reasonable reasons, and that I suspect that the takeaway here is not necessarily to attempt to make final romantic decisions before reaching the horizon, but that you will only be fully informed past the horizon, and that is just the hand that Nature has dealt you. I don’t necessarily think you disagree with that but wanted to write that in case you did.