I’m trying to prevent doom from AI. Currently trying to become sufficiently good at alignment research. Feel free to DM for meeting requests.
Towards_Keeperhood
Karma: 413
Main pieces I remember were: Orcas already dominating the planet (like humans do), large sea creatures going extinct due to orcas (similar to how humans drove several species extinct, (Megalodon? Probably extinct for different reasons, weak evidence against? Most other large whales are still around)).
To clarify for other readers: I do not necessarily endorse this is what we would expect if orcas were smart.
(Also I read somewhere that apparently chimpanzees sometimes/rarely can experience menopause in captivity.)
If the species is already dominating the environment then the pressure from the first component compared to the second decreases.
I agree with this. However I don’t think humans had nearly sufficient slack for most of history. I don’t think they dominated the environment up until 20000years [1]ago or so, and I think most improvements in intelligence come from earlier.
That’s why I’m attributing the level of human intelligence in large part to runaway sexual selection. Without it, as soon as interspecies competition became the most important for reproductive success, natural selection would not push for even grater intelligence in humans, even though it could improve our ability to dominate the environment even more.
I’m definitely not saying that group selection lead to intelligence in humans (only that group selection would’ve removed it over long timescales if it wasn’t useful). However I think that there were (through basically all of human history) significant individual fitness benefits from being smarter that did not come from outwitting each other, e.g. being better able to master hunting techniques and thereby gaining higher status in the tribe.
- ^
Or could also be 100k years, idk
- ^
I’m not sure how it’s relevant.
I thought if humans were vastly more intelligent than they needed to be they would already learn all the relevant knowledge quickly enough so they reach their peak in the 20s.
And if the trait, the runaway sexual selection is propagating, is itself helpful in competition with other species, which is obviously true for intelligence, there is just no reason for such straightening over a long timescale.
I mean for an expensive trait like intelligence I’d say the benefits need to at least almost be worth the costs, and then I feel like rather attributing the selection for intelligence to “because it was useful” rather than “because it was a runaway selection”.
(For reference I think Tsvi and GeneSmith have much more relevant knowledge for evaluating the chance of superbabies being feasible and I updated my guess to like 78%.)
(As it happens I also became more optimistic about the orca plan (especially in terms of how much it would cost and how long it would take, but also a bit in how likely I think it is that orcas would actually study science) (see footnote 4 in post). For <=30y timelines I think the orca plan is a bit more promising, though overall the superbabies plan is more promising/important. I’m now seriously considering pivoting to the orca plan though.) (EDIT: tbc I’m considering pivoting from alignment research, not superbaby research.)
(haha cool. perhaps you could even PM Abram if he doesn’t PM you. I think it would be pretty useful to speed up his agenda through this.)
Thanks!
I agree that sexual selection is a thing—that it’s the reason for e.g. women sometimes having unnecessarily large breasts.
But I think it gets straightened out over long timescales—and faster the more expensive the trait is. And intelligence seems ridiculously expensive in terms of metabolic energy our brain uses (or childbirth motality).
A main piece that updated me was reading anecdotes in Scott Alexander’s Book review of “The Secret of our success” where I now think that humans did need their intelligence for survival. (E.g. 30 year old hunter gatherers perform better at hunting etc than hunter gatherers in their early 20s, even though the latter are more physically fit.)
A few more thoughts:
It’s plausible that for both humans and orcas the relevant selection pressure mostly came from social dynamics, and it’s plausible that there were different environmental pressures.
Actually my guess would be that it’s because intelligence was environmentally adaptive, because my intuitive guess would be that group selection[1] is significant enough over long timescales which would disincentivize intelligence if it’s not already (almost) useful enough to warrant the metabolic cost, unless the species has a lot of slack.
So an important question is: How adaptive is high intelligence?
In general I would expect that selection pressure for intelligence was significantly stronger in humans, but maybe for orcas it was happening over a lot longer time window, so the result for orcas could still be more impressive.
From what I observed about orca behavior I’d perhaps say a lower bound of their intelligence might roughly be like human 15 year olds or so. So up to that level of intelligence there seem to be benefits that allow orcas to use more sophisticated hunting techniques.
But would it be useful for orcas to be significantly smarter than humans? My prior intuition would’ve been that probably not very much.
But I think observing the impressive orca brains mostly screens this off: I wouldn’t have expected orcas to evolve to be that smart, and I similarly strongly wouldn’t have expected them to have that impressive brains, and seeing their brains updates me that there had to be some selection pressure to produce that.
But the selection pressure for intelligence wouldn’t have needed to be that strong compared to humans for making the added intelligence worth the metabolic cost, because orcas are large and their neurons make up a much smaller share of their overall metabolic consumption. (EDIT: Actually (during some (long?) period of orca history) selection pressure for intelligence also would’ve needed to be stronger than selection pressure for other traits (e.g. making muscles more efficient or whatever).)
And that there is selection pressure is not totally implausible in hindsight:
Orcas hunt very collaboratively, and maybe there are added benefits from coordinating their attacks better. (Btw, orcas live in matrilines, and I’d guess that from an evolutionary perspective the key thing to look at is how well a matriline performs, not individuals, but not sure. So there would be high selection for within-matriline cooperation (and perhaps communication!).)
Some/(many?) Orca sub-species prey on other smart animals like dolphins or whales, and maybe orcas needed to be significantly smarter to be able to outwit the defensive mechanisms they learn to adapt.
But overall I know way too little about orca hunting techniques to be able to evaluate those.
- ^
I mean group selection that could potentially be on a level of species where species go extinct. Please lmk if that’s actually called differently.
thanks. Can you say more about why?
I mean runaway sexual selection is basically H1, which I updated to being less plausible. See my answer here. (You could comment there why you think my update might be wrong or so.)
My prior intuitive guess would be that H1 seems quite a decent chunk more likely than H2 or H3.
Actually I changed my mind.
Why I thought this before: H1 seems like a potential runaway-process and is clearly about individual selection which has stronger effects than group selection (and it was mentioned in HPMoR).
Why I don’t think this anymore:
It would also be incredibly huge coincidence if intelligence mostly evolved because of social dynamics but happened to be useful for all sorts of other survival techniques hunters and gatherers use. See e.g. Scott Alexander’s Book review of “The Secret of our success”.
If there was only individual benefits for intelligence but it was not very useful otherwise then over long timelines group selection[1] would actually select against smarter humans because their neurons would use up more metabolic energy.
However, there’s a possibly very big piece of evidence for H3: Humans are both the smartest land animals and have the best interface for using tools, and that would seem like a suspicious coincidence.
I think this is not a coincidence but rather that tool use let humans fall into an attractor basin where payoffs of intelligence were more significant.
- ^
I mean group selection that could potentially be on a level of species where species go extinct. Please lmk if that’s actually called differently.
Some of my own observations and considerations:
Anecdotal evidence for orca intelligence
Intimate cooperation between native australian hunter gatherers and orcas for whale hunting: https://en.wikipedia.org/wiki/Killer_whales_of_Eden,_New_South_Wales
Orcas being skillful at turning boats around and even sinking a few vessels[1][2]: https://en.wikipedia.org/wiki/Iberian_orca_attacks
Orcas have a wide variety of cool hunting strategies. (e.g. see videos (1, 2)). I don’t know how this compares to human hunter gatherers. (EDIT: Ok I just read Scott Alexander’s Book review of “The Secret of our success” and some anecdotes on hunter gatherers there seem much more impressive. (But also plausible to me that other orca hunting techniques are also more sophisticated than the examples but in ways it might not be legible to us.))
(ADDED: Tbc, while this is more advanced than I’d a priory expected from animals, the absence of observations of even more clearly stunning techniques is some counterevidence of orcas being smarter than humans. Though I also don’t quite point to an example of what I’d expect to see if orcas were actually 250 IQ but what I don’t observe, but I also didn’t think for long and maybe there would be sth.)
Orca language
(Warning: Low confidence. What I say might be wrong.)
I didn’t look deep into research into orca language (not much more than watching this documentary), my impression is that we don’t know much yet.
Some observations:
Orcas language seems to be learned, not innate. Different regions have different languages and dialects. Scientists seem to analogize it to how humans speak different languages in different countries.
For some orca groups that were studied, scientists were able to cluster their calls into 23 or 24 different calls clusters, but still with significant variation of calls within a call cluster.
(I do not know how tightly calls are clustered, or whether there often are outliers.)
Orcas communicate a lot. (This might be wrong but I think they spend a significant fraction of their time socializing where they exchange multiple calls per minute.)
(Orcas emit clicks and whistles. The clicks are believed to be for spacial navigation (especially in the dark), the whistles for communication.) (EDIT: Actually also pulsed calls, which I initially lumped in with whistles but are emitted in pulses. Those are probably the main medium of communication.)
I’d count (2) as some weakish evidence against orcas having as sophisticated language as humans, however not very strongly. Some considerations:
Sentences don’t necessarily need to be formed through having temporal sequences of words, but words could also be some different frequency signals or so which are then simultanously overlayed.
(The different 24 call types could be all sorts of things. E.g. conveying what we convey through body language, facial expressions, and tone. Or e.g. different sentence structures. Idk.)
Their language might be very alien. I only have shitty considerations here but e.g.:
Orca language doesn’t need to have at all similar grammar. E.g. could be something as far from our language as logic programming is, though in the end still not nearly that simple.
Orcas might often describe situations in ways we wouldn’t describe them. E.g. rather about what movements they and their prey executed or sth.
Orcas might describe more precisely where in 3D water particular orcas and animals were located, and they might have a much more efficient encoding for that than if we tried to communicate this.
More considerations
The only piece of evidence that makes me wonder whether orcas might actually be significantly smarter than humans is their extremely impressive brain. I think it’s pretty strong though.
As mentioned, orcas have 2.05 times as many neurons in their neocortex as humans, and when I look through the wikipedia list (where I just trust measured[3] and not estimated values), it seems to be a decent proxy for how intelligent a species is.
There needs to be some selection pressure for why they have 160 times more neurons in their neocortex than e.g. brown bears (which weigh like 1/8th of an orca or so). Size alone is not nearly a sufficient explanation.
It’s plausible that for both humans and orcas the relevant selection pressure mostly came from social dynamics, and it’s plausible that there were different environmental pressures. (I’m keen to learn.) It’s possible that caused humans to be smart more strongly incentivized our brains to be able to do abstract reasoning, whereas for orcas it might’ve been useful for some particular skills that generalize less well for doing other stuff.
If I’d only ever seen hunter gatherer humans, even if I could understand their language, I’m not sure I’d expect that species to be able to do science on priors. But humans are able to do it. Somehow our intelligence generalized far outside the distribution we were optimized on. I don’t think that doing science is similar to anything we’ve been optimized on, except that advanced language might be necessary.
On priors I wouldn’t really see significant reasons why whatever selection pressures optimized orcas to have their astounding brains, would make their intelligence generalize less well to doing science, than whatever selection pressures produced our impressive human brains.
One thing that would update me significantly downwards on orcas being able to do science is if their prefrontal cortex doesn’t contain that many neurons. (I didn’t find that information quickly so please lmk if you find it.) Humans have a very large prefrontal cortex compared to other animals. My guess would be that orcas have too, and that they probably still have >1.5 times as many neurons in their prefrontal cortex than humans, and TBH I even wouldn’t be totally shocked if it’s >2.5 times.
- ^
Btw there is no recorded case of a human having been killed by an orca in the wild, including when they needed to swim when the vessel was sunk. (Even though orcas often eat other mammals.) (I think I even once heard it mention that it seemed like the orcas made sure that no humans died from their attacks, though I don’t at all know how active the role of the orcas was there (my guess is not very).)
- ^
I’d consider it plausible that they were trying to signal us to please stop fishing that much, but I didn’t look nearly deeply enough into it to judge.
- ^
Aka optical or isotropic fractionator in the method column.
Thanks!
What’s the ect? Or do you have links for where to learn more? (What’s the name of the field?)
(I thought wikipedia would give me a good overview but your list was already more useful to me.)
Thanks. No I didn’t. (And probably don’t have time to look into it but still nice to know.)
Justification for this:
I don’t think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.
One could naively expect that the neuron count (especially touch and motor) sensory processing modules are proportional to the surface area of an organism. However I think this is unrealistic: Bears don’t need nearly as fine precision on what square centimeter of skin was touched (or what millimeter the paw moves) than mice, and generally this is because precision gets less relevant given body size.
So let’s say the precision an organism needs is proportional to the square root of the 1-dimensional-size (aka sqrt(surface_area)) of the organism. Aka if a mice is 5cm tall and a bear 2m, the spacing between sensors on the mouse skin vs on the bear skin would be sqrt(0.05) vs sqrt(2). The number of sensors on the skin surface is proportional to the square of the distancing between sensors, so the overall number of sensors is proportional to the 1-dimensional-size (aka sqrt(surface_area)).
A brown bear has 250million neorons in the neocortex and is maybe 2m tall. So to get just by scaling size to 40billion neorons an organism would have to be 40⁄0.25 * 2m = 320m tall. So actually bigger than godzilla.
I don’t think I’m the right person to look into this.
I just updated quickly via conservation of expected probability. (I agree though that I’d be a bit concerned about most people updating that quickly. If you think I’ve gone slightly psychotic please bet with me so I update harder if I notice you’re right.)
(EDIT: actually it’s sorta shitty because we might not get more evidence because I have even more important things to do and probably don’t have time to look into it myself, but i’m happy to bet, though I’d probably want to revise my betting probability.)
I’m happy to bet on “By the end of 2034, does Tsvi think that it’s >60% likely that orcas could do superhuman science if they had similar quality and quantity of science education as scientists and were motivated for this, conditional on Tsvi having talked to me for at least 2 hours about this {sometime in 2030-2034}/{when I might have more evidence}?”
I’d currently be at like
30%26% on this, though if you take more time to think about it I might adjust this estimate I am willing to bet on.I’m happy to bet up to 200$ per bit (or maybe more but would have to think about it). Aka if it’s resolved “Yes”, money flowing from you to me would be (and if it’s resolved “No” it would be ). (Where negative money flow indicates flow into the other direction.)
(Also obviously you’d need to commit to talking to me for 2h sometime when i have more evidence, and not just avoid resolution by not talking to me.)
I don’t know what you mean by this:
The thing is, there’s probably gonna be like ten other posts in the reference class of this post, and they just… don’t leave much of a dent in things?
I don’t think it makes that much sense to just look at cortical neuron counts. Big bodies ask for many neurons, including cortical motor neurons. Do cetaceans have really big motor cortices? Visual cortices? Olfactory bulbs? Keyword “allometry”. Yes, brains are plastic, but that doesn’t mean orcas are actually ever doing higher mathematics with their brains.
See this comment.
Scale matters, but I doubt it’s very close to being the only thing! Humans likely had genetic adaptations for neuroanatomical phenotypes selected-for by some of: language; tool-making; persisting transient mental content; intent-inference; intent-sharing; mental simulation; prey prediction; deception; social learning; teaching; niche construction/expansion/migration. Orcas have a few of these. But how many, how much, for how long, in what range of situations and manifestations?
I already considered this. (I just posted a question about this.) I don’t have good information on to what extent orcas have those, but my guesses are already reflected in my overall guess in the post.
Why do you think orcas have few of those? For me it seems plausibe that orcas have everything except tool use and niche construction.I do think there was some significant selection for some kind of intelligence in dolphins and orcas—the main question here is whether being optimized on tool use (IF that was a significant driver in what selected humans for intelligence) would be significantly more useful for having the brain potential generalize to doing science than if the brains were optimized because of social dynamics or hunting strategies.
But of course there are other considerations like “maybe you need fully recursive language to be able to have the abstract reasoning take off, and this might very well come from some adaptations that are not just about neoron counts, and maybe orcas don’t have that”.
I already took all my current uncertain consideration on this into account when I said “50% that they would be superhuman at science if they had similar quality and quantity of science education as scientists and were motivated for this”.
Or do you think a cow brain scaled to 40 billion neurons would be superhuman?
I don’t know what you’re asking for here. I don’t think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.
I’m not really excited about just smushing together more brain tissue without it having been optimized to work well together, but orca brains were optimized.
Culture matters. The Greeks could be great philosophers… But could a kid living in 8000 BCE, who gets to text message with an advanced alien civilization of kinda dumb people, become a cutting edge philosopher in the alien culture? Even though almost everyone ze interacts with is preagricultural, preliterate? I dunno, maybe? Still seems kinda hard actually?
Yep that’s why I’m only at like 15% that we get very significant results out of it in the next 30 years even if we tried hard. (aka 30% conditional on orcas being smart enough.)
Superbabies is good. It would actually work. It’s not actually that hard. There’s lots of investment already in component science/tech. Orcas doesn’t scale. No one cares about orcas. There’s not hundreds of scientists and hundreds of millions in orca communications research. Etc. The sense of this plan being weird is a good sense to investigate further. It’s possible for superficial weirdness to be wrong, but don’t dismiss the weirdness out of hand.
I mean if Orcas are smarter they might be super vastly smarter so you wouldn’t need that many.
Superbabies would work well given multiple generations but also only like 30% that we’d get +7std humans born within 10 years even if we tried similarly hard[1], and I think it’s pretty unlikely we have more than 40 years left without strong governance success. (E.g. afaik we still have problems cloning primates well (even though it’s been a thing for long) and those are just sub-difficulties[2] of e.g. creating superbabies through repeated embryo selection.)
I think number of neurons in neocortex (or even more prefrontal cortex—but unfortunately i didn’t quickly find how big the orca prefrontal cortex is—though I’d guess it to still be significantly bigger than for humans) is a much much better proxy for intelligence of species than brain size (or encephalization quotient). (E.g. see the wikipedia list linked in my question here.)
(Also see here. There are more examples, e.g. a Blue and yellow macaw has 1.9 billion, whereas brown bears have only 250million.)
EDIT: Tbc I do think that larger bodies require more neurons in touch-sense and motor parts of the neocortex, so there is some effect of how larger animals need a bit larger brains to be similarly smart, but I don’t think this effect is very strong.
But yeah there are other considerations too, which is why I am only at 50% that orcas could do science significantly better than humans if they tried.
I feel like life-force seems like a sensation that’s different from what I’d expect from just having a thing in the world model with inherent surprisingness and ends-without-trajectory-predictions/”optimizerness” attached. (“Life-force” sounds more like “as if the thing had a soul” to me. I do not understand where this comes from but I don’t see how I’d predict such a sensation in advance given just the inherent-surprisingness + optimizerness hypothesis.)
Thanks for communicating your model well again!
I think we might mostly agree, but let’s clarify.
I agree with all of:
In the course of predicting them well, the world-model invents some slightly-higher-level concept (or family of closely-interlinked concepts) that we call “cold”. And it notices and memorizes predictively-useful relationships between this new “cold” concept and other things in the world-model, e.g. shivering and ice.
I don’t think there’s more to the concept “cold” than the sum total of its associations with every other concept, with sensory input, and with motor output.
I also basically agree with:
I like to draw the distinction between understanding learning algorithms and understanding trained models. The former is kinda like what you learn in an ML course (gradient descent, training data, etc.) , the latter is kinda like what you learn in a mechanistic interpretability paper. I don’t think it’s realistic to “write code” for the “cold” concept, because I think it (like all concepts) emerges at the trained model level. It emerges from a learning algorithm, training environment, loss function, etc.
I agree that fully writing code would be quite a daunting task. I think my phrasing of “write code” was not great. But it’s already some reductionist progress if you have something like:
if coldness concept gets more activated: increase activation of shivering anticipation; weakly increase activation of snow concept; ...I don’t think it’s a worthwhile exercise to get very precise.
An important point I wanted to make here is just that the meaning of “cold” comes from the interactions with other concepts, and there’s no such thing as an inherent independent meaning of the word “cold”. (So when I hear ‘If we look at naturalistic visual inputs that directly or indirectly trigger C, and they’re disproportionately pictures of clocks, then that’s some evidence that C “means” clock.’ this seems a bit off to me, though not too bad.)
I guess I best try to explain why I felt some unease with your initial description of the cold example:
Suppose somebody said:
There’s a certain kind of interoceptive sensory input, consisting of such-and-such signal coming from blah type of thermoreceptor in the peripheral nervous system. Your brain does its usual thing of transforming that sensation into its own “color” of “metaphysical paint” (as in §3.3.2) that forms a concept / property in your conscious awareness and world-model, and you know it by the everyday term “cold”.
On the one hand, I would defend this passage as basically true.
Basically I think that some people—though a priory not you—would think that sth like “i feel cold because the cold-thermorecepters activate the corresponding cold concept” explains their sense of cold. However, if you just take this hypothesis which basically is “some sensors activate some concept” without anything else, then the concept would be completely shapeless and uninterpretable—unrelated to anything known.
I now think you probably didn’t mean it in a nearly that bad way but not sure.
(But some parts of what you write seem to me like you have slightly weaker sensors about “how does a hypothesis actually constrain my anticipations / concentrate probability mass” or “what would this hypothesis predict if I didn’t already know how I perceive it”, and I do think those sensors are useful.)
(I also think that there is some hypothalamus-or-so buisness logic for what responses to trigger (e.g. shivers) from significant cold input signals that would need to be figured out if you want to get a good model of freezing/feeling-uncomfortably-cold, but that’s about freezing in particular and not temperature as a property we model on objects.)
The post is not joking. (But thanks for feedback that you were confused in this way!)
I basically didn’t know much about orcas before I learned that they have 2.05 times 3 days ago as many neurons in the neocortex than humans, and then yesterday and the day before I spent looking into how smart orcas might be and evaluating the evidence. So I’m far from an expert but I also didn’t run over strong evidence that they are dumber than hunter-gatherer humans and some weak-medium strong evidence that they are might at least as smart as 15 year olds. But it’s still possible that orca researchers have observed orcas not finding some strategies they would’ve thought of or sth.
But yeah the only piece of evidence that they might be significantly smarter than humans is their brains. I consider it reasonably strong though.
ADDED:
I edited “Though I don’t know that much about orcas” to “Though I only tried to form a model of orca intelligence for 1-2 days”. thanks!
Another thought:
In what animals would I on priors expect intelligence to evolve?
Animals which use collaborative hunting techniques.
Large animals. (So the neurons make up a smaller share of the overall metabolic cost.)
Animals that can use tools so they benefit more from higher intelligence.
(perhaps some other stuff like cultural knowledge being useful, or having enough slack for intelligence increase from social dynamics being possible.)
AFAIK, orcas are the largest animals that use collaborative hunting techniques.[1] That plausibly puts them second behind humans for where I would expect intelligence to evolve. So it doesn’t take that much evidence for me to be like “ok looks like orcas also fell into some kind of intelligence attractor”.
Though I heard sperm whales might sometimes collaborate too, but not nearly that sophisticated I guess. But I also wouldn’t be shocked if sperm whales are very smart. They have the biggest animal brains, but I don’t whether the cortical neuron count is known.