I’m trying to prevent doom from AI. Currently trying to become sufficiently good at alignment research. Feel free to DM for meeting requests.
Towards_Keeperhood
Karma: 657
If I did, I wouldn’t publicly say so.
It’s of course not yes or no, but just a probability, but in case it’s high I might not want to state it here, so I should generally not state it here, so you cannot infer it is high by the fact that I didn’t state it here.
I can say though that I only turned 22y last week and I expect my future self to grow up to become much more competent than I am now.
2. I mentioned that there should be much more impressive behavior if they were that smart; I don’t recall us talking about that much, not sure.
You said “why don’t they e.g. jump in prime numbers to communicate they are smart?” and i was like “hunter gatherer’s don’t know prime numbers and perhaps not even addition” and you were like “fair”.
I mean I thought about what I’d expect to see, but I unfortunately didn’t really imagine them as smart but just as having a lot of potential but being totally untrained.
3. I recommended that you try hard to invent hypotheses that would explain away the brain sizes.
(I’m kinda confused why your post here doesn’t mention that much; I guess implicitly the evidence about hunting defeats the otherwise fairly [strong according to you] evidence from brain size?)
I suggest that a bias you had was “not looking hard enough for defeaters”. But IDK, not at all confident, just a suggestion.
Yeah the first two points in the post are just very strong evidence that overpower my priors (where by priors i mean considerations from evolution and brain size, as opposed to behavior). Ryan’s point changed my priors, but I think it isn’t related enough to “Can I explain away their cortical neuron count?” that asking myself this question even harder would’ve helped.
Maybe I made a general mistake like “not looking hard enough for defeaters”, but it’s not that actionable yet. I did try to take all the available evidence and update properly on everything. But maybe some motivated stopping on not trying even longer to come up with a concrete example of what I’d have expected to see from orcas. It’s easier to say in retrospect though. Back then I didn’t know in what direction I might be biased.
But I guess I should vigilantly look out for warning signs like “not wanting to bother to think about something very carefully” or so. But it doesn’t feel like I was making the mistake, even though I probably did, so I guess the sensation might be hard to catch at my current level.
Yes human intelligence.
I forgot to paste in that it’s a follow up to my previous posts. Will do now.
In general, I wish this year? (*checks* huh, only 4 months.)
Nah I didn’t loose that much time. I already quit the project end of January, I just wrote the post now. Most of the technical work was also pretty useful for understanding language, which is a useful angle on agent foundations. I had previously expected working on that angle to be 80% as effective as my previous best plan, but it was even better, around similarly good I think. That was like 5-5.5 weeks and that was not wasted.
I guess I spent like 4.5 weeks overall on learning about orcas (including first seeing whether I might be able to decode their language and thinking about how and also coming up with the whole “teach language” idea), and like 3 weeks on orga stuff for trying to make the experiment happen.
I changed my mind about orca intelligence
Yeah I think I came to agree with you. I’m still a bit confused though because intuitively I’d guess chimps are dumber than −4.4SD (in the interpretation for “-4.4SD” I described in my other new comment).
When you now get a lot of mutations that increase brain size, while this contributes to smartness, this also pulls you away from the species median, so the hyperparameters are likely to become less well tuned, resulting in a countereffect that also makes you dumber in some ways.
Actually maybe the effect I am describing is relatively small as long as the variation in brain size is within 2 SDs or so, which is where most of the data pinning down the 0.3 correlation comes from.
So yeah it’s plausible to me that your method of estimating is ok.
Intuitively I had thought that chimps are just much dumber than humans. And sure if you take −4SD humans they aren’t really able to do anything, but they don’t really count.
I thought it’s sorta in this direction but not quite as extreme:
(This picture is actually silly because the distance to “Mouse” should be even much bigger. The point is that chimps might be far outside the human distribution.)
But perhaps chimps are actually closer to humans than I thought.
(When I in the following compare different species with standard deviations, I don’t actually mean standard deviations, but more like “how many times the difference between a +0SD and a +1SD human”, since extremely high and very low standard deviation measures mostly cease to me meaningful for what was actually supposed to be measured.)
I still think −4.4SD is overestimating chimp intelligence. I don’t know enough about chimps, but I guess they might be somewhere between −12SD and −6SD (compared to my previous intuition, which might’ve been more like −20SD). And yes, considering that the gap in cortical neuron count between chimps and humans is like 3.5x, and it’s even larger for the prefrontal cortex, and that algorithmic efficiency is probably “orca < chimp < human”, then +6SDs for orcas seem a lot less likely than I initially intuitively thought, though orcas would still likely be a bit smarter than humans (on the way my priors would fall out (not really after updating on observations about orcas)).
Thanks for describing a wonderfully concrete model.
I like that way you reason (especially the squiggle), but I don’t think it works quite that well for this case. But let’s first assume it does:
Your estimamtes on algorithmic efficiency deficits of orca brains seem roughly reasonable to me. (EDIT: I’d actually be at more like −3.5std mean with standard deviation of 2std, but idk.)
Number cortical neurons != brain size. Orcas have ~2x the number of cortical neurons, but much larger brains. Assuming brain weight is proportional to volume, with human brains being typically 1.2-1.4kg, and orca brains being typically 5.4-6.8kg, orca brains are actually like 6.1/1.3=4.7 times larger than human brains.
Taking the 5.4-6.8kg range, this would be 4.15-5.23 range of how much larger orca brains are. Plugging that in for `orca_brain_size_difference` yields 45% on >=2std, and 38% on >=4std (where your values ) and 19.4% on >=6std.
Updating down by 5x because orcas don’t seem that smart doesn’t seem like quite the right method to adjust the estimate, but perhaps fine enough for the upper end estimates, which would leave 3.9% on >=6std.Maybe you meant “brain size” as only an approximation to “number of cortical neurons”, which you think are the relevant part. My guess is that neuron density is actually somewhat anti-correlated with brain size, and that number of cortical neurons would be correlated with IQ rather at ~0.4-0.55 in humans, though i haven’t checked whether there’s data on this. And ofc using that you get lower estimates for orca intelligence than in my calculation above. (And while I’d admit that number of neurons is a particularly important point of estimation, there might also be other advantages of having a bigger brain like more glia cells. Though maybe higher neuron density also means higher firing rates and thereby more computation. I guess if you want to try it that way going by number of neurons is fine.)
My main point is however, that brain size (or cortical neuron count) effect on IQ within one species doesn’t generalize to brain size effect between species. Here’s why:
Let’s say having mutations for larger brains is beneficial for intelligence.[1]
On my view, a brain isn’t just some neural tissue randomly smished together, but has a lot of hyperparameters that have to be tuned so the different parts work well together.
Evolution basically tuned those hyperparameters for the median human (per gender).
When you now get a lot of mutations that increase brain size, while this contributes to smartness, this also pulls you away from the species median, so the hyperparameters are likely to become less well tuned, resulting in a countereffect that also makes you dumber in some ways.So when you get a larger brain as a human, this has a lower positive effect on intelligence, than when your species equilibriates on having a larger brain.
Thus, I don’t think within species intelligence variation can be extended well to inter-species intelligence variation.As for how to then properly estimate orca intelligence: I don’t know.
(As it happens, I thought of something and learned something yesterday that makes me significantly more pessimistic about orcas being that smart. Still need to consider though. May post them soon.)
- ^
I initially started this section with the following, but I cut it out because it’s not actually that relevant: “How intelligent you are mostly depends on how many deleterious mutations you have that move you away from your species average and thereby make you dumber. You’re mostly not smart because you have some very rare good genes, but because you have fewer bad ones.
Mutations for increasing sizes of brain regions might be an exception, because there intelligence trades off against childbirth mortality, so higher intelligence here might mean lower genetic fitness.”
- ^
Thanks for the suggestion, though I don’t think they are smart enough to get far with grammar. No non-cetaceans non-humans seem to be.
One possibility is to try it with bottlenose dolphins (or beluga whales). (Bottlenose dolphins have shown greater capacity to learn grammar than great apes.[1]) Those are likely easier to get research access to than orcas. I think we might get some proof of concept of the methodology there, though I’m relatively pessimistic about them learning a full language well.
- ^
See the work of Louis Herman in the 80s (and 90s)
- ^
By >=+6std I mean potential of how smart they could be if they were trained similarly to us, not actual current intelligence. Sorry I didn’t write this in this post, though I did in others.
I’d be extremely shocked if orcas were actually that smart already. They don’t have science and they aren’t trained in abstract reasoning.
Like, when an orca is +7std, he’d be like a +7std hunter gatherer human, who is probably not all that good at abstract reasoning tasks (like learning a language through brute-force abstract pattern recognition). (EDIT: Ok actually it would be like a +7std hunter gatherer society, which might be significantly different. Idk what I’m supposed to expect there. Still wouldn’t expect it to be dangerous to talk to them though. And actually when I think about +7std societies I must admit that this sounds not that likely. That they ought to have more information exchange outside their pods and related pods or so and coordinate better. I guess that updates me downwards a bit on orcas being actually that smart—aka I hadn’t previously properly considered effects from +7std cultural evolution rather than just individual intelligence.)
Thanks for letting me know it sounded like that. I definitely know it isn’t legible at all, and I didn’t expect readers to buy it, just wanted to communicate that that’s how it’s from my own perspective.
You’re right. I’ll edit the post.
Help make the orca language experiment happen
Considerations on intelligence of wild orcas vs captive orcas
I’ve updated to thinking it’s relatively likely that wild orcas are significantly smarter than captive orcas, because (1) wild orcas might learn proper language and captive orcas don’t, and (2) generally orcas don’t have much to learn in captivity, causing their brains to be underdeveloped.
Here are the most relevant observations:
Observation 1: (If I analyzed the data correctly and the data is correct,) all orcas currently alive in captivity have been either born in captivity or captured when they were at most 3-4 years old.[1] I think there never were any captive orcas that survived for more than a few months that were not captured at <7 years age, but not sure. (EDIT: Namu (the first captive orca) was ~10y, but he died after a year. Could be that I missed more cases where older orcas survived.)
Observation 2: (Less centrally relevant, but included for completeness:) It takes young orcas ca 1.5 years until the calls they vocalize aren’t easily distinguishable from calls of other orcas by orca researchers. (However, as mentioned in the OP, it’s possible the calls are only used for long distance communication and orcas have a more sophisticated language at higher frequencies.)
Ovservation 3: Orcas in captivity don’t get much stimulation.
Genie, discovered in 1970 at age 13, was a victim of extreme abuse and isolation who spent her formative years confined to a small room with minimal human interaction. Despite intensive rehabilitation efforts following her rescue, Genie’s cognitive impairments proved permanent. Her IQ remained in the moderate intellectual disability range, with persistent difficulties in abstract reasoning, spatial processing, and problem-solving abilities.
Her language development, while showing some progress, remained severely limited. She acquired a vocabulary of several hundred words and could form basic sentences, but never developed proper grammar or syntax. This case provides evidence for the critical period hypothesis of language acquisition, though it’s complicated by the multiple forms of deprivation she experienced simultaneously.
Genie’s case illustrates how early environmental deprivation can cause permanent cognitive and linguistic deficits that resist remediation, even with extensive intervention and support.
Inferences:
If orcas need input from cognitively well-developed orcas (or richer environmental stimulation) for becoming cognitively well-developed, no orca in captivity became cognitively well-developed.
Captive orcas could be cognitively impaired roughly similarly to how Genie was. Of course, there might have been other factors contributing to the disability of Genie, but it seems likely that abstract intelligence isn’t just innate but also requires stimulation for being learned.
(Of course, it’s possible that wild orcas don’t really learn abstract reasoning either, and instead just hunting or so.)
- ^
Can be checked from table here. (I checked it a few months ago and I think back then there was another “(estimated) birthdate” column which made the checking easier (rather than calculating from “age”), but possible I misremember.)
- ^
Content warning: The “Background” section describes heavy abuse.
- ^
When asking claude for more examples, it wrote:
Romanian Orphanage Studies
Children raised in severely understaffed Romanian orphanages during the Ceaușescu era showed lasting deficits:
Those adopted after age 6 months showed persistent cognitive impairments
Later-adopted children (after age 2) showed more severe and permanent deficits
Brain scans revealed reduced brain volume and activity that persisted into adolescence
Cognitive impairments correlated with duration of institutionalization
The Bucharest Early Intervention Project
This randomized controlled study followed institutionalized children who were either:
Placed in foster care at different ages, or
Remained in institutional care
Key findings:
Children placed in foster care before age 2 showed significant cognitive recovery
Those placed after age 2 showed persistent IQ deficits despite intervention
Executive functioning deficits remained even with early intervention
Isolated Cases: Isabelle and Victor
Isabelle: Discovered at age 6 after being isolated with her deaf-mute mother, showed initial severe impairments but made remarkable recovery with intervention, demonstrating that recovery is still possible before age 6-7
Victor (the “Wild Boy of Aveyron”): Found at approximately age 12, made limited progress despite years of dedicated intervention, similar to Genie
Of course, it’s possible there’s survivorship bias and actually a larger fraction recover. It’s also possible that cognitive deficits are rather due to malnurishment or so.
Seems totally unrelated to my post but whatever:
My p(this branch of humanity won’t fulfill the promise of the night sky) is actually more like 0.82 or sth, idk. (I’m even lower on p(everyone will die), because there might be superintelligences in other branches that acausally trade to save the existing lives, though I didn’t think about it carefully.)
I’m chatting 1 hour every 2 weeks with Erik Jenner. We usually talk about AI safety stuff. Otherwise also like 1h every 2 weeks with a person who has sorta similar views to me. Otherwise I currently don’t talk much to people about AI risk.
ok edited to sun. (i used earth first because i don’t know how long it will take to eat the sun, whereas earth seems likely to be feasible to eat quickly.)
(plausible to me that an aligned AI will still eat the earth but scan all the relevant information out of it and later maybe reconstruct it.)
ok thx, edited. thanks for feedback!
(That’s not a reasonable ask, it intervenes on reasoning in a way that’s not an argument for why it would be mistaken. It’s always possible a hypothesis doesn’t match reality, that’s not a reason to deny entertaining the hypothesis, or not to think through its implications. Even some counterfactuals can be worth considering, when not matching reality is assured from the outset.)
Yeah you can hypothesize. If you state it publicly though, please make sure to flag it as hypothesis.
How long until the earth gets eaten? 10th/50th/90th percentile: 3y, 12y, 37y.
Catastrophes induced by narrow capabilities (notably biotech) can push it further, so this might imply that they probably don’t occur.
No it doesn’t imply this, I set this disclaimer “Conditional on no strong governance success that effectively prevents basically all AI progress, and conditional on no huge global catastrophe happening in the meantime:”. Though yeah I don’t particularly expect those to occur.
Ah, thx! Will try.