If they’re that smart, why will they need to be persuaded?
kman
Karma: 1,237
What would it mean for them to have an “ASI slave”? Like having an AI that implements their personal CEV?
(And that’s not even addressing how you could get super-smart people to work on the alignment problem).
I mean if we actually succeeded at making people who are +7 SD in a meaningful way, I’d expect that at least good chunk of them would figure out for themselves that it makes sense to work on it.
In that case I’d repeat GeneSmith’s point from another comment: “I think we have a huge advantage with humans simply because there isn’t the same potential for runaway self-improvement.” If we have a whole bunch of super smart humans of roughly the same level who are aware of the problem, I don’t expect the ruthless ones to get a big advantage.
I mean I guess there is some sort of general concern here about how defense-offense imbalance changes as the population gets smarter. Like if there’s some easy way to destroy the world that becomes accessible with IQ > X, and we make a bunch of people with IQ > X, and a small fraction of them want to destroy the world for some reason, are the rest able to prevent it? This is sort of already the situation we’re in with AI: we look to be above the threshold of “ability to summon ASI”, but not above the threshold of “ability to steer the outcome”. In the case of AI, I expect making people smarter differentially speeds up alignment over capabilities: alignment is hard and we don’t know how to do it, while hill-climbing on capabilities is relatively easy and we already know how to do it.
I should also note that we have the option of concentrating early adoption among nice, sane, x-risk aware people (though I also find this kind of cringe in a way and predict this would be an unpopular move). I expect this to happen by default to some extent.
like the fact that any control technique on AI would be illegal because of it being essentially equivalent to brainwashing, such that I consider AIs much more alignable than humans
A lot of (most?) humans end up nice without needing to be controlled / “aligned”, and I don’t particularly expect this to break if they grow up smarter. Trying to control / “align” them wouldn’t work anyway, which is also what I predict will happen with sufficiently smart AI.
I mean hell, figuring out personality editing would probably just make things backfire. People would choose to make their kids more ruthless, not less.
Not at all obvious to me this is true. Do you mean to say a lot of people would, or just some small fraction, and you think a small fraction is enough to worry?
How to Make Superbabies
“DL training == human learning” is a bad analogy
I think I mostly agree with the critique of “pause and do what, exactly?”, and appreciate that he acknowledged Yudkowsky as having a concrete plan here. I have many gripes, though.
Whatever name they go by, the AI Doomers believe the day computers take over is not far off, perhaps as soon as three to five years from now, and probably not longer than a few decades. When it happens, the superintelligence will achieve whatever goals have been programmed into it. If those goals are aligned exactly to human values, then it can build a flourishing world beyond our most optimistic hopes. But such goal alignment does not happen by default, and will be extremely difficult to achieve, if its creators even bother to try. If the computer’s goals are unaligned, as is far more likely, then it will eliminate humanity in the course of remaking the world as its programming demands. This is a rough sketch, and the argument is described more fully in works like Eliezer Yudkowsky’s essays and Nick Bostrom’s Superintelligence.
This argument relies on several premises: that superintelligent artificial general intelligence is philosophically possible, and practical to build; that a superintelligence would be more or less all-powerful from a mere human perspective; that superintelligence would be “unfriendly” to humanity by default; that superintelligence can be “aligned” to human values by a very difficult engineering program; that superintelligence can be built by current research and development methods; and that recent chatbot-style AI technologies are a major step forward on the path to superintelligence. Whether those premises are true has been debated extensively, and I don’t have anything useful to add to that discussion which I haven’t said before. My own opinion is that these various premises range from “pretty likely but not proven” to “very unlikely but not disproven.”
I’m thoroughly unimpressed with these paragraphs. It’s not completely clear what the “argument” is from the first paragraph, but I’m interpreting it as “superintelligence might be created soon and cause human extinction if not aligned, therefore we should stop”.
Firstly, there’s an obvious
conjunction fallacymultiple stage fallacy thing going on where he broke the premises down into a bunch of highly correlated things and listed them separately to make them sound more far fetched in aggregate. E.g. the 3 claims:[that superintelligence is] practical to build
that superintelligence can be built by current research and development methods
that recent chatbot-style AI technologies are a major step forward on the path to superintelligence
are highly correlated. If you believe (1) there’s a good chance you believe (2), and if you believe (2) then you probably believe (3).
There’s also the fact that (3) implies (2) and (2) implies (1), meaning (3) is logically equivalent to (1) AND (2) AND (3). So why not just say (3)?
I’m also not sure why (3) is even a necessary premise; (2) should be cause enough for worry.
I have more gripes with these paragraphs:
that superintelligent artificial general intelligence is philosophically possible
What is this even doing here? I’d offer AIXI as a very concrete existence proof of philosophical possibility. Or to be less concrete but more correct: “something epistemically and instrumentally efficient relative to all of humanity” is a simple coherent concept. He’s only at “pretty likely but not proven” on this?? What would it even mean for it to be “philosophically impossible”?
That superintelligence can be “aligned” to human values by a very difficult engineering program
Huh? Why would alignment not being achievable by “a very difficult engineering program” mean we shouldn’t worry?
that a superintelligence would be more or less all-powerful from a mere human perspective
It just needs to be powerful enough to replace and then kill us. For example we can very confidently predict that it won’t be able to send probes faster than light, and somewhat less confidently predict that it won’t be able to reverse a secure 4096 bit hash.
Here’s a less multiple-stagey breakdown of the points that are generally contentious among the informed:
humans might soon build an intelligence powerful enough to cause human extinction
that superintelligence would be “unfriendly” to humanity by default
Some other comments:
Of course, there is no indication that massive intelligence augmentation will be developed any time soon, only very weak reasons to suspect that it’s obtainable at all without multiple revolutionary breakthroughs in our understanding both of genetics and of the mind
Human intelligence variation is looking to be pretty simple on a genetic level: lots of variants with small additive effects. (See e.g. this talk by Steve Hsu)
and no reason at all to imagine that augmenting human intelligence would by itself instill the psychological changes towards humility and caution that Yudkowsky desires.
No reason at all? If Yudkowsky is in fact correct, wouldn’t we expect people to predictably come to agree with him as we made them smarter (assuming we actually succeeded at making them smarter in a broad sense)? If we’re talking about adult enhancement, you can also just start out with sane, cautious people and make them smarter.
The plan is a bad one, but it does have one very important virtue. The argument for the plan is at least locally valid, if you grant all of its absurd premises.
I hope I’ve convinced the skeptical reader that the premises aren’t all that absurd?
EDIT: I’d incorrectly referred to the multiple stage fallacy as the “conjunction fallacy” (since it involves a big conjunction of claims, I guess). The conjunction fallacy is when someone assesses P(A & B) > P(A).
[Question] Cryonics considerations: how big of a problem is ischemia?
You acknowledge this but I feel you downplay the risk of cancer—an accidental point mutation in a tumour suppressor gene or regulatory region in a single founder cell could cause a tumour.
For each target the likely off-targets can be predicted, allowing one to avoid particularly risky edits. There may still be issues with sequence-independent off-targets, though I believe these are a much larger problem with base editors than with prime editors (which have lower off-target rates in general). Agree that this might still end up being an issue.
Unless you are using the term “off-target” to refer to any incorrect edit of the target site, and wider unwanted edits—in my community this term referred specifically to ectopic edits elsewhere in the genome away from the target site.
This is exactly it—the term “off-target” was used imprecisely in the post to keep things simple. The thing we’re most worried about here is misedits (mostly indels) at noncoding target sites. We know a target site does something (if the variant there is in fact causal), so we might worry that an indel will cause a big issue (e.g. disabling a promoter binding site). Then again, the causal variant we’re targeting has a very small effect, so maybe the sequence isn’t very sensitive and an indel won’t be a big deal? But it also seems perfectly possible that the sequence could be sensitive to most mutations while permitting a specific variant with a small effect. The effect of an indel will at least probably be less bad than in a coding sequence, where it has a high chance of causing a frameshift mutation and knocking out the coded-for protein.
The important figure of merit for editors with regards to this issue is the ratio of correct edits to misedits at the target site. In the case of prime editors, IIUC, all misedits at the target site are reported as “indels” in the literature (base editors have other possible outcomes such as bystander edits or conversion to the wrong base). Some optimized prime editors have edit:indel ratios of >100:1 (best I’ve seen so far is 500:1, though IIUC this was just at two target sites, and the rates seem to vary a lot by target site). Is this good enough? I don’t know, though I suspect not for the purposes of making a thousand edits. It depends on how large the negative effects of indels are at noncoding target sites: is there a significant risk the neuron gets borked as a result? It might be possible to predict this on a site-by-site basis with a better understanding of the functional genomics of the sequences housing the causal variants which affect polygenic traits (which would also be useful for finding the causal variants in the first place without needing as much data).
This seems unduly pessimistic to me. The whole interesting thing about g is that it’s easy to measure and correlates with tons of stuff. I’m not convinced there’s any magic about FSIQ compared to shoddier tests. There might be important stuff that FSIQ doesn’t measure very well that we’d ideally like to select/edit for, but using FSIQ is much better than nothing. Likewise, using a poor man’s IQ proxy seems much better than nothing.
This may have missed your point, you seem more concerned about selecting for unwanted covariates than ‘missing things’, which is reasonable. I might remake the same argument by suspecting that FSIQ probably has some weird covariates too—but that seems weaker. E.g. if a proxy measure correlates with FSIQ at .7, then the ‘other stuff’ (insofar as it is heritable variation and not just noise) will also correlate with the proxy at .7, and so by selecting on this measure you’d be selecting quite strongly for the ‘other stuff’, which, yeah, isn’t great. FSIQ, insofar as it had any weird unwanted covariates, would probably much less correlated with them than .7
Non-coding means any sequence that doesn’t directly code for proteins. So regulatory stuff would count as non-coding. There tend to be errors (e.g. indels) at the edit site with some low frequency, so the reason we’re more optimistic about editing non-coding stuff than coding stuff is that we don’t need to worry about frameshift mutations or nonsense mutations which knock-out the gene where they occur. The hope is that an error at the edit site would have a much smaller effect, since the variant we’re editing had a very small effect in the first place (and even if the variant is embedded in e.g. a sensitive binding site sequence, maybe the gene’s functionality can survive losing a binding site, so at least it isn’t catastrophic for the cell). I’m feeling more pessimistic about this than I was previously.
Another thing: if you have a test for which g explains the lion’s share of the heritable variance, but there are also other traits which contribute heritable variance, and the other traits are similarly polygenic as g (similar number of causal variants), then by picking the top-N expected effect size edits, you’ll probably mostly/entirely end up editing variants which affect g. (That said, if the other traits are significantly less polygenic than g then the opposite would happen.)
I should mention, when I wrote this I was assuming a simple model where the causal variants for g and the ‘other stuff’ are disjoint, which is probably unrealistic—there’d be some pleiotropy.
Even out of this 10%, slightly less than 10% of that 10% responded to a 98-question survey, so a generous estimate of how many of their customers they got to take this survey is 1%. And this was just a consumer experience survey, which does not have nearly as much emotional and cognitive friction dissuading participants as something like an IQ test.
What if 23&me offered a $20 discount for uploading old SAT scores? I guess someone would set up a site that generates realistically distributed fake SAT scores that everyone would use. Is there a standardized format for results that would be easy to retrieve and upload but hard to fake? Eh, idk, maybe not. Could a company somehow arrange to buy the scores of consenting customers directly from the testing agency? Agree that this seems hard.
Statistical models like those involved in GWASes follow one of many simple rules: crap in, crap out. If you want to find a lot of statistically significant SNPs for intelligence and you try using a shoddy proxy like standardized test score or an incomplete IQ test score as your phenotype, your GWAS is going to end up producing a bunch of shoddy SNPs for “intelligence”. Sample size (which is still an unsolved problem for the reasons aforementioned) has the potential to make up for obtaining a low amount of SNPs that have genome-wide significance, but it won’t get rid of entangled irrelevant SNPs if you’re measuring something other than straight up full-scale IQ.
This seems unduly pessimistic to me. The whole interesting thing about g is that it’s easy to measure and correlates with tons of stuff. I’m not convinced there’s any magic about FSIQ compared to shoddier tests. There might be important stuff that FSIQ doesn’t measure very well that we’d ideally like to select/edit for, but using FSIQ is much better than nothing. Likewise, using a poor man’s IQ proxy seems much better than nothing.
Thanks for leaving such thorough and thoughtful feedback!
You could elect to use proxy measures like educational attainment, SAT/ACT/GRE score, most advanced math class completed, etc., but my intuition is that they are influenced by too many things other than pure g to be useful for the desired purpose. It’s possible that I’m being too cynical about this obstacle and I would be delighted if someone could give me good reasons why I’m wrong.
The SAT is heavily g-loaded: r = .82 according to Wikipedia, so ~2/3 of the variance is coming from g, ~1/3 from other stuff (minus whatever variance is testing noise). So naively, assuming no noise and that the genetic correlations mirror the phenotype correlations, if you did embryo selection on SAT, you’d be getting .82*h_pred/sqrt(2) SDs g and .57*h_pred/sqrt(2) SDs ‘other stuff’ for every SD of selection power you exert on your embryo pool (h_pred^2 is the variance in SAT explained by the predictor, we’re dividing by sqrt(2) because sibling genotypes have ~1/2 the variance as the wider population). Which is maybe not good; maybe you don’t want that much of the ‘other stuff’, e.g. if it includes personality traits.
It looks like the SAT isn’t correlated much with personality at all. The biggest correlation is with openness, which is unsurprising due to the correlation between openness and IQ—I figured conscientiousness might be a bit correlated, but it’s actually slightly anticorrelated, despite being correlated with GPA. So maybe it’s more that you’re measuring specific abilities as well as g (e.g. non-g components of math and verbal ability).
Another thing: if you have a test for which g explains the lion’s share of the heritable variance, but there are also other traits which contribute heritable variance, and the other traits are similarly polygenic as g (similar number of causal variants), then by picking the top-N expected effect size edits, you’ll probably mostly/entirely end up editing variants which affect g. (That said, if the other traits are significantly less polygenic than g then the opposite would happen.)
this would be extremely expensive, as even the cheapest professional IQ tests cost at least $100 to administer
Getting old SAT scores could be much cheaper, I imagine (though doing this would still be very difficult). Also, as GeneSmith pointed out we aren’t necessarily limited to western countries. Assembling a large biobank including IQ scores or a good proxy might be much cheaper and more socially permissible elsewhere.
The barriers involved in engineering the delivery and editing mechanisms are different beasts.
I do basically expect the delivery problem will gated by missing breakthroughs, since otherwise I’d expect the literature to be full of more impressive results than it actually is. (E.g. why has no one used angiopep coated LNPs to deliver editors to mouse brains, as far as I can find? I guess it doesn’t work very well? Has anyone actually tried though?)
Ditto for editors, though I’m somewhat more optimistic there for a handful of reasons:
sequence dependent off-targets can be predicted
so you can maybe avoid edits that risk catastrophic off-targets
unclear how big of a problem errors at noncoding target sites will be (though after reading some replies pointing out that regulatory binding sites are highly sensitive I’m a bit more pessimistic about this than I was)
even if they are a big problem, dCas9-based ABEs have extremely low indel rates and incorrect base conversions, though bystanders are still a concern
though if you restrict yourself to ABEs and are careful to avoid bystanders, your pool of variants to target has shrunk way down
I mean, your basic argument was “you’re trying to do 1000 edits, and the risks will mount with each edit you do”, which yeah, maybe I’m being too optimistic here (e.g. even if not a problem at most target sites, errors will predictably be a big deal at some target sites, and it might be hard to predict which sites with high accuracy).
It’s not clear to me how far out the necessary breakthroughs are “by default” and how much they could be accelerated if we actually tried, in the sense of how electric cars weren’t going anywhere until Musk came along and actually tried (though besides sounding crazy ambitious, maybe this analogy doesn’t really work if breakthroughs are just hard to accelerate with money, and AFAIK electric cars weren’t really held up by any big breakthroughs, just lack of scale). Getting delivery+editors down would have a ton of uses besides intelligence enhancement therapy; you could target any mono/oligo/poly-genic diseases you wanted. It doesn’t seem like the amount of effort currently being put in is concomitant with how much it would be worth, even putting ‘enhancement’ use cases aside.
one could imagine that if every 3rd or 4th or nth neuron is receiving, processing, or releasing ligands in a different way than either the upstream or downstream neurons, the result is some discordance that is more likely to be destructive than beneficial
My impression is neurons are really noisy, and so probably not very sensitive to small perturbations in timing / signalling characteristics. I guess things could be different if the differences are permanent rather than transient—though I also wouldn’t be surprised if there was a lot of ‘spatial’ noise/variation in neural characteristics, which the brain is able to cope with. Maybe this isn’t the sort of variation you mean. I completely agree that its more likely to be detrimental than beneficial, it’s a question of how badly detrimental.
Another thing to consider: do the causal variants additively influence an underlying lower dimensional ‘parameter space’ which then influences g (e.g. degree of expression of various proteins or characteristics downstream of that)? If this is the case, and you have a large number of causal variants per ‘parameter’, then if your cells get each edit with about the same frequency on average, then even if there’s a ton of mosaicism at the variant level there might not be much at the ‘parameter’ level. I suspect the way this would actually work out is that some cells will be easier to transfect than others (e.g. due to the geography of the extracellular space that the delivery vectors need to diffuse through), so you’ll have some cells getting more total edits than others: a mix of cells with better and worse polygenic scores, which might lead to the discordance problems you suggested if the differences are big enough.
For all of the reasons herein and more, it’s my personal prediction that the only ways humanity is going to get vastly smarter by artificial means is through brain machine interfaces or iterative embryo selection.
BMI seems harder than in-vivo editing to me. Wouldn’t you need a massive number of connections (10M+?) to even begin having any hope of making people qualitatively smarter? Wouldn’t you need to find an algorithm that the brain could ‘learn to use’ so well that it essentially becomes integrated as another cortical area or can serve as an ‘expansion card’ for existing cortical areas? Would you just end up bottlenecked by the characteristics of the human neurons (e.g. low information capacity due to noise)?
I don’t think this therapy as OP describes it is possible for reasons that have already been stated by HiddenPrior and other reasons
Can you elaborate on this? We’d really appreciate the feedback.
We’d edit the SNPs which have been found to causally influence the trait of interest in an additive manner. The genome would only become “extremely unlikely” if we made enough edits to push the predicted trait value to an extreme value—which you probably wouldn’t want to do for decreasing disease risk. E.g. if someone has +2 SD risk of developing Alzheimer’s, you might want to make enough edits to shift them to −2 SD, which isn’t particularly extreme.
You’re right that this is a risk with ambitious intelligence enhancement, where we’re actually interested in pushing somewhat outside the current human range (especially since we’d probably need to push the predicted trait value even further in order to get a particular effect size in adults) -- the simple additive model will break down at some point.
Also, due to linkage disequilibrium, there are things that could go wrong with creating “unnatural genomes” even within the current human range. E.g. if you have an SNP with alleles A and B, and there are mutations at nearby loci which are neutral conditional on having allele A and deleterious conditional on having allele B, those mutations will tend to accumulate in genomes which have allele A (due to linkage disequilibrium), while being purged from genomes with allele B. If allele B is better for the trait in question, we might choose it as an edit site in a person with allele A, which could be highly deleterious due to the linked mutations. (That said, I don’t think this situation of large-conditional-effect mutations is particularly likely a priori.)
Promoters (and any non-coding regulatory sequence for that matter) are extremely sensitive to point mutations.
A really important question here is whether the causal SNPs that affect polygenic traits tend to be located in these highly sensitive sequences. One hypothesis would be that regulatory sequences which are generally highly sensitive to mutations permit the occasional variant with a small effect, and these variants are a predominant influence on polygenic traits. This would be bad news for us, since even the best available editors have non-negligible indel rates at target sites.
Another question: there tend to be many enhancers per gene. Is losing one enhancer generally catastrophic for the expression of that gene?
We accounted for inflation of effect sizes due to assortative mating, assuming a mate IQ correlation of 0.4 and total additive heritability of 0.7 for IQ.
IIUC that R = 0.55 number was just the raw correlation between the beta values of the sibling and population GWASes, which is going to be very noisy given the small sample sizes and given that effects are sparse.You can see that the LDSC based estimate is nearly 1, suggesting ~null indirect effects.