If we consider the TM to be “infinitely more valuable” than the rest of our life as I suggested might make sense in the post, then we would accept whenever . We will never accept if i.e. accepting does not decrease the description length of the TM.
kman
Karma: 1,275
Time is not the bottleneck (on making progress thinking about difficult things)
Right. I think that if we assign measure inverse to the exponent of the shortest description length and assume that the probability increases the description length of the physically instantiated TM by (because the probability is implemented through reality branching which means more bits are needed to specify the location of the TM, or something like that), then this actually has a numerical solution depending on what the description lengths end up being and how much we value this TM compared to the rest of our life.
Say is the description length of our universe and is the length of the description of the TM’s location in our universe when the lottery is accepted, is the description length of the location of “the rest of our life” from that point when the lottery is accepted, is the next shortest description of the TM that doesn’t rely on embedding in our universe, is how much we value the TM and is how much we value the rest of our life. Then we should accept the lottery for any , if I did that right.
I see. When I wrote
such a TM embedded in our physical universe at some point in the future (supposing such a thing is possible)
I implicitly meant that the embedded TM was unbounded, because in the thought experiment our physics turned out to support such a thing.
physicality of the initial states of a TM doesn’t make its states from sufficiently distant future any more physically computed
I’m not sure what you mean by this.
Let’s suppose the description length of our universe + bits needed to specify the location of the TM was shorter than any other way you might wish to describe such a TM. So with the lottery, you are in some sense choosing whether this TM gets a shorter or longer description.
Suppose I further specify the “win condition” to be that you are, through some strange sequence of events, able to be uploaded in such a TM embedded in our physical universe at some point in the future (supposing such a thing is possible), and that if you do not accept the lottery then no such TM will ever come to be embedded in our universe. The point being that accepting the lottery increases the measure of the TM. What’s your answer then?
Unbounded Intelligence Lottery
Sure, having just a little bit more general optimization power lets you search slightly deeper into abstract structures, opening up tons of options. Among human professions, this may be especially apparent in mathematics. But that doesn’t make it any less scary?
Like, I could have said something similar about the best vs. average programmers/”hackers” instead; there’s a similarly huge range of variation there too. Perhaps that would have been a better analogy, since the very best hackers have some more obviously scary capabilities (e.g. ability to find security vulnerabilities).
It’s certainly plausible that something like this pumps in quite a bit of variation on top of the genetics, but I don’t think it detracts much from the core argument: if you push just a little harder on a general optimizer, you get a lot more capabilities out.
Specialization on different topics likely explains much more than algorithmic tweaks explain.
That the very best mathematicians are generally less specialized than their more average peers suggests otherwise.
There are other reasons why top mathematicians could have better output compared to average mathematicians. They could be working on more salient problems, there’s selection bias in who we call a “top mathematician”, they could be situated in an intellectual microcosm more suitable for mathematical progress, etc.
Do you really think these things contribute much to a factor of a thousand? Roughly speaking, what I’m talking about here is how much longer it would take for an average mathematician to reproduce the works of Terry Tao (assuming the same prior information as Terry had before figuring out the things he figured out, of course).
However, those log(n) bits of optimization pressure are being directly applied towards that goal, and it’s not easy to have a learning process that applies optimization pressure in a similarly direct manner (as opposed to optimizing for something like “ability to do well on this math problem dataset”).
I think Terry Tao would do noticeably much better on a math problem dataset compared to most other mathematicians! This is where it’s important to note that “optimization in vs. optimization out” is not actually a single “steepness” parameter, but the shape of a curve. If the thing you’re optimizing doesn’t already have the rough shape of an optimizer, then maybe you aren’t really managing to do much meta-optimization. In other words, the scaling might not be very steep because, as you said, it’s hard to figure out exactly how to direct “dumb” (i.e. SGD) optimization pressure.
But suppose you’ve trained an absolutely massive model that’s managed to stumble onto the “rough shape of an optimizer” and is now roughly human-level. It seems obvious to me that you don’t need to push on this thing very hard to get what we would recognize as massive performance increases for the reason above: it’s not very hard to pick out a Terry Tao from the Earth’s supply of mathematicians, even by dumb optimization on a pretty simple metric (such as performance on some math dataset).
Finally, AI Impacts has done a number of investigations into how long it took for AI systems to go from ~human level to better than human level in different domains. E.g., it took 10 years for diagnosis of diabetic retinopathy. I think this line of research is more directly informative on this question.
I don’t see this as very informative about how optimizers scale as you apply meta-optimization. If the thing you’re optimizing is not really itself an optimizer (e.g. a narrow domain tool), then what you’re measuring is more akin to the total amount of optimization you’ve put into it, rather than the strength of the optimizer you’ve produced by applying meta-optimization.
A compressed take on recent disagreements
I doubt you could use numpy to compute this efficiently, since (afaik) numpy only gives you a big speedup on very regular vector/matrix type computations, which this is not.
Do you think it would be a good idea to delete this and repost it at the beginning of August?
Alternatively, I could repost an easier version in a month, since I’d be shocked if anyone solved this one. Though I guess that was part of the point—to show that induction of fairly simple programs is super hard for humans in general. The previous challenge was too easy because each element in the output sequence was a simple function of two elements earlier in the sequence (and the ‘elements’ were easy to identify as they were output in the standard floating point format). On the other hand, it would be neat to make tough-but-solvable program induction challenges a recurring monthly thing as you suggested. Thoughts?
Challenge: A Much More Alien Message
This is true, but the farther out into the tails of the distribution we get the more likely we are to see negative effects that from traits that aren’t part of the index we’re selecting on.
True, but we wouldn’t need to strictly select for G by association with IQ via GWASes. I suspect G variation is largely driven by mutation load, in which case simply replacing each rare variant with one of its more common counterparts should give you a huge boost while essentially ruling out negative pleiotropy. To hedge your bets you’d probably want to do a combined approach.
I guess there’s some risk that rare variants are involved in people who, e.g., tend to take x-risk very seriously, but I doubt this. I suspect that, to whatever extent this is heritable, it’s controlled by polygenic variation over relatively common variants at many loci. So if you started out with the genomes of people who care lots about x-risk and then threw out all the rare variants, I predict you’d end up with hugely G boosted people who are predisposed to care about x-risk.
As you pointed out, this is moot if genome synthesis is out of reach.
I mostly think the value would be in more actual understanding of alignment difficulties among people working on AI capabilities.
Seems sensible.
Even the brightest geniuses don’t really start having much of an impact on a field until about 20. And it takes further time for ideas to spread, so perhaps they’d need to reach the age of 30.
We could probably create humans vastly smarter than have ever previously existed with full genome synthesis, who could have a huge impact at a much younger age. But otherwise I agree.
Another short-term technology not even mentioned on your list is gamete sequencing. Sperm and eggs are produced in groups of four, with two complementary pairs per stem cell. If we could figure out how to get a good enough read from three of those cells, we could infer the genome of the fourth and pair up the best sperm and egg. That would naively allow us to double the gain, so 24 points.
Wouldn’t it be a factor of sqrt(2), not double?
There are other technologies like in-vitro oogenesis that could raise the gain by perhaps 50% (assuming we could produce a couple of thousand embryos). And there are groups that are working on that right now.
That sounds fairly promising and worth looking into.
I don’t think genome synthesis is likely to be possible in time. CRISPR or some other editing technique might work in the next 10 years, but the public seems to be much less comfortable with editing as opposed to selection, so that might be more politically difficult.
Agreed, which makes my previous point somewhat moot. I’m tempted to say we should at least keep synthesis in the back of our minds in case the problems on the critical path end up being easier than expected.
Lastly, even if we could create such a predictor, what weirdo parents would select for “likely to work on x-risk-reduction”? The parents themselves would have to be convinced that x-risk is a problem, so it’s a somewhat circular solution.
Alignment-problem-aware people could be early adopters of embryo-selection-for-G. There are lots of smart alignment-problem-aware people who read this forum and may be open to this idea, so it’s not necessarily circular.
I am very nervous about any solutions which require the government to enforce selection for certain traits.
I think it’s super unlikely we’d be able to get this sort of large scale coordination anyways.
The only strategy that seems viable to me is enhanced intelligence + changing the memetic environment. I don’t think genetics is going to provide a substitute for the work that has to be done by us stone-brainers to convince more people that misaligned AI is a serious threat.
I don’t think large scale awareness is necessary (see my above point). Even if you could do it, pushing for large scale awareness could backfire by drawing the wrong sort of attention (e.g. by resulting in public outrage about selection-for-G so politicians move to ban it). Though I admittedly don’t place much confidence in my current ability to gauge the likelihood of this sort of thing. More awareness of the alignment problem is probably good.
I am also optimistic that more intelligent people would better grasp the arguments about AI safety and other sources of X-risk. There’s also some research about intelligent people’s disproportionate tendency to support enforcement of rules encouraging positive-sum cooperation that I wrote about in my first post on genetic engineering, so I can see this potentially helping with the coordination aspects of AI and other fields.
Agreed, society wide gains in G would likely have the general effect of raising the sanity waterline.
Anyhow, I’ve updated slightly towards focusing more on thinking about near-term embryo selection strategies as a result of reading and responding to this.
(Edited because I don’t think my original terse reply made my thoughts on this very clear)
If we’re in a (very) long timeline world, I suspect the default thing that ends up happening is that embryo selection is gradually adopted, and G slowly rises population-wide. The reason timelines are long in such a world is that AGI ended up being way harder than it currently looks, so the gradually rising G levels would indeed increase the probability that unaligned AGI is created, unless this somewhat-higher-G world also manages to get large scale coordination right (don’t hold your breath). Alignment research would directly benefit from more capable researchers, and would probably benefit from far greater public awareness than it currently receives (due to generally higher sanity and also just more time for the ideas to percolate into the mainstream), which in turn means far more researchers working on it. People in alignment-aware communities would likely be early adopters of embryo selection, which could give alignment a head start (this is one strategy we might wish to consider: the point of my post was to get us to start thinking about these sorts of strategies).
If we’re only in a medium~longish timeline world (AGI in the latter half of this century, say) then there won’t be enough time for this sort of large scale adoption: a quick G boosting intervention would be used by a small group of early-adopters long before it catches on more broadly. So, strategically, we’d want to be thinking about making sure that the “small group of early-adopters” is alignment-aware.
Good point, I didn’t address this at all in the post. Germline editing is indeed outside the current Overton window. One thing I’m curious about is whether there are any shreds of hope that we might be able to accelerate any of the relevant technical research: one thing this implies is not specifically focusing on the use case of enhancement, to avoid attracting condemnation (which would risk slowing existing research due to e.g. new regulations being levied).
For some techniques this seems harder than for others: iterated embryo selection is pretty clearly meant for enhancement (which could also mean animal enhancement, i.e. efficient livestock breeding). The Cas9 stuff has lots of potential uses, so it’s currently being heavily pursued despite norms. There’s also lots of ongoing work on the synthesis of simple genomes (e.g. for bacteria), with many companies offering synthesis services. Of course, the problems I identified as likely being on the critical path to creating modal human genomes are pretty enhancement specific (again, the only other application that comes to mind is making better livestock) which is unfortunate, given the massive (and quick!) upside of this approach if you can get it to work.
Since I currently have the slack to do so, I’m going to try getting into a balanced biphasic schedule to start with. If I actually manage to pull it off I’ll make another post about it.