-
You people are somewhat crazy overconfident about humanity knowing enough to make AGI this decade. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
-
One hope on the scale of decades is that strong germline engineering should offer an alternative vision to AGI. If the options are “make supergenius non-social alien” and “make many genius humans”, it ought to be clear that the latter is both much safer and gets most of the hypothetical benefits of the former.
TsviBT
Karma: 5,445
How many proteins are left after these 60 seconds?
I wonder if there’s a large-ish (not by percentage, but still) class of mechanically self-destructing proteins? E.g. suppose you have something like this:
RRRRRAARRRRRRAAAEEEEEEEXXXXXXXXXXXXXXXEEEEEEEE
where R could be any basic amino, E any acidic one, A any neutral one. And then the Xs are some sequence that eventually forms a strong extended thing. So the idea is that you get strong bonds between the second R island and the first E island, and between the first R island and the second E island. Then the X segment pulls the two Es apart, ripping the protein between the two R islands. like:
ARRRRRA <<--------------->> ARRRRRR
A.| | | | | | …...................................… | | | | | |
AEEEEEEXXXXXXXXXXXXEEEEEEE
This is 100% made up, no idea if anything like it could happen.
Eternity can seem kinda terrifying.
A lifeist doesn’t say “You must decide now to live literally forever no matter what happens.”!
In what sense were you lifeist and now deathist? Why the change?
General warning / PSA: arguments about the limits of selection power are true… BUT with an important caveat: selection power is NOT limited to postnatal organisms. Gametogenesis and fertilization involve many millions of cells undergoing various selective filters (ability to proliferate, ability to appropriately respond to regulatory signals, passing genetic integrity checks, undergoing meiosis, physically passing through reproductive organs, etc.). This also counts as selection power available to the species-evolution.
Say a “deathist” is someone who says “death is net good (gives meaning to life, is natural and therefore good, allows change in society, etc.)” and a “lifeist” (“anti-deathist”) is someone who says “death is net bad (life is good, people should not have to involuntarily die, I want me and my loved ones to live)”. There are clearly people who go deathist → lifeist, as that’s most lifeists (if nothing else, as an older kid they would have uttered deathism, as the predominant ideology). One might also argue that young kids are naturally lifeist, and therefore most people have gone lifeist → deathist once. Are there people who have gone deathist → lifeist → deathist? Are there people who were raised lifeist and then went deathist?
(Still impressive and interesting of course, just not literally SOTA.)
According to the article, SOTA was <1% of cells converted into iPSCs
I don’t think that’s right, see https://www.cell.com/cell-stem-cell/fulltext/S1934-5909(23)00402-2
metapreferences are important, but their salience is way out of proportion to their importance.
You mean the salience is too high? On the contrary, it’s too low.
one of the most immediate natural answers is “metapreferences!”.
Of course, this is not an answer, but a question-blob.
as evidenced by experiences like “I thought I wanted X, but in hindsight I didn’t”
Yeah I think this is often, maybe almost always, more like “I hadn’t computed / decided to not want [whatever Thing-like thing X gestured at], and then I did compute that”.
a last-line fallback for extreme cases
It’s really not! Our most central values are all of the proleptic (pre-received; foreshadowed) type: friendship, love, experience, relating, becoming. They all can only be expressed in an either vague or incomplete way: “There’s something about this person / myself / this collectivity / this mental activity that draws me in to keep walking that way.”. Part of this is resolvable confusion, but probably not all of it. Part of the fun of relating with other people is that there’s a true open-endedness; you get to cocreate something non-pre-delimited, find out what another [entity that is your size / as complex/surprising/anti-inductive as you] is like, etc. “Metapreferences” isn’t an answer of course, but there’s definitely a question that has to be asked here, and the answer will fall under “metapreferences” broadly construed, in that it will involve stuff that is ongoingly actively determining [all that stuff we would call legible values/preferences].
“What does it even mean to be wrong about our own values? What’s the ground truth?”
Ok we can agree that this should point the way to the right questions and answers, but it’s an extremely broad question-blob.
“The Future Loves You: How and Why We Should Abolish Death” by Dr Ariel Zeleznikow-Johnston is now available to buy. I haven’t read it, but I expect it to be a definitive anti-deathist monograph. https://www.amazon.com/Future-Loves-You-Should-Abolish-ebook/dp/B0CW9KTX76
The description (copied from Amazon):
A brilliant young neuroscientist explains how to preserve our minds indefinitely, enabling future generations to choose to revive us
Just as surgeons once believed pain was good for their patients, some argue today that death brings meaning to life. But given humans rarely live beyond a century – even while certain whales can thrive for over two hundred years – it’s hard not to see our biological limits as profoundly unfair. No wonder then that most people nearing death wish they still had more time.
Yet, with ever-advancing science, will the ends of our lives always loom so close? For from ventilators to brain implants, modern medicine has been blurring what it means to die. In a lucid synthesis of current neuroscientific thinking, Zeleznikow-Johnston explains that death is no longer the loss of heartbeat or breath, but of personal identity – that the core of our identities is our minds, and that our minds are encoded in the structure of our brains. On this basis, he explores how recently invented brain preservation techniques now offer us all the chance of preserving our minds to enable our future revival.
Whether they fought for justice or cured diseases, we are grateful to those of our ancestors who helped craft a kinder world – yet they cannot enjoy the fruits of the civilization they helped build. But if we work together to create a better future for our own descendants, we may even have the chance to live in it. Because, should we succeed, then just maybe, the future will love us enough to bring us back and share their world with us.
But like, I wouldn’t be surprised if, say, someone trained something that performed comparably to LLMs on a wide variety of benchmarks, using much less “data”… and then when you look into it, you find that what they were doing was taking activations of the LLMs and training the smaller guy on the activations. And I’ll be like, come on, that’s not the point; you could just as well have “trained” the smaller guy by copy-pasting the weights from the LLM and claimed “trained with 0 data!!”. And you’ll be like “but we met your criterion!” and I’ll just be like “well whatever, it’s obviously not relevant to the point I was making, and if you can’t see that then why are we even having this conversation”. (Or maybe you wouldn’t do that, IDK, but this sort of thing—followed by being accused of “moving the goal posts”—is why this question feels frustrating to answer.)
But ok:
Come up, on its own, with many math concepts that mathematicians consider interesting + mathematically relevant on a similar level to concepts that human mathematicians come up with.
Do insightful science on its own.
Perform at the level of current LLMs, but with 300x less training data.
I did give a response in that comment thread. Separately, I think that’s not a great standard, e.g. as described in the post and in this comment https://www.lesswrong.com/posts/i7JSL5awGFcSRhyGF/shortform-2?commentId=zATQE3Lhq66XbzaWm :
Second, 2024 AI is specifically trained on short, clear, measurable tasks. Those tasks also overlap with legible stuff—stuff that’s easy for humans to check. In other words, they are, in a sense, specifically trained to trick your sense of how impressive they are—they’re trained on legible stuff, with not much constraint on the less-legible stuff (and in particular, on the stuff that becomes legible but only in total failure on more difficult / longer time-horizon stuff).
In fact, all the time in real life we make judgements about things that we couldn’t describe in terms that would be considered well-operationalized by betting standards, and we rely on these judgements, and we largely endorse relying on these judgements. E.g. inferring intent in criminal cases, deciding whether something is interesting or worth doing, etc. I should be able to just say “but you can tell that these AIs don’t understand stuff”, and then we can have a conversation about that, without me having to predict a minimal example of something which is operationalized enough for you to be forced to recognize it as judgeable and also won’t happen to be surprisingly well-represented in the data, or surprisingly easy to do without creativity, etc.
My p(AGI by 2045) is higher because there’s been more time for algorithmic progress, maybe in the ballpark of 20%. I don’t have strong opinions about how much people will do huge training runs, though maybe I’d be kinda skeptical that people would be spending $10^11 or $10^12 on runs, if their $10^10 runs produced results not qualitatively very different from their $10^9 runs. But IDK, that’s both a sociological question and a question of which lesser capabilities happen to get unlocked at which exact training run sizes given the model architectures in a decade, which of course IDK. So yeah, if it’s 10^30 but not much algorithmic progress, I doubt that gets AGI.
I still basically think all of this, and still think this space doesn’t understand it, and thus has an out-of-whack X-derisking portfolio.
If I were writing it today, I’d add this example about search engines from this comment https://www.lesswrong.com/posts/oC4wv4nTrs2yrP5hz/what-are-the-strongest-arguments-for-very-short-timelines?commentId=2XHxebauMi9C4QfG4 , about induction on vague categories like “has capabilities”:
Would you say the same thing about the invention of search engines? That was a huge jump in the capability of our computers. And it looks even more impressive if you blur out your vision—pretend you don’t know that the text that comes up on your screen is written by a human, and pretend you don’t know that search is a specific kind of task distinct from a lot of other activity that would be involved in “True Understanding, woooo”—and just say “wow! previously our computers couldn’t write a poem, but now with just a few keystrokes my computer can literally produce Billy Collins level poetry!”.
I might also try to explain more how training procedures with poor sample complexity tend to not be on an unbounded trajectory.
What I mainline expect is that yes, a few OOMs more of compute and efficiency will unlock a bunch of new things to try, and yes some of those things will make some capabilities go up a bunch, in the theme of o3. I just also expect that to level off. I would describe myself as “confident but not extremely confident” of that; like, I give 1 or 2% p(doom) in the next 10ish years, coming from this possibility (and some more p(doom) from other sources). Why expect it to level off? Because I don’t see good evidence of “a thing that wouldn’t level off”; the jump made by LLMs of “now we can leverage huge amounts of data and huge amounts of compute at all rather than not at all” is certainly a jump, but I don’t see why to think it’s a jump to an unbounded trajectory.
The standard way to measure compute is FLOPS. Besides other problems, this measure has two major flaws: First, no one cares exactly how many FLOPS you have; we want to know the order of magnitude without having to incant “ten high”. Second, it sounds cute, even though it’s going to kill us.
I propose an alternative: Digital Orders Of Magnitude (per Second), or DOOM(S).
Yes, it is: tsvibt
I’m not sure what “concrete” is supposed to mean; for the one or two senses I immediately imagine, no, I would say the feedback is indeed concrete. In terms of consensus/outcome, no, I think the feedback is actually concrete. There is a difficulty, which is that there’s a much smaller set of people to whom the outcomes are visible.
As an analogy/example: feedback in higher math. It’s “nonconcrete” in that it’s “just verbal arguments” (and translating those into something much more objective, like a computer proof, is a big separate long undertaking). And there’s a much smaller set of people who can tell what statements are true in the domain. There might even be a bunch more people who have opinions, and can say vaguely related things that other non-experts can’t distinguish from expert statements, and who therefore form an apparent consensus that’s wrong + ungrounded. But one shouldn’t conclude from those facts that math is less real, or less truthtracking, or less available for communities to learn about directly.
It is still the case that some people don’t sign up for cryonics simply because it takes work to figure out the process / financing. If you do sign up, it would therefore be a public service to write about the process.