Out of curiosity — how relevant is Holden’s 2021 PASTA definition of TAI still to the discourse and work on TAI, aside from maybe being used by Open Phil (not actually sure that’s the case)? Any pointers to further reading, say here or on AF etc?
AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement. I will call this sort of technology Process for Automating Scientific and Technological Advancement, or PASTA.3 (I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation.) …
By talking about PASTA, I’m partly trying to get rid of some unnecessary baggage in the debate over “artificial general intelligence.” I don’t think we need artificial general intelligence in order for this century to be the most important in history. Something narrower—as PASTA might be—would be plenty for that. …
I don’t particularly expect all of [the above] to happen as part of a single, deliberate development process. Over time, I expect different AI systems to be used for different and increasingly broad tasks, including and especially tasks that help complement human activities on scientific and technological advancement. There could be many different types of AI systems, each with its own revenue model and feedback loop, and their collective abilities could grow to the point where at some point, some set of them is able to do everything (with respect to scientific and technological advancement) that formerly required a human. (For convenience, though, I’ll sometimes refer to such a set as PASTA in the singular.)
When I first read this I thought Holden had pointed to the right working definition, because he worked backward from the outcome that matters (explosive econ growth by debottlenecking innovation by decoupling it from human pop growth; call it “transformative” for short).
In contrast, I think (say) Metaculus’ when AGI? doesn’t use quite the right definition in the “transformative” context, which is fine since I read it as operationalising the sort of AGI we think about in e.g. sci-fi, but given its additional definitional constraints it seems like AI system collectives could start radically transforming the economy before any single robot-embodied Metaculus-approved AGI arrives to resolve the question in the affirmative, rendering the 2030 prediction a bit of a red herring:
We will thus define “an AI system” as a single unified software system that can satisfy the following criteria, all completable by at least some humans.
Able to reliably pass a 2-hour, adversarial Turing test during which the participants can send text, images, and audio files (as is done in ordinary text messaging applications) during the course of their conversation. An ‘adversarial’ Turing test is one in which the human judges are instructed to ask interesting and difficult questions, designed to advantage human participants, and to successfully unmask the computer as an impostor. A single demonstration of an AI passing such a Turing test, or one that is sufficiently similar, will be sufficient for this condition, so long as the test is well-designed to the estimation of Metaculus Admins.
Has general robotic capabilities, of the type able to autonomously, when equipped with appropriate actuators and when given human-readable instructions, satisfactorily assemble a (or the equivalent of a) circa-2021 Ferrari 312 T4 1:8 scale automobile model. A single demonstration of this ability, or a sufficiently similar demonstration, will be considered sufficient.
High competency at a diverse fields of expertise, as measured by achieving at least 75% accuracy in every task and 90% mean accuracy across all tasks in the Q&A dataset developed by Dan Hendrycks et al..
Able to get top-1 strict accuracy of at least 90.0% on interview-level problems found in the APPS benchmark introduced by Dan Hendrycks, Steven Basart et al. Top-1 accuracy is distinguished, as in the paper, from top-k accuracy in which k outputs from the model are generated, and the best output is selected.
By “unified” we mean that the system is integrated enough that it can, for example, explain its reasoning on a Q&A task, or verbally report its progress and identify objects during model assembly. (This is not really meant to be an additional capability of “introspection” so much as a provision that the system not simply be cobbled together as a set of sub-systems specialized to tasks like the above, but rather a single system applicable to many problems.)
I admittedly don’t follow the whole TAI discourse that closely, I just take an occasional interest, but it seems like it leans more towards the latter (maybe sans the robotics component of Metaculus’ definition)?
(The rest of this is an irrelevant series of tangents)
Goal-content integrity for final goals is in a sense even more fundamental than survival as a convergent instrumental motivation. Among humans, the opposite may seem to be the case, but that is because survival is usually part of our final goals. For software agents, which can easily switch bodies or create exact duplicates of themselves, preservation of self as a particular implementation or a particular physical object need not be an important instrumental value. Advanced software agents might also be able to swap memories, download skills, and radically modify their cognitive architecture and personalities. A population of such agents might operate more like a “functional soup” than a society composed of distinct semi-permanent persons.
and even more tangentially — this is far afield of my original question but I’ve always wondered what other folks think of this — Charles Stross’ dystopian depiction below in Accelerando of a possible future civilisational endpoint seems to follow pretty naturally from explosive econ growth of the misaligned sort:
“How much for just the civilization?” asks the Slug.
Pierre looks down at it thoughtfully. It’s not really a terrestrial mollusk: Slugs on Earth aren’t two meters long and don’t have lacy white exoskeletons to hold their chocolate-colored flesh in shape. But then, it isn’t really the alien it appears to be. It’s a defaulting corporate instrument that has disguised itself as a long-extinct alien upload, in the hope that its creditors won’t recognize it if it looks like a randomly evolved sentient. … Actually, the Slug is a surgical disguise. Both it and the quasi-fungal ecosystem have been extinct for millions of years, existing only as cheap stage props in an interstellar medicine show run by rogue financial instruments. The Slug itself is one such self-aware scam, probably a pyramid scheme or even an entire compressed junk bond market in heavy recession, trying to hide from its creditors by masquerading as a life-form.
“Corporations are life-forms back home, too, aren’t they? And we trade them. We give our AIs corporations to make them legal entities, but the analogy goes deeper. Look at any company headquarters, fitted out with works of art and expensive furniture and staff bowing and scraping everywhere –”
” – They’re the new aristocracy. Right?”
“Wrong. When they take over, what you get is more like the new biosphere. Hell, the new primordial soup: prokaryotes, bacteria, and algae, mindlessly swarming, trading money for plasmids.” The Queen passes her consort a wineglass. When he drinks from it, it refills miraculously. “Basically, sufficiently complex resource-allocation algorithms reallocate scarce resources … and if you don’t jump to get out of their way, they’ll reallocate you. I think that’s what happened inside the Matrioshka brain we ended up in: Judging by the Slug it happens elsewhere, too. You’ve got to wonder where the builders of that structure came from. And where they went. And whether they realized that the destiny of intelligent tool-using life was to be a stepping-stone in the evolution of corporate instruments.”
“Maybe they tried to dismantle the companies before the companies spent them.” Pierre looks worried. “Running up a national debt, importing luxurious viewpoint extensions, munching exotic dreams. Once they plugged into the Net, a primitive Matrioshka civilization would be like, um.” He pauses. “Tribal. A primitive postsingularity civilization meeting the galactic net for the first time. Overawed. Wanting all the luxuries. Spending their capital, their human – or alien – capital, the meme machines that built them. Until there’s nothing left but a howling wilderness of corporate mechanisms looking for someone to own.”
(I think of this as Stross’ answer to the Fermi paradox: “they got dissembled by corporations, because the destiny of intelligent tool-using life was to be a stepping-stone in the evolution of corporate instruments”. Very disquieting the first time I read it. Ivan Vendrov’s Meditations on machinic desire gives me the same vibe.)
Out of curiosity — how relevant is Holden’s 2021 PASTA definition of TAI still to the discourse and work on TAI, aside from maybe being used by Open Phil (not actually sure that’s the case)? Any pointers to further reading, say here or on AF etc?
When I first read this I thought Holden had pointed to the right working definition, because he worked backward from the outcome that matters (explosive econ growth by debottlenecking innovation by decoupling it from human pop growth; call it “transformative” for short).
In contrast, I think (say) Metaculus’ when AGI? doesn’t use quite the right definition in the “transformative” context, which is fine since I read it as operationalising the sort of AGI we think about in e.g. sci-fi, but given its additional definitional constraints it seems like AI system collectives could start radically transforming the economy before any single robot-embodied Metaculus-approved AGI arrives to resolve the question in the affirmative, rendering the 2030 prediction a bit of a red herring:
I admittedly don’t follow the whole TAI discourse that closely, I just take an occasional interest, but it seems like it leans more towards the latter (maybe sans the robotics component of Metaculus’ definition)?
(The rest of this is an irrelevant series of tangents)
Tangentially, Drexler’s CAIS model seems PASTA-flavored; Bostrom’s “functional soup” seems midway as it presupposes agents but not necessarily the kind most folks think of
and even more tangentially — this is far afield of my original question but I’ve always wondered what other folks think of this — Charles Stross’ dystopian depiction below in Accelerando of a possible future civilisational endpoint seems to follow pretty naturally from explosive econ growth of the misaligned sort:
(I think of this as Stross’ answer to the Fermi paradox: “they got dissembled by corporations, because the destiny of intelligent tool-using life was to be a stepping-stone in the evolution of corporate instruments”. Very disquieting the first time I read it. Ivan Vendrov’s Meditations on machinic desire gives me the same vibe.)