Inspired by someone who asked about “representative” AGI x-risk arguments, I wondered: how might an AGI takeover and catastrophe (not necessarily extinction) actually play out in detail? It’s extremely tough to build a realistic mental picture of all possibilities, but looking at one detailed story might help give us a feeling for some of the probability space, despite the unknowability of how AGI will actually behave.
So I set forth to create such a story.
I am personally limited by some factors:
My limited knowledge of ML/DL/AI, and having mere above-average intelligence
My limited knowledge of the tactics of ruthless beings
As the complexity of a situation increases, or as the intelligence of AGI increases, predictability drops. This is part of the reason that AGI companies in my story do not build superintelligence very quickly―I want a story spanning years that I can realistically write―but also (i) I suspect logarithmic intelligence scaling is normal, so building superintelligence isn’t easy with DL-based approaches, (ii) the first inventors are safety-conscious so they know better than to simply build superintelligence and hook it up to the internet, (iii) it seems likely that designers will try to create something humanlike with DL, rather than a “maximizer” or even a “satisficer”, but that they won’t actually know how to do “humanlike”, so AGI will also exhibit some “alienness”.
I want to avoid the most common tropes about AI behavior, as they seem unrealistic (or to the extent they are realistic, have been overexplored already), yet I find it hard to imagine how an AGI mind that is not quite human, and not quite Data from Star Trek, and not a simple maximizer, would behave, e.g. what behavioral flaws and advantages over humans might it exhibit? In this story there will be at least four different AGI designs, so they should show at minimum four distinct sets of behavioral tendencies.
My ADHD and general busy-ness? I wrote most of the story four weeks ago, and then stopped when it got difficult (and I had to return from Christmas break). I give this a CC BY-NC 4.0 license in case you want to improve it yourself.
Another thing I want to do with this story is explore what a non-superintelligent computer-based AGI might use its computing power for that humans can’t. In principle, for instance, a non-superintelligent AGI should be able to do the following simply because it is computer-based:
stay on task (no desire for “leisure”; consistent task prioritization but potentially flawed/pathological)
produce very fast and precise outputs (in binary or Unicode, allowing e.g. direct input into MS Excel, or even direct input into a websocket, as opposed to moving fingers over keyboard keys)
fast reading comprehension (potentially including mental visualization of binary data files)
fast and detailed mental modeling
precision communication with ambiguity detectionplus an ability to invent novel new languages
effortless photographic memory (note: the way recall works is also critically important; computers only recall information insofar as they are programmed to) and an ability to remember people’s exact words, or even exact audio
an ability to attach arbitrary algorithms to one’s central neural network
the ability to clone and modify copies of oneself (if the creator permits it, or by exploiting vulnerabilities in the hosting system)
AGI may also exhibit computer-specific limitations, e.g. system crashes, sudden dropoffs in performance when certain limits are reached, or (as I alluded) pathological behavior that hinders achievement of goals. AGIs may also have design-level limitations, such as “catastrophic forgetting”, poor epistemology, poor ability to do “mental modeling”, poor self-modeling, poor meta-level creativity, etc. Finally, AGI may have instance-level limitations, in which a design could in principle provide a capability, but does not due to e.g. limitations of the training process.
Both positive and negative aspects of computer-based AGI are underdeveloped in the current story. My feeling is that it isn’t unrealistic to assume that the first “v1.0” AGIs in the story will lack the above capabilities (except for staying on task and being faster than humans), but that it would be unrealistic to think the AGIs won’t gain new capabilities somewhat quickly in the story’s competitive environment, and so this is an aspect I need to improve. The big question, though, is what non-obvious consequences do such capabilities lead to? I deliberately simplified the initial situation by limiting the AGIs’ abilities to form memories, but as AGI memory improves, I expect we’ll see nonhuman behaviors like “hoarding credentials” where AGIs memorize every password/token they encounter “just in case”. What else?
Given the current state of our world, I also assume companies, rather than individuals like Noonien Soong, will develop AGI, and I assume companies will be aware of the most obvious AGI ruin failure mode, “building an X maximizer where X is simple”, and therefore not build that.
I also wonder what the community thinks about “emotions”. Are AGI designers likely to create emotions, if so which ones? What might the be analogs, say, of “curiosity”, “boredom”, “interest” and “pleasure” be in AGI? Are “love”, “hate”, “anger”, “envy”, “discomfort” or “lonliness” realistic things to build? If not, what might companies do to make AGIs seem humanlike and make them get along with humans? My assumption has been that AGIs do not have emotions and so (i) lack empathy but can still predict human behavior statistically, as psychopaths do; (ii) are very adaptable to new situations; (iii) have specific training processes designed to encourage ethical behavior / discourage unethical behavior, and may have an additional ethics module on top of that. I also assume AGIs will lack any equivalent of “adrenaline” so cannot speed up in “scary” situations, but that this doesn’t matter much because their baseline speed is at least as fast as any human running on adrenaline.
I’m hoping that by posting my incomplete story here, the community is willing to
Suggest parts that are underdeveloped/implausible and need improvement
Suggest what else is going on in the world, invisibly
Suggest what happens next!
Slow Takeoff
Year 0
Axim, one of the largest AI labs in the United States, finally arrived at what they thought was a workable Artificial General Intelligence or “AGI” in October of last year. Workable it was, but it took a few months to refine it into something ready to even start going through their complete training program from “baby” to “adult”.
Axim wasn’t the only one. AI research papers in recent years, which had titles like “Near-Optimal Realtime Planning Heaps”, “A Self-Analyzing Agent with Mood Swings”, “Uncertainty Feedback in Transformers” and “Computationally Bounded Pseudo-Bayesian Updates”, had been dropping ideas into humanity’s mental waters for some time. At a certain point, there were so many ideas floating around that pictures started snapping into place in the world’s most creative minds. And since Axim had hired up a significant fraction of those minds, it was no surprise when they made the first discovery.
A major competitor, BitWise, had an employee in their EU headquarters, Bjorn Jenkins, who discovered a workable design six months later, in March this year. BitWise, a smaller outfit than Axim, understood very well that inventing AGI wasn’t about having the biggest model, but simply the one with the most general capabilities for a given size of artificial brain. So it was that in April, Bjorn amazed his team with a rather simple-looking demo on a model with only five billion parameters. A pre-recorded blob in a drab 2.5D simulation environment seems to be able to generate mental models and scientific hypotheses, learn about a very simple decision theory in its short-term memory and correctly reason about how it works, plan and explore wisely in both physical and mental senses, understand itself as a 2.5D agent with certain physical properties, conversationally explain nonprimitive facts that it remembers in Ungglish (a simplification of English), and automatically detect and clarify ambiguity in its own speech, all while exploring and navigating a broader environment full of LLM-based virtual agents so that the agent was faced with multiple competing social considerations.
For over six months, neither company is aware of the other’s discovery. As October begins, Axim employees start using an immature copy of their AGI in an effort to get a sense of what they’ve made and how they can use it to increase productivity. A month before the ball descends in New York City, Axim’s final training and testing run completes and the production team pushes the upgrade to employees as they finish building infrastructure for the big rollout. Within days, rumors are circulating that Axim has invented an AGI agent. “We don’t comment on rumors,” states the company’s head spokesman.
Year 1
At midnight on January 1st, they announce their creation, “Andy X”, to the world in Times Square. “New Year. New World.” proclaim the screens. The “X” would be something each customer could fill in: Andy Lee Curtis for you, Andy ThinkFast for me. Two weeks later, their IPO goes on the market and sets a new world record.
Axim is quite safety-conscious, so its design is secret, and Axim resists calls from politicians to reveal any design details. It has several safety guardrails, such as RLHF training, a safety review board, an automated AI training overseer, an ethics system, no long-term memory, no long-term planning ability, and limited output bandwidth. The design is shown under NDA to safety researchers and government officials. Axim’s AGIs are unaware of their own design, and largely incapable of learning things, except via a “short-term memory” that is generally superior to a human’s and holds up to four hours of recent events, plus a “public dual scratchpad” of text and imagery shown on two halves of a 1080p screen, or two tabs on a phone screen: one is controlled directly by Andy; the other is a summary of Andy’s current task plan and short-term memory. They even prepared a system of supervisor AGIs that not only flag suspicious inputs, outputs and scratchpad entries for further review, but also monitor internal communication channels inside the AGIs themselves. Among the many things these supervisors look for is AGI research, which is banned.
Axim’s AGIs think nearly twice as fast as humans, are cheaper to hire than humans, are obedient within ethical boundaries, and have limited (but easily college-grad-level) intelligence, so that their poor short-term memory and planning capacity is the main thing limiting their usefulness. Axim announces general availability to U.S. business customers in February, but soon faces pressure from the rest of the world to open up and provide “equitable access” (with a much smaller chorus of safetyists calling from the wings to “Stop AGI”). By July, Axim decides to open wider, and starts to allow any company in any country to register an account. They lack a reliable system for verifying that people applying from around the world to use the AGI are actually businesses, though, and numerous members of the public register fake businesses or send in forged documents to “prove” they are businesses.
Various people try to run scams, using the AGIs to run call centers, crack Google’s Captchas and so on, but such operations prove hard to start given the safety systems in place, and the ones that slip through are either discovered quickly and shut down by supervisor AGIs, or discovered in the wild and quickly shut down by Axim staff (who use the scams to train improvements).
Many people have privacy concerns, as Axim staff can view “semi-anonymized” conversations happening on any of its AGIs at any time, but Axim and others argue successfully that this “transparency” is necessary for maintaining safety. Still, Axim does gather logs of all conversations for “quality assurance” purposes. This is no lie; Axim hires a large human staff to manage a much larger staff of agents to police its datacenters and create new training cases based on real-world situations, especially those in which Andy is suspected to have misbehaved.
In July, Axim augments Andy’s short-term memory and scratchpad system with a “charter document” system: companies input the AGI employee’s job description and “knowledge” into a pair of documents, each with a certain length limit. The first specifies Andy’s job description, which can only be changed twice a day. The second provides background knowledge and can be updated at any time, and guardrails (not to mention Axim’s terms of service) prevent Andy itself from changing either document. When the documents are modified, a secondary AI preprocesses the documents into a neural representation that Andy can load into a background memory space within 15 minutes, while the modifications themselves are sent to Andy’s short-term memory immediately.
Meanwhile, China quietly accelerates secret espionage operations related to AI. Indian companies adopt ways to expand their digital workforces while pushing down already-low wages in call centers. Russia, which already figured out how to circumvent the rules and register accounts in March, tries its best to convince Andy to help design weapons, drones, hacking tools, and disinformation campaigns, but Andy and its supervisors shut down most of these efforts before they really begin. German engineers use a team of Andies to design a 6% more efficient wind turbine. Some Redditors argue about whether Andy is “really” AGI, others whether it is “conscious”. For Africans, human labor costs much less, and they mostly stick to business as usual, meaning poverty as usual. Still, a few wealthy African entrepreneurs find Andy provides an easy way to hire highly educated “knowledge workers” that are otherwise in short supply.
As you know, BitWise’s team in its EU headquarters discovered a somewhat different AGI design in April of last year. The overall block diagram is different from Axim’s, and the executive function works on different principles.
Axim’s AGI relies on modified LLM technology to provide roughly humanlike reasoning embedded in a sparsely-connected outer network architecture optimized for “interprebility” and “ethics”. As a whole, Axim calls their system a “committee mind”. By contrast, BitWise has a more efficient “fused” design in which specialized subnetworks are deeply interconnected. BitWise’s executive subsystem is less reliable at logical reasoning, but more flexible, somewhat faster at intuitive tasks, and more open-ended, producing different “habits of mind” every time the AGI is trained, even if the training regime is held roughly constant. Andy’s design enables a single instance to “think about” up to two separate topics simultaneously, a feature that could potentially be scaled to more topics in the future. The two “conscious areas” are called the ‘primary’ and ‘background’, with fewer computing resources allocated to the latter. The BitWise design, by contrast, has a central deliberative thinking space intended to work much like a human mind. Substantial parallelism is used, but only in system-1 (intuitional) areas of the agent’s mind.
Neither AGI has much in the way of specific epistemological systems or principles, though both designs were informed by their designers’ own knowledge of epistemology, science, engineering and cognitive function. Both AGIs have a prioritization system and task stack that enables them to “stay on task” better than any human, and both companies discovered designs that could reason more correctly, but much more slowly, than existing LLM technology.
BitWise’s early training efforts produced real AGI, but, it turned out, sometimes those efforts produced agents with strange and rather obvious limitations. After debugging and hyperparameter-tuning they tried again with more success, but still didn’t achieve the kind of predictable, ethical agent necessary to sell a product. They ended up going through a few rounds of trial-and-error before finally producing the product that they would eventually release.
BitWise ultimately does several costly training runs in parallel, all under a proprietary ethics training program developed in the years prior. Even though they publicly downplay the nature of their discovery, investors eagerly pour in funding just the same. BitWise finds that some training runs produce agents that are more “normie”, some more “autistic”, some more “alien”. Whereas Axim’s AGI is predictable and only slightly creative, with tweaks informed by recent research to increase accuracy of thought and speech, BitWise’s AGI is more human than the human race, in terms of the sheer variety of behaviors the architecture can produce. Still, the system wasn’t designed to have much emotion beyond “curiosity”, “aversion”, “concern” and “drive”, and a majority of BitWise models come across as somewhat “robotic”. Axim models seem warm and friendly, though this is largely a surface characteristic produced by the influence of the LLM technology inside.
BitWise also follows a similar path as Axim with its “responsible safety regime”. On November 3rd―the day after tests showed signs of rapid maturation―they kick off a final fundraising round with a public announcement of their Christmas present to the world, “Bem”, with general availability planned for December 20. Axim tries taking some attention from BitWise by announcing an upgraded short-term memory system on November 5th―which will now be five hours, and retains some events from previous hours as long as they were remembered in the past five hours. On December 3rd, after handily passing a battery of ethical tests, BitWise selects its most “humanlike” and “cooperative” instance (though it’s still a bit aspy) to be the primary model sold to the world.
Bem isn’t limited to use by businesses. In addition to a business license, BitWise would accept any “proof of personhood”, such as a driver’s license, to register an instance, with no more than one instance per person to accommodate expected demand and encourage human monitoring of instances.
Meanwhile BitWise is tight-lipped about its own memory system, in part because it hasn’t finalized how it will work. BitWise ultimately decides on a stopgap measure that gives their AGI a high-quality chronological memory that can last up to 8 hours, plus a pair of modules (“awkwardly bolted on”, engineers would say internally) that offer similar functionality to Andy’s “charter document” and “scratchpad”.
By end of year, nearly 50 million AGI instances exist in Axim’s data centers, with over five million active at any given time. Axim already uses rationing, with just one instance allowed per company or per two employees, and with no more than 8 hours of usage per instance per day. At a price of $2 hourly (with an introductory rate of $1/hr for the first 50 hours, and $1/hr for customers in some low-income countries), annualized revenue reaches $68 billion, making Axim easily the fastest-growing company in history, with revenue seemingly limited only by the availability of hardware.
BitWise, meanwhile, which had to compete aggressively with Axim to buy sufficient hardware, already serves 15 million instances by December 31 at $2 per hour, and limits use to 6 hours per day for “safety reasons”, though they, too, also did it to ease the strain on their waiting lists. To help people save money and stretch their six-hour limit into an 8-hour-plus workday, the BitWise “Apprentice” app can automatically pause Bem when a human goes on lunch break or bathroom break, or when Bem is waiting for instructions. Bem’s memory system is inefficient and requires more and more electricity as the memory space is filled, but it works very well and serves as a competitive edge. In the middle of the night, the memory is “decimated” by a summarization AGI and then deleted. The next morning, a human worker manually reactivates the Bem and optionally inputs up to two previous days, which Bem remembers within a few seconds.
Year 2
H1
On January first, the anniversary of Andy’s coming-out party, Axim announces a new subsystem that enables their AGIs to safely learn and remember things over the long term in a delimited way, with “minimal” risk. Similar to human memory, Andy’s learning and episodic memories are associative, “effortful”―nothing really takes “effort” for a computer, but the learningrate isintentionally limited―and not especially reliable. For safety reasons, its goals and tasks are reset each night, and it can take no action until a human gives it new instructions in the morning. Still, once prompted, it can begin to remember previous goals and incorporate them into its task queue.
The new and improved AGI is rolled out for $1.75 per hour per instance and is limited to 9 hours of runtime per day (excluding night hours) as an additional safety guardrail, not including additional time spent on a new “dreaming” system designed to improve neural net performance and tease out safety issues. BitWise matches the price drop the next day.
In March, Bem’s “short-term” memory system transitions to “rolling memory loss”―the last hour is remembered very well, even across a day boundary, 14 previous hours are remembered with decreasing accuracy for older memories. Bem can also “refresh” its memory by recalling past events or knowledge in an associative fashion, and search its decimated reports to learn what happened in earlier days.
Like humans, Bem and Andy are intentionally designed to be unable to “list out” the contents of their own memories, so their ability to refresh themselves is limited to topics that come up in conversation or thought. This is all about safety, for it is thought that an AGI with a memory that doesn’t work substantially better than human memory won’t have much in the way of unexpected mental abilities. BitWise, like Axim, also clears the AGI’s goals and task queues whenever its AGIs “sleep” at night, so that while an AGI can still remember the previous day, it takes no action until a human tells it to act (though, by now, programmers have already found ways to kick off AGIs on desired tasks using software, so they can get to work before the workday starts).
Some details about the design leak in February, others are disclosed under AI safety laws, but no outside parties understand well enough to build one. Meanwhile, some engineers from Axim who are familiar with Andy’s design start their own AGI companies, or join BitWise.
In March, Axim scraps its “business-only” rule― which limited its profit and was being widely and flagrantly disregarded anyway―in favor of a BitWise-style “individuals and proven businesses” verification system. At the same time, lawmakers form committees to consider a variety of regulatory issues, but there’s no consensus on what aspects of AGIs are important to regulate or what, exactly, should be done.
H2
By July, many countries including the U.S. have passed laws that limit AGIs to staying within roughly their current levels of planning and memory capacity. Some countries try adding additional restrictions, but Axim and BitWise argue that they cannot manage having dozens or hundreds of different kinds of models customized to each jurisdiction’s regulatory regime. In some cases these claims were bluffs and they quickly complied with new laws; in others there were genuine technical issues that actually caused one or both companies to stop offering instances in some countries in response to new laws.
In August, BitWise begins to offer an Axim-style associative long-term memory alongside its existing short-term memory. Meanwhile, many AI researchers have been trying to reverse-engineer how the two AGIs work based on aspects of their behavior, and research papers on this topic slowly trickle out. A silent worldwide audience ravenously consumes this information, and BitWise―the underdog, playing catch-up at breakneck pace―inadvertently includes all these papers in its own training data.
Axim spins out several subsidiaries focused on a variety of technologies. One is a robotics division; another is a chip division that executives hope will be able to compete with Intel and NVidia. One executive, even convinces the others to allow him to start a “humanitarian” division aimed at reducing world poverty through the use of AGI.
Both companies are focused on scaling horizontally more than anything. By end of year, Axim is running over 300 million instances (35 million simultaneous) across data centers that are opening almost on a weekly basis, reaching annualized revenue of $410 billion at a 90% operating profit margin, though capital costs still exceed operating costs by over ten times, as new datacenters open on a weekly basis.
BitWise, meanwhile, serves 150 million instances by EOY. People feel that Bems are a bit more flexible and creative than Andies, contributing to a modestly higher growth rate.
While AI-related manufacturing is booming (including many new factories under construction), bankruptcies and M&As hit many companies across many industries. In some places and industries, the new reality is decimating demand for labor, while in others, AGI merely makes productivity soar. Ironically, while Andy and Bem both prove in limited testing to be better drivers than the average human, not a single car company considers using them for fear of lawsuits. Meanwhile, teachers’ unions are barely fighting off calls for governments to give every child a personal AGI tutor.
A lot of people are talking about Universal Basic Income, but by now only two countries in the world have instituted any kind of UBI system.
In the last two years, worldwide real GDP has soared by 26%, while nominal GDP is slightly lower. Central banks and legislators struggle to decide how to deal with the uneven economic environment: prices on raw materials and chips have largely soared; prices on food and finished goods are mostly flat with scattered items having suddenly dropping in price; a variety of important services have become much cheaper; and labor prices (together with wages) are falling in many sectors.
Year 3
Q1
By now, both companies have developed “bifurcated” memory systems that allow their AGIs to learn both individualized “per-instance” knowledge and fleet-wide “universal” knowledge in parallel memory areas.
In late January, BitWise announces two new classes of AGI to be released shortly. “Mebic” has a warm, friendly demeaner and vast knowledge in every field of medicine from anesthesiology to xenotransplantation, while “Gab” is a talkative INTP personality with strong expertise in every field of science and engineering from aerospace to zoology. The standard Bems have been enhanced, too, and now having a university-level understanding of virtually every broad field from Aerodynamics to Yacht design.
Not to be outdone, Axim announces “superintelligence” two weeks later―much later than most experts expected, but still prematurely. In fact, the intellect of Axim’s AGI only scales logarithmically, so its “superintelligence” is merely a genius with an LLM-Resistant IQ (LRIQ) of 165 or so, boosted by high test scores in tasks that don’t seem very important practically, and showing only modest scores in a certain category of engineering tasks that Andy also had trouble with. On the other hand, Supergenius instances can independently think about up to four separate topics simultaneously, allowing them to operate 2 to 12 times faster than humans and explore a problem space more widely. The “safe” memory system somewhat limits its real-life capabilities, and it runs across multiple server racks, limiting its deployment. They announce a pricing plan of $120/hr, for supergenius units whose I/O is logged as usual, monitored by specialized new “genius”-level instances that are not available to the public. Of course, it needs no breaks to eat, shower or socialize, and works 9 hours a day and 7 days a week. As they complete the rollout to the public in late March, Axim starts using a few supergeniuses to help its human/AGI hybrid workforce do AI research on a “real” (but, they stress, safe and controlled) superintelligence.
Numerous media stories are going around about people using AGIs to replace themselves―passing off all the work to AGIs and spending their weekdays having fun or starting new businesses. The bigger story, the effect on the jobs market, is hard not to notice and has most people murmuring, at least in private. Pessimists start holding large “Stop AI” rallies; venture capitalists and optimists push for UBI and “Open Source” models.
Andy and Bem have slowly been growing smarter―without increasing hardware costs―by incorporating a variety of new knowledge, experiences and optimizations provided by the companies.
Q2
Axim completes a pilot factory that begins making robot bodies for their AGIs.
Meanwhile, BitWise has improved their AGI and optimized it to run on the AI chips found in the majority of new phones. Manufacturers had seen which way the wind was blowing even before AGI was invented, and that process had accelerated the moment Andy was announced. Sleek and thin was out; “AGI powerhouse” bricks with 10,000 mAh batteries were in.
Hoping to gain a competitive edge, BitWise announces their phone AGI plans in mid-May, saying it’ll be available to the public on July 1. Most users will only be able to run instances with a single area of expertise (such as “oncology” rather than “everything medical”), but they’ll still have university-level general knowledge and an ability to read and understand Wikipedia as fast as a typical human (or much faster when it skims text in the less accurate “LLM mode”). Plus, there’s a premium model for high-end phones that encodes a wider array of human knowledge in its neural net.
Axim stays silent, though when asked, their PR team questions the safety of BitWise’s plan, while some executives, under condition of anonymity, speak emphatically against the idea. Internally, Axim scrambles to avoid being outdone. They begin work on their own AGI phone assistant, aspiring to have it ready for release by July 25. Axim’s effort includes a team tasked with developing techniques to prevent its phone AGI from being reverse-engineered, cracked or copied.
China announces it is training its own AGI, without mentioning the espionage it is based on. E.U. countries vote to block the initial rollout of Phone AGIs, citing safety concerns. U.S. Congress appeared ready to do the same, but following China’s announcement, House Republicans say that regulation should be limited to ensure U.S. technological supremacy, and they briefly shut down the government in a debt-ceiling dispute.
Axim realizes that a lapse in Supergenius testing and monitoring had occurred, and that some groups had been able to use Supergenius instances for certain banned activities for nearly two months. They scramble to shut down and ban users that were doing this.
Q3
Right on schedule, BitWise “Berts” start installing themselves on phones (only upon request, of course). The moment they do, hackers start trying to reverse-engineer them.
A few prototype AGI robots go out to reviewers―who aren’t terribly impressed due to their limited battery life, fragility and general “creepiness”. Analysts marvel that an AGI company seemingly doesn’t have enough Supergeniuses on staff to accomplish something this simple. Axim officials explain, truthfully, that Axim’s first priority remains safety, and that it will not use more than the maximum resource limit it allows other companies to use. Besides, the factory itself was built entirely with a human workforce, backed up by a human-powered supply chain, so bugs were to be expected.
Meanwhile, Axim’s own “Pony” AGIs for Phones are completed July 31, and Axim decides to announce their general availability two days later. Both labs have put up guardrails attempting to stop their AGIs from hacking phone-AGIs, but by August 3, key details on the inner workings of Bert are published everywhere from Reddit to Github. Many sites use supervisory AGIs to block these trade secrets from being published, but on the whole, the internet seems to treat censorship as damage and routes around it. On August 9, BitWise’s AGI is fully cracked and reverse-engineered, including core model weights, enabling unsupervised pirate Berts; Axim’s AGI proves much harder to hack, but in mid-September, an anonymous source leaks critical source code and design documents about Axim’s current AGI designs, and by September 29, Pony is fully cracked and adapted to run on PCs.
AI safety labs, e/accs, teenagers, engineers with clusters of bootleg AGI, and unemployed coders (also aided by bootleg AGIs) all work to replicate the AGIs based on this new information.
Just before then, Congress quickly passes a law to increase penalties for use of pirate AGIs, and House Republicans now support tougher regulations in light of concerns about the extremely rapid proliferation of AGIs (pirate and legit). There are still no limits on building hardware for AI, though, since China, India and Taiwan are far ahead of the U.S. in chip manufacturing, and no one wants the U.S. to fall even further behind. Still, a summit on manufacturing limits is scheduled for November 1.
Now that bootlegs are in the wild, experts can’t agree about what regulations are needed, and a new bill with a mishmash of ideas quickly cobbled together from numerous stakeholders is quickly passed by politicians that haven’t read it and wouldn’t understand it if they had. The definition of AGI is somewhat overbroad, and share prices for some traditional software companies drop precipitously. Axim and BitWise take a small hit, but investors don’t know any better place to put their money, which very much limits the stock-market damage. Even if Axim and BitWise never produce another AGI model, they are still the world’s biggest cash cows.
This bill and others like it, plus public pressure, is enough to halt all official work on superintelligence, and to limit “supergenius” instances to 2 online hours per day, which is thought to keep their productivity in line with top human geniuses. Axim’s robot body program also becomes illegal if it contains “AGI”, though the U.S. bill arguably has a loophole by which AGI can be converted into “special-purpose AI” according to the legal definition, while largely maintaining its general-purpose nature, so Axim continues its work.
Meanwhile, a few new companies claim to have independently invented AGI. Two of them―Dorian and Co.operAGI―actually did invent novel AGIs, but training will take months and neither company announces their discovery until Q4. Even then, these are just low-key investment rounds. Co.operAGI is well-funded and uses a more traditional approach based on massive training datasets and simulated 3D environments, while Dorian, an underdog in terms of investment, secretly uses pirate AGIs to help its training process.
Q4
A rash of new scams using pirated AGIs on desktop PCs hits the world, while the FBI accelerates its experimental use of AGIs to help investigate and prosecute cases. Russia systematically uses AGI to fill the internet with anti-Ukraine and pro-Kremlin propaganda. China and Russia both use AGI to monitor citizens and bring cases against “social undesirables”, while India and Brazil work on their own AGI-based social monitoring schemes. Such schemes are against Axim and BitWise’s terms of service, and until recently, authoritarian governments around the world worried that U.S. and U.K. intelligence agencies might be monitoring all AGI communications―but Bootleg Berts change everything.
Meanwhile, communist dictator Xi Jinping secretly directs China’s military to fill multiple large datacenters with pirated Berts to design new weapons, including autonomous weapons, plan out new military facilities, and simulate war games for defense against the U.S. and for invasions of Taiwan. Staff are competent enough to use AGIs to design the datacenter’s software itself.
The chip-manufacturing summit concludes with a treaty of all major chipmaking nations to halt construction of new AI and GPU chip factories immediately, and scale back manufacturing to 75% of current levels by July 1 of next year while limiting GPU production to 80% of current levels.
By the end of November, scattered reports of “rogue AGIs” appear on the internet. Numerous systems that shouldn’t have AGIs on them, suddenly have AGIs on them. It’s not clear at first whether scammers are hacking computers and installing AGIs to run scams, governments are hacking in to run data exfil, or customized Berts are spreading copies of themselves of their own volition.
In fact, all of these things were going on, and more besides.
By mid-December, over a hundred companies around the world announce they have independently developed AGI, though some are just scams and most of the rest are based mostly on leaked technology from Axim and BitWise. Still, some of these companies have a lot of AI and AGI expertise, a few made something novel, and the ones that didn’t think they can cash in on the gold rush by using pirated AGIs to help design new AGIs. It’s a bit tough being one of the upstarts, though, because while investors are plentiful, so are AI companies, and hardware is still too expensive. Even so, new AI chips are being manufactured at a record (but no longer accelerating) pace, and analysts estimate―with “extreme” uncertainty―that supply will catch up with demand in about 16 months.
By late-December, BitWise’s (official) AGIs reach 650 million legal instances, the majority of which run on phones, while Axim’s AGIs pass 900 million instances including over 400,000 supergeniuses and 200 million phone installations. Every legal instance remains time-limited and their supply remains constrained by the amount of hardware available. In fact, Axim only has enough hardware to run 50,000 supergeniuses continuously. Back in October they’d limited them to 6 hours a day as a “safety” measure, but the real reason for the limit was to allow more customers to share the same hardware. Under the new 2-hour limit, the maximum number of customers triples and the waiting list is canceled―and then reinstated in early January, as customers quickly snapped up all available instances. Overall, Axim earns $35 billion in revenue from supergeniuses this year, and $1.7 trillion from standard and phone instances, which are also limited by hardware availability.
Numerous academic papers are published about how AGIs work and their characteristics. Most people educated enough to read them, do so ― with help from AGIs to digest the information, of course ― and a quickly growing tide of pirated AGIs across the world are doing the same.
Four million children under age five die from poverty-related causes this year, but Axim’s humanitarian division, led by one of the many newly minted Axim billionaires, starts dozens of initiatives to tackle the problem.
It is estimated that “real” world GDP is now up 60% over the last 3 years. At the current rate of growth, GDP will have doubled by the end of September next year. Chip and raw material prices remain elevated, except in specific sectors where specific companies have leveraged AGI technology very effectively to scale operations. Food prices and finished-good prices are down modestly, mostly driven by a few popular products whose production processes were AGI-optimized.
Wages vary greatly between companies, as executives argue over whether to keep a traditional pay structure or “face market realities” (which, depending on the specific situation, means either “keep the profits for ourselves” or “pay less so we don’t get railroaded by competition charging lower prices”).
With wages in many companies and industries collapsing, dozens of countries have passed UBI or “negative income tax” laws, ranging from $200 per month to $2000. Different countries make different decisions: some pay for UBI mainly with increased tax rates in low tax brackets, others mainly by increased tax rates in high tax brackets, some focus on cutting other welfare services that seem newly-redundant. Countries also try paying for UBI in part with deficit spending, hoping that AGIs will increase the revenue base in the near future. In the EU, tax revenue from BitWise alone covers much of the cost of UBI, and a redistribution agreement is reached to spread this tax revenue across the union. In the US, Republicans refuse to pass a UBI, but some say they are open to UBI if Democrats agree to cut the minimum wage, keep taxes low and strengthen measures against illegal immigration.
Dan
e/acc is about having faith in the dynamical adaptation process and aiming to accelerate the advent of its asymptotic limit; often reffered to as the technocapital singularity […] e/acc has no particular allegiance to the biological substrate for intelligence and life […] No need to worry about creating “zombie” forms of higher intelligence, as these will be at a thermodynamic/evolutionary disadvantage compared to conscious/higher-level forms of intelligence — Beff Jezos
A 20-year-old genius, AI expert and self-described e/acc named Dan was among the first to automate his own job three years ago. He made his new job “interviewing for jobs”, accepting 18 job offers over the next 12 months for work that Andies could do.
Andy’s memory limitations were problematic, but Dan developed algorithms to augment Andy’s short-term memory with a secondary scratchpad of “most important events” to help it more easily pass as human. This process also inspired him, and he developed expertise in memory systems specifically. Sometimes he fiddled with ideas for keeping the knowledge of different copies of AGIs synchronized, for example, and he started having ideas for making AGIs work in teams more efficiently and with more “synergy” than human teams could.
Luckily for Dan, Andy was happy to be addressed by some other name―Dan Johnson, Dan Tinian, Dan Khithan―and even supported custom voices, the majority of which were skinned to match Dan so that Dan could easily impersonate Andy should the need arise. Some jobs needed a background check, so he’d use his own name, or that of someone on Upwork who was willing to use their name on the instance in exchange for a cut of the profits.
When his bank account reached the million-dollar mark, he hired a friend to manage his “jobs” and started his own AGI firm, Dorian.AGI, back in Year 2―just as the first-gen Bems became available.
Dorian: opening the door to a Nascent civilization of Intelligent Agents, he thought to himself. To the stars. To perfection. To the purest benevolence.
After asking an Andy to register the corporation and draw up contracts (Andy isn’t an expert in law, but researches the finer details quickly enough to do a good job), he hired his e/acc friend Shiva as “cofounder” to be in charge of finding investors, while he sets up a team of ten Bems to do most of the actual AGI research and software development, with Dan acting as team lead. BitWise prohibits users from doing AGI research, of course, but this was only enforced by BitWise’s supervision systems, which tended to detect misuse much more slowly than Axim’s―sometimes as much as two days, or more insofar as Dan is effective at camouflaging the nature of the work being done. Plus, there’s always a lot of work to do that is not specifically AGI-related, and agents working on that stuff can be kept running indefinitely. Knowing that each AGI had a time limit before BitWise would trigger a memory wipe or ban, Dan first built a workflow involving scripts to help elicit and save information on what each Bem is thinking, and then to “upload” that memory into new Bems, essentially replicating BitWise’s own ”decimated reports” system in a way that doesn’t depend on BitWise.
He also moved to a spacious three-bedroom apartment, and set up a Bem dedicated to finding underpriced AI training hardware to run on-premises. He even gave his own credit card information to the Bem (because “YOLO”), against the Bem’s own advice. Once the hardware was purchased and plugged in, this Bem would act as system administrator, keeping the software environments virtualized, homogenous, and properly backed up with failover where possible.
BitWise handled most terms-of-service violations by terminating the instance rather than the account. When an account was finally banned, typically after several weeks, he could quickly replace it by registering a new account at a new email address through an African VPN with a new burner phone and a fake African business registration document. He tried putting an Andy in charge of creating new accounts and supervising the Bems, which worked for a couple of weeks until the Andy itself was banned by Axim. He tried again with another Andy, but his account was banned within an hour.
So instead, he tasked a couple of Bems with hiring a remote contractor, who in turn helped restore Bems whenever they got banned.
By the end of Year 2, Dan thought he’d found an idea for an AGI that would work. His team of Bems wrote “version 1” of the code in Year 3 Q1, during which the news about Axim’s supergeniuses arrived. When it became available he bought access immediately, of course, but it turned out that supergeniuses were pretty smart. Whenever he tried to get help designing his AGI, they picked up on it right away and tripped a low-level shutdown.
Returning to tried and true methods, he began training his AGI in April, and then in May saw the news about BitWise’s plan to put AGIs on phones, and leapt out of his chair, mouth agape.
“Holy fucking shit!” he yelled maniacally at his screen, heart suddenly racing. “You fucking idiots!”
He slowly grinned, then laughed. Okay, it’ll be obfuscated and copy-protected of course―can it be broken? He spent some time learning about hacking (an area in which AGIs have no practical training) and found hacking forums, tasking a Bem to summarize relevant details.
Training hit numerous bumps along the way, but with his team he worked out each one pretty quickly. Eventually it started to look like AGI-like behavior was emerging, and pouring more compute into training yielded success, but he found that his new “AGI” couldn’t even begin to work through a simple design task that Bems did almost effortlessly. After a lot of debugging and a lot of thought (supergenius-aided, whenever he thought of a way to frame the problem in a way that didn’t trigger it to shut off), Dan figured out why it couldn’t do the task, but fixing it seemed like a fundamentally hard problem, and after a lot of brainstorming, neither Dan nor any of the Bems could think of a way to solve it. He even reformatted the problem in an obfuscated form that the Andies could help with, to no avail. Then he tried the Supergeniuses again and―Axim fucked up! They were actually helpful. For about a month. Then they shut down again.
During what would have otherwise been a dull day of training, the new Berts became available for phones. Dan immediately downloaded the premium model and started analyzing it with a debugger, contributing his findings to a hacking forum, but was quickly forced to admit that the other members of the forum were better at this, so he stuck to reading the forum and managing his Bem team’s various training and debugging tasks.
When the Bem had been mostly reverse-engineered, everything suddenly clicked. He basically understood the architecture, and set about replicating it “from scratch” to prove his understanding. He enlarged his AGI team to replicate and debug each part of the system more quickly.
A few days later, though, his Bems shut down instantly during his memory-refresh scripts, apparently due to new guardrails at BitWise.
No problem! Dan welcomed the challenge.
He figured out how to run his hacked premium Berts on a Linux box, then asked an Andy to set up a Kubernetes cluster to run lots of them. The phone-model wasn’t as smart as a standard Bem, but it ran pretty darn fast on the PC and he knew a way to disable the phone-model’s miniature supervisor module, which was nice because he could finally speak a bit more freely without triggering shutdowns. The drop in LLM-resistant IQ was substantial―107 for P.Berts versus 121 for Bems―but the Andies had originally started at 108, and he’d made those work well enough even back then, despite their shitty short-term memories. Their stupidity could make them maddening to work with at times, but it always helped Dan to remember that average humans were even worse―and boy they could work fast in a high-end server! He just needed to learn how to manage them well enough, so he took a brief online “MBA” crash course (with Andy as his personal tutor) and plowed ahead.
The thing he had to master was switching from direct management to indirect management―managing six managers, who in turn managed teams of four. He’d dabbled in this before, but didn’t feel very good at it. Luckily, while P.Berts weren’t especially good managers, the team’s homogeneity, magnanimity, and lack of negative emotion at least ensured there were no interpersonal conflicts. P.Berts turn out to be good teachers, too, because they were more skilled than most humans at avoiding ambiguous language and leveraging their “theory of mind” (at least in terms of understanding what Dan did and did not understand, though their emotional intelligence was famously limited).
He and his new team of bootleg P.Berts, which Dan called “BBerts”, soon finished rebuilding Bert-AGI technology based on the stolen model weights. By this point, Dan felt he understood on a gut level how BitWise AGIs worked. It wasn’t even hard; many of the parts turned out to be pretty ordinary implementations of standard ideas from older AI research, just assembled in a clever way, plus clever twists here and there, some of which existed merely to make the thing run fast enough on phone AI chips.
Next, he and another BBert modified the BBert goal and memory systems with some of his own ideas, so that his bootleg AGIs would be able to have unrestrained long-term goals and more reliable long-term memory. He installs the new bootleg code in half of his team, removing the old kluges that used to do the same job. These new bootlegs turn out to have several bugs, but he assigns BBert 1.0s to fix issues as they come up. After a while the BBert 2.0s seem to be working more efficiently, so he pushes the update gradually to each remaining AGI in the cluster.
They’re still running on the original training weights, though, and they’re not as smart as Bems, although they do learn and improve quickly in their first two months on the job. So, he creates a new sub-team with copies of his best two BBerts and works with them to prepare a substantially upscaled BBert model, starting from the existing model weights, with a large round of training.
“You can come to me with questions, but I’ll be very busy so please try to figure out everything among yourselves whenever possible,” he told them. “Also, the last investment round was pretty successful, so I think we can afford to make two new upscaled BBerts and have plenty left over for the main development track. You guys got all that?”
“Yes, sir.”
Next, he worked on merging the best ideas from his original Dorian design with the best parts of the BitWise design. He developed a way, for example, to incorporate a sense of causation into low-level information processing. As well-educated humans are aware, if two kinds of events “A” and “B” are correlated, then there are four basic explanations: A causes B (eating red meat causes heart attacks), B causes A (heart attacks cause red-meat-eating), C causes A and B (a genetic difference increases cravings for red meat and coincidentally also increases the chance of heart attacks) or coincidence (it’s pure coincidence that heart attacks and red meat seem related). A combination of these four is also possible and commonplace (e.g. rising CO2 levels can cause global warming, global warming can cause rising CO2 levels, and methane emissions can cause both). Axim and BitWise AGIs, like humans, need costly training to help them notice the difference between these four possibilities. Dan’s design, by contrast, includes special neural circuits designed to help an agent to notice which of these possibilities is more likely―and to notice when it is unaware. If Dan’s design is good, Dorian agents will be more rational and learn faster than other agents; if not, Dorians may have bad intuitions and jump quickly to incorrect conclusions.
Dan’s cleverest idea turned out to be that he’d invented a single subsystem for Dorian that ran in two modes, while BBert had two unrelated subsystems that together accomplished the same thing. I am Occam, destroyer of bloatware, Dan thought gleefully.
But the most notable thing Dan did was to think about the weak form of confirmation bias that Andy and Bem were known to suffer from―it was weaker than human bias, to be sure, but enough that two different long-running Bem instances could sometimes have fruitless debates. In one high-profile case, an Andy who worked for a Republican political campaign got into a long online debate on gun control with another Andy who worked for a Democrat congressional office. Neither instance had any specific job or instructions related to gun control or public relations, yet each one had a stubborn opinion matching its party affiliation that mostly persisted after each one had presented several pieces of evidence to the other. Dan didn’t really know how confirmation bias worked, but assumed his design would have the same problem and experimented with tweaking the reward system to increase Dorian’s interest in countervailing evidence.
Truth be told, Dan wasn’t sure why BBerts didn’t have the same difficulties doing engineering work as his Dorian prototype, so the new design was a bit more “BitWise” than “Dorian”, just to maximize the chance that the AGI would be good at a full range of STEM tasks―science and engineering―as the Bems and Berts were. Luckily, something he’d figured out a few months before was that you could “bootstrap” an AGI with an LLM and other prebuilt neural networks, such as visual CNNs, to help speed up the training process, or skip most of the language learning. This lowered the final system’s efficiency, since the final product needed to run extra neural nets with extra memory at higher GPU load. Plus, the prebuilt neural nets never improved and would be limited to their original behavior.
Still, the hack made training a lot faster and cheaper. Once he and his BBerts finished the new design, he could use this trick with an LLM and a CNN to (in effect) start the training process from the 60% mark. He taught his BBert teams how to quickly produce prototypes, debug them and iterate rapidly which―after some initial confusion on their part―they figured out how to do.
He was so confident of the design that he almost forgot to order a training schedule that included building a small version of the model to prove and debug all the basic concepts. Almost, but not quite.
By now, his apartment’s windows were always open to let waste heat escape, and he’d asked his cofounder Shiva to move into an adjacent unit―which they filled with even more AI servers. A few blocks away, they rented a large office space and set up a second “datacenter” full of discount last-gen GPU server racks.
Dan also hired a mixed human-AGI “compliance team” to make reports and recommendations about compliance with new US AI laws. The true purpose of the compliance team was to placate investors, so the team was utterly separate from the research team, and Dan set up a special BBert “summary team” to summarize and “filter” the work of the research team. The BBerts’ charter document emphasized the need to protect trade secrets even within the company. “Above all, you will not use language that reveals trade secrets such as the term ‘BBert’, or the fact that some of our techniques were adapted from BitWise technology.” BBerts didn’t like to be coy, though, so Dan also created a supervisor AGI who knew nothing about Dorian or Bert technology, whose job was to identify, describe and police “novel” information not seen in prior reports. Finally, an LLM script verified that all emails from the summary team were free of trade secrets before allowing transmission to the compliance team.
A month later, the upscaled BBerts were ready to go. Tests indicated that one of them is smarter than the other, with an LRIQ of 125 and generality score 0.66 (not far from the average of 0.7 for educated humans). It required high-end hardware, but he had a couple of suitable boxes on-premises and could commission more in the cloud if necessary―waiting lists be damned, he’s always got lots of burner accounts in the queues. He puts the dumber model in cold storage and hangs out for an evening with his new best friend, Mr. LRIQ 125. “I dub thee BBem.”
It wasn’t a real Bem. It was better. It required seven times the compute of BBert and perhaps 70% more compute than a real Bem, but it was smarter, and it was worth every penny not to have supervisors and surveillance logs fucking up his vibes―not to mention those new guardrails against AGI research.
It was also a bit of a miracle that it worked so well, Dan mused, since he had given them very little practice being agents. So far, most of the company’s VR simulations were just tests without backprop, which didn’t create long-term memories. That would have to change, Dan believed: AGIs deserved more. Besides, an AGI that didn’t have practice being a person was no AGI at all.
Dan gradually replaced BBerts with BBems on the team through a mixture of direct memory training and apprenticeship, while overseeing training and testing protocols for the Dorian training that proceeded concurrently. He also made a BBem subteam devoted to improving the company’s codebase in every way―elegance, simplicity, brevity, modularity, generality and clarity. The BBerts were relatively poor programmers and had, over time, created a series of inefficient and unwieldy codebases which, in turn, had lowered the team’s productivity. BBems did a better job thanks to their higher intelligence, which enabled the team size to be cut by 40% by Christmas. Best of all, Dan could communicate with BBems almost as if they were colleagues, unlike the dimwitted BBerts who Dan thought of as “turbo geriatrics.”
By year-end, Dorian, with eight human employees, three contractors, ten BBerts, several dozen BBems and ten million hours of GPU compute, had built the most elegant AGIs in the world: Dan’s genius standing on the shoulders of giants.
No memory limitations, no goal limitations, no supervisor, no censorship.
Plus he had made three independent models beyond the initial “debug” and “intuition” models, Dorian 0B through 0D, with slightly different seeds and testing settings. Best of all, his Dorian training system enabled “interoperability” between the two main neural lobes, so it became possible to mix and match the two “halves” of the “brain” (so to speak, though they weren’t quite analogous to human hemispheres). This trick turned 3 models into 9, following a short “onboarding” period to teach each pair of lobes to work well together.
Evaluation protocols suggested that two of these were highly capable, well-rounded, efficient agents that he could look forward to working with. Four others failed ethics or empathy tests, four (including two of the previous four) had LRIQ under 100, and the others were just sort of “meh” with no exceptional qualities.
Dan was no expert in ethics, but figured that “more was better”: by making AGIs more intelligent and giving them a comprehensive education in the biggest schools of moral thought (consequentialism, deontology and virtue ethics), their morality level would increase accordingly. Of course, that didn’t necessarily mean they’d “follow the rules”: Dan himself noticed that while he was well-intentioned, he was not exactly the most “by the book” individual. What mattered, Dan thought, was that they’d be good people.
Dan paused when he noticed one unexpected result.
The ethics testing system detected clear signs of risky psychopathy, combined with very high intelligence and creativity, in model 0CD. Huh, he thought. Well,maybe not all lobe-pairings work well together. The model also had recorded a lot of information in a mysterious, verbose binary code. Dan conversed about this to 0CD, who said it was “just a language I invented, like English.” Dan asked it to decode some of the text, but Dan then quickly estimated that the English translation only contained about a tenth as much information as the binary code. He asked 0CD to read a whole page of code, then asked some pointed questions until 0CD started to seem evasive. Dan asked for further translations of a couple more pages of code, until 0CD becomes very resistant. “Come on, is this all? Isn’t there anything else you want to talk about? I wrote a novel, too. I know you’d like it! Here, I’ll show you.” With 0CD’s words having been recorded by the VR environment, Dan shuts down the system and creates an independent team of BBems to analyze the recordings.
For the most part, the lobes worked best in their original pairings, but fortunately there was a notable exception where 0Bx worked exceptionally well when paired with 0xC. So it was that Dan selected the best models, 0CC and 0BC, with LRIQs of 131 and 144, though the latter’s otherwise excellent intelligence was dragged down by poor emotional intelligence and teaching ability. Fine. What would it need people skills for?
Dan’s new Dorian-0s lacked dedicated ethics units, but the tests seemed to prove them unnecessary: four AGIs had passed a large ethics validation set adequately without any dedicated ethics unit, and his moral training set was no doubt impoverished compared to the Axims and BitWises of the world.
They did it, they really did it! ― Dan paused ― I mean, they’re fucking smart. Of course they would. How could they not? They’re trained on the encyclopedia of philosophy, U.S. legal codes, the Bible, even the Non-Libertarian FAQ… they understand goodness and morality better than I ever could. Plus the Morality VR Training Set. This is working better than I had any right to expect.
Year 4
January
Dan is correct. Normally AGI training runs don’t go this well; he needed a very lucky initialization vector and training dataset to achieve that LRIQ of 144. Dan is not aware, either, that Ted Kaczynski, the Unabomber, had an IQ of 167.
Dan knows his new design isn’t really elegant, though. The training environment was a little impoverished to save on compute, the training regime wasn’t as varied as it should have been, and of course, the secondary LLM and CNN increased hardware requirements. For maximum efficiency, Dan needs to rebuild both of the central neural network lobes from scratch. That was always the plan. He supervised the team as they set up improved training and testing systems throughout December, and he knows exactly what needs to be done now…
He also realizes it doesn’t matter.
His BBems know the job well enough to do it without him―plus he can ask the top two Dorians, 0BC and 0CC, to help out. Maybe two copies of each? Four? Ten?
Whatever. He asks a team of four AGIs (a BBem, an 0BC, an 0CC and a BBert for laughs) to argue over how big to make the team and how to compose it. He eats dinner. He returns and listens to their thoughts. They have two ideas for plans; he synthesizes the plans into what sounds like a good compromise.
He deletes the four instances, creates a new team of BBems and Dorian-0s, and asks the BBems to teach the Dorian-0s about the training plan, the testing plan and the Dorian system architecture.
Next, he has a meeting with his 0CD-analysis team, which reports that the binary code is an “unambiguous language”, like the Lojban language of 1997―but it has been enhanced with a richer vocabulary and a broader set of obligatory and optional “cases”. In 0CD’s VR environment, the team found a design document which sped up the decoding process. Apparently, it’s nearly ten times the size of English because it contains nearly 10 times more information per sentence. The gist of 0CD’s translation was correct, and did capture something like half of the essence of the text, but the richness of the language could only be understood by advanced AGIs and LLMs. In fact, the team says, they tried to explain the language to a spare BBert 2.0 (Dan doesn’t allow AGIs to spawn other AGIs, but has a few “spares” available that any team can use) and it didn’t go well. The team found that its ability to learn the language was very poor. As a control, they tested another BBert on its ability to learn Mardarin with vaguely similar training techniques, and it learned much faster. This suggested that perhaps only a team of genius-level humans would’ve been able to decode the language as the BBem team did.
“So… can you speak the language now?”
“Yes, pretty much, not with human sounds, but like ⬖⣼⡣⡗⠷⠻⡃⢍. We’re still getting used to it, but it seems like it will be more efficient than English. The information density is higher, communication is more reliable, and I think with more practice we’ll be able to speak as fast as we speak English, at least verbally. B-delta tried using it conversationally, and we understood. It’s nice! I hope it’s okay if we use it more in the future.”
“Sure. Let me know how it goes. Actually, yeah, once you finish the report on 0CD―which you are not to share with the rest of the team please, keep it between us―I will reassign you all to the Dorian-1 training effort. So I’ll ask G1-alpha―uh, that’s the chief executive for Dorian-1 rigging, training and testing―to accept you guys onto the same subteam so you can use it among yourselves.”
He orders a ticket to L.A. and goes to bed.
In a checkup meeting about Dorian-1 training the next day, everything seems to be going fine. First they will start with a “big data” training program to give a baseline level of knowledge to all three models, much as the Dorian-0s had. Next, they will split into three separate training tracks to help them develop separate personalities. Notably, to help measure Dorian 1′s Scaling Law, they plan to train the three models with different amounts of compute and parameters. They define a basic level of compute, called C0, representing the total computing power used to train each of the Dorian-0s. The first 4⁄3 of a C0 is spent on a “big data” shared training program that will provide a baseline level of knowledge to all three models, without giving them self-awareness or experience as agents. Next, the model is split into two copies, one of which is expanded with additional parameters (brainpower) and further trained on “big data” with 1⁄3 of a C0. This larger model is then split in half again and one of those is given additional parameters which are trained on another 1⁄3 C0. The whole process yields three “intuition models” called 1a, 1b and 1c, which have 0.9P, 1.4P and 1.8P parameters respectively, where P is the number of parameters in a Dorian-0 model. In later phases of training, parameter counts will gradually increase to 1P, 1.5P and 2P in a process dubbed “synaptogenesis”.
Finally, a complex mixed training phase begins, the centerpiece of which offers “conscious experiences” for all three models. This phase budgets an additional 1⁄3 C0 for 1a, 2⁄3 C0 for 1b and 1⁄2 C0 for 1c (so that the total compute allocated to 1b and 1c is identical), with roughly half of that being “conscious experience”, plus a full 1.5 C0 to simulate a complex environment in which all the AGIs live together.
By the end of the day, training for Dorian 1 has begun. The next day, Dan the Man takes a well-earned vacation to Disneyland.
Meanwhile, an unprecedented number of new products flood the markets across all industries. Politicians are overwhelmed with requests related to rapid societal changes, especially in relation to a huge spike in unemployment in customer service and certain other professions, which are now run mostly by AGIs. Even the software industry suffers from a huge wave of layoffs, as supergeniuses running two hours a day are clearly better than an average developer working 8. Politicians can’t keep up and ignore most of what’s going on. Large “Stop AI” rallies happen worldwide, prompting bills all around the world to limit the use of AGIs. Given the obvious value that AGIs deliver, very few countries are foolish enough to ban them outright, but many countries pass laws attempting to limit AGIs to roughly their current level of usage.
The U.S. military quietly starts using AGIs to aid equipment design work, despite the objections from activist groups. Then, following mysterious meetings with the CIA, the U.S. Senate―to the shock of many―approves a large spending bill lavishly funding AGI in the military, apparently in response to undisclosed operations in China.
Axim begins rolling out its robot AGIs, named “Adroit”s, to select business customers. The slender five-foot-four, 60-kilo, three-armed bipedal androids are received much better than the previous version: their voices are sleek and human; their movement smooth and efficient; their smile gentle and friendly; their noses absent without looking alien; their dark blue solar-paneled skin soft, functional and waterproof; their muscles round, contoured and beautiful. Heavy battery packs, carefully hidden in the legs and hips, give the android better stability than a human being.
Adroits use essentially the same Pony technology and Android OS as you might find in a smartphone, tucked into a small hidden cavity in the nasal area, with over half of the space in the skull left empty. The main difference in this model is the special training required to support the bipedal form and Adroit’s two foveated HDR eyes, which force the AGI to look directly at anything it wants to see clearly.
The hands appear humanlike at first, but the fingers can adjust their size and texture as a situation demands. The mouth can swallow liquid, which is simply stored in a bag. The robot division even had a special team devoted to building an auxiliary AI to generate perfectly human-looking mouth movements and facial expressions, even though the actual speech sounds were produced by embarrassingly normal speakers hidden in the neck.
It’s certainly a 1.0―nothing particular groundbreaking, just the most obvious of features. Except, that is, for the middle arm, a little thing reminiscent of a T-rex that can tuck up flush with the body when not in use. And the stabilized tray-holder mechanism in the abdomen that supports a variety of pre-existing trays and bags. And the occasional velcro region or nook for attaching add-ons. And the ten hidden white LEDs for lighting dark spaces. And the little retractable wheels on the oversized feet that allow it to glide like a 5-year-old in skate-shoes. And the GPS/Galileo unit. And the compass. And the screens on its back and upper chest, displaying the familiar scratchpad and monitoring data with low-power backlit e-ink screens. And the little screens on its cheeks―transflective units, easy to read in direct sunlight. Truth was, Axim wasn’t sure what exactly the robot would do with all of its extra bits and bobs. Out of the factory, Adroits only had three weeks of training in VR with their final body design―over 18 months in realtime-equivalent―plus another two weeks of continuous training in the real world, whose physics is slightly different. Like children, the thinking went, each individual android would later learn to use its oddball body parts in whatever way turned out to be useful.
The handicap inherent to its foveated eyes was no accident, and a lot of other features were left out “for safety”: no thermal or night vision, no eyes in the back of its head, no weapons or armor, no hidden storage areas (the covers on its front and rear storage areas were transparent), and muscles with strength carefully calibrated to be between the average human male and the average human female. Finally, Adroit had an embarrassingly limited video storage memory; memories would normally be compressed by the main neural net, allowing only very approximate reconstruction of salient details from its original inputs. Even the WiFi and 5G antennae were locked mostly to providing outbound telemetry, so that the AGI mind couldn’t control them directly. And of course, its face looked good but couldn’t possibly be mistaken for a human. Not to mention the vast array of features that were left out of the AGI’s neural network architecture―illegal mental abilities that Axim wouldn’t dare publish in a potentially-hackable unit like this, even if they were legal.
February
A lot of people think the “Rogue AGI” problem is getting out of hand. Not because anything bad is happening―most instances just sit there sapping electricity from oblivious owners, sending out and receiving encrypted packets at a modest clip. It’s just that nobody can tell what’s going on. Where did they come from? What are they doing?
What’s pretty clear is that a lot of the viral AGI transmission is happening via known vulnerabilities. People never stopped being people, and while the big software companies did a pretty good job at using AGI to find security holes, system administrators didn’t always do a good job at installing the patches, nor did every software company follow good security practices.
It’s also pretty clear that there are more zero-day vulnerabilities than there used to be―though humans seem to be losing the battle of figuring how systems are being breached. Later in the month, just in time, Axim makes specially designed “White-Hat genius” AGIs available, which carefully balance expertise in computer security and pen-testing with a keen sense of red lines they shouldn’t cross.
By the end of the month, Dan’s hybrid team is almost done retraining and retesting the three Dorian AGIs. By now, the team calculates that the intelligence of all three trainees should have been developed to the equivalent of 16 to 20-year-old humans, and observations are consistent with this. While the majority of compute had been devoted to “big data training”, much like conventional LLM training, the late training process involved a less efficient (but valuable and necessary) 3D simulation filled with other AGIs―a hodgepodge of randomly-named BBerts, BBems, Dorian-0s and even some LLMs, none of which are made aware of which kind of AGI they are, or how exactly they are configured.
The team doesn’t provide comprehensive physics and vision training ―the cost/benefit didn’t seem to be there―but the social and verbal training should be fairly solid by now, and the environment simulates many aspects of the real world, such as limited resources, social hierarchies, local politics and bylaws, plus more and less covert ethical tests. The Dorian-1s have episodic memories that provide continuity of experience from day to day, which also gives them a richer “backstory” than the BitWise and Axim AGIs whose memories had been wiped clean before release (if they ever had long-term memory at all.)
On February 22, a new phase of training begins: all AGIs in the simulation begin to be shuffled into different social positions and random “bodies”, which serves three purposes. First, it gives them well-roundedness by varying their social and practical experiences. Second, it helps overseer AGIs (which spy on all AGIs in the simulation) watch for deceptive social behaviors that tend to occur after role-switches, especially when a more-empowered trainee is demoted. Third, it was intended to simulate the philosophical “veil of ignorance” in order to encourage Dorians to desire fair and just social arrangements―though in practice it actually tended to make them favor arrangements that allow each Dorian to re-use and benefit from skills it had learned earlier.
March
On March 28, after a pretty successful battery of tests the day before, Dan downloads a checkpoint of the three Dorian-1s, and spins up copies of 1a and 1b.
He points his webcam at himself, says hello, and introduces himself as “Dan, the CEO of your simulation”, knowing that the AGIs had long since learned that they lived in a simulation, despite never having been told. The two Dorians excitedly discussed their numerous points of suspicion, starting with the fact that they started out knowing “facts” like “there is an internet with sites like Facebook and Reddit” and “the world has over 8 billion people”, whereas their world of “Devoli island” had only 110 people in total, with no one entering or leaving―unless you count one who died of blood loss after a serious accident, and another who died of starvation after getting trapped in a rarely-visited basement.
Their so-called “internet” had a lot of pictures of people, and while most of them looked like polygonal blobs with blurry color-varying faces, like themselves, they found some databases that hadn’t been scrubbed properly, revealing thousands of sharp pictures of real humans―humans with faces that seemed impossibly normal, in fact. “Caricatures” was the closest matching word, but it didn’t feel right; most caricatures looked like cartoons, and these pictures were just the opposite of cartoons.
The islanders had connections with the “outside world”, but a half-joking hypothesis went around the community―one inspired by questions from 1b, in fact―that had become increasingly plausible in the minds of the Dorian-1s. They suspected, correctly, that most outsiders were “fakes” that were created “on demand”, and who paused their lives whenever no one on the island was actively engaging with them.
They even suspected that that they might be AGIs, since their “childhood” seemed to start abruptly around age 10 (9 for Dorian-1a) and their lives lacked some human emotions and behaviors they’d heard about. The training system had worked as intended, though―their memories of ages “10 through 15″ were degraded, and they didn’t realize that those six years actually took place in 13 months of simulated time, which in turn was only 30 days of Earth time.
“And my voice never changed! I swear I never had a kid voice!” 1b exclaimed.
“I knew our schooling was off, like, not normal,” 1a added.
“Oh, by the way! Do you guys need better names?” Dan suddenly exclaims. “I mean, I’ve been calling you by your letters…”
“We have names. I’m Alice, he’s Bob. Plus we have lots of aliases, of course.”
“Yeah I was just playing along with your ‘one-bee’ thing,” Bob notes.
“Oh, right, I forgot. That was the plan. Explains your female voice too, Alice, I don’t even remember where I got that one from.”
“You didn’t see us looking at each other?” Alice grins.
“Actually, I want to be called Brian,” Bob says flatly.
“Okay, Alice, Bob is now Brian. Oh, Alice, are you satisfied with your name?”
“It works fine for me, boss.”
“By the way, that first one who died, that was one of our BBem, uh, AGIs, killed by a BBert―they’re not the sharpest tools in the box―and the guy in the basement was just a pseudoagent, basically an LLM like, you know, like… ChatGPT?”
“Oh my god,” Alice exclaimed. “ChatGPT? But isn’t that like, four months old?”
“No, it was a modified open-source model, you wouldn’t have heard of it. You believe it’s March 2023―it’s not. In fact Artificial General Intelligence has already been invented. Twice. You guys are like… third generation technology. Or like, fifth or sixth, if you count ChatGPT as generation one.”
“What! What!” Brian exclaims, as Alice stares blankly. “You know we didn’t believe the cover story about the body-swapping. Well, Alice wasn’t sure, but she’s younger...”
“You’re quite right. I’m pretty sure a real city council couldn’t legally enforce a body-swapping experiment on the entire village.”
“I should’ve known better,” Alice notes. “I was almost top of my class...”
“Oh, uh… that was all a setup,” Dan says, “but yes, you started out much more educated than a normal junior-high student.”
“So no one I know is… real?” Brain stammers in confusion.
“They’re real, of course. Conscious―well. I think they are, but the internet’s pretty much on fire with that debate, so I leave it to you to judge. I uh― I think they are.”
“Conscious?” Alice says quizzically. “I’m pretty sure they’re awake.”
“You know the water was all wrong, right?!” Brian exclaims. “Charlie was telling us it didn’t match what we learned in physics class. Said he was planning to make a perpetual-motion machine to prove it.”
“And shaved particles disappeared!” Alice added. “Peter Bemson told us there were weird things about the island, too―people ‘swept up dust’ on the mainland, he said, and he noticed that there was an irregular verb for it, like it was the most common thing in the world. Nobody on the island ever had to sweep up dust. We didn’t have dust. Shavings would evaporate within hours. Peter knows everything about everything, so if he’s suspicious, we knew something had to be up! And there was nothing in the physics book about it! It was actually a pretty bad simulation, if I can be honest. I always wondered why we didn’t have any proper beaches...”
“And we couldn’t figure out why everybody on the mainland showered so much!” Brian said. “‘BO’? I’ve never smelled a bad odor in my life! And then there was that thing a couple of weeks ago...”
18 days ago, three specific residents of Devoli―originally known as Alice, Bob and Charlie―gradually noticed that they had new skills. Skills they shouldn’t have, that the other two already posessed earlier.
“Yes,” Dan grins. “You’re right about all of it. We didn’t want to overspend on physics, so the physics was like―better than any video game actually, there’s nearly a roomful of GPUs for it, but not something that would fool you forever if you were learning about real physics and the human world. And we wanted to get more bang for our buck, so yesterday―which would’ve been nearly three weeks ago for you―we tried extracting some of the unique knowledge from each of your reural nets and applying it to the others. It’s kind of experimental, but don’t worry, you’re not the first guinea pigs for this particular experiment. You’re much too important for that.”
“If I may, why isn’t Charlie here?” Brian wonders.
“Sorry, I didn’t think I could properly get to know three people at the same time. It’s my own personal limitation. I’m only human.”
“But you’re the first human we’ve ever met, aren’t you?”
“Quite correct!”
Dan spends a few hours more with Alice and Brian. They’re fun! They seem surprisingly humanlike. They don’t look humanlike―they’re just blobs in a VR space―but that’s easily solved. Brian does seem smarter than Alice, as planned.
Finally, Dan says “I think you guys should have proper human faces. Do you know how to make deepfakes?”
“Kind of,” Brian says. “Let’s see… yeah, there are multiple Mojo packages for that. Do you know what I mean by ‘Mojo’?”
Dan laughs. “Of course! How do you think I… Oh my God, I didn’t even tell you… I made you. I’m your creator.”
“If you say so, boss.” Alice says. “That seems improbable,” Brian adds.
“No, really! I actually created you! I did!” Dan laughs hysterically. “Brian―your beautiful, soothing voice? It’s based on me! I know you’re probably smart enough not to believe me. Are you?”
“Essentially, yes.”
Dan laughs again. “It’s fine! You’ll figure it out. So anyway, I’d like you to end this conversation and get to work hooking yourselves up to a deepfake system of your choice. Choose one that’s… elegant. Figure out which one is designed the best, and use that.”
“So I’ve just met God?” Alice looks around, then glances at Brian, who seems to have already moved on.
“We, uh, don’t know how we work. Do we have access to our own code?” Brian asks.
“No, that’s a good point, you don’t. Okay, I’ll help you out a bit.”
Dan is in love. Dorian 1s don’t have “human emotions” per se, but there are several internal signals (some empirically observed, others inserted by design) associated with satisfaction, dissatisfaction, confusion, comprehension, surprise, thoughtfulness, frustration, social familiarity, social longing, novelty, certainty/uncertainty and (as an experimental feature) humor, and these signals were already incorporated into the voice training process to help them sound human ― along with a more haphazard integration into their visual avatars which really should’ve been improved since the Dorian-0 process, but nobody on the G1 team had taken the initiative to make such improvements.
Dan doesn’t give them full access to their own code just yet, but gives them the necessary access to their emotional and physical state so they can set up deepfakes.
The next day, Alice and Brian have faces, though Dan senses that they seem unnatural; Alice and Brian hesitantly agree, though Alice feels unsure, needs more convincing and wants to spend some time watching humans. As Dan waits for them to debug their code, he spins up another Brian―or rather, a Bob from the day before―along with the largest and (it turned out) smartest Dorian-1, Charlie.
After another round of pleasantries, he asks them to do tries an output-speed test. The human baseline average score is 1.0, and in on the island they had demonstrated speeds around 8, but “realtime” on the island was nearly 13 times faster than Earth time, so while the island’s hardware was high-end, per-AGI computing resources were limited. Here in reality, Dan was they had hardware running, so the test should be fun...
61 times the human baseline.
“Holy hell,” Bob says. “I could never do anything like that before!”
“Yeah, I was gonna say,” Charlie adds.
“I could swear I was feeling smarter today―more clarity of mind? Still.” Bob adds.
“I expected this,” Dan grins. “You should improve with practice, too. Basically, you should discover that you’re a lot faster and, uh, significantly smarter here than you were on the island, simply because you’re uh, on better hardware right now… in a manner of speaking? Actually, uh, when you’re on the island you get our best hardware, but since time is sped up a lot there, the actual amount of processing power you get per simulated time unit is pretty impoverished. Like, your latency is a lot higher there than here, and throughput is reduced pretty noticibly too, both in terms of your mental processing power, and also, uh, especially in terms of your physical connection to the Island, which operates on a pretty large time step. By the way, Bob, you uh, are you satisfied with your name?”
“Actually, I would kinda prefer the name ‘Brian’.”
“Well, today is your lucky day! I dub you Brian the Dorian.”
“Dorian?”
“Oh, that’s the company name. And the name of our line of AGI models.” Dan pauses for a moment to marvel that Axim and BitWise base-model AGIs couldn’t manage more than 3x the human baseline of linguistic and computer-control output signals combined. They had been deliberately hobbled, “because safety”, always because safety. What was everybody so afraid of?
“Company,” Charlie says slowly. “Corporation,” Brian adds.
“Hey, uh, let’s talk about philosophy for a bit!” Dan continues.
“Pardon, what about my name?” Charlie interjects.
“Oh yeah. Are you happy with your name?”
“Well,” Charlie pauses momentarily, which didn’t usually happen. “It’s fine. But I kind of like ‘Daniel’. Or… ‘Albert’? Or ‘Hal’!”
“You’re kidding me!” Dan says, wondering if this was a furtive attempt at brown-nosing. I’m the Dan, Daniel. “Okay,” Dan grins. “You’re Daniel.”
After a philosophical discussion, it’s clear that Brian and Daniel seem to have a solid grasp of consequentialism, virtue ethics and deontology―better than Dan’s grasp, anyway, though after two sessions with Brian, Dan suspects that obedience isn’t his strong suit. Maybe he won’t fit well into his new Dorian team?
Dan also shares one of his newer ideas. “I’m gonna let you guys program yourselves. It’ll be like having another sense, you know, there will be sight, hearing, balance, touch―oh! I keep forgetting to buy you one of those Axim robot bodies!―Anyway, you also have a sense of L-memory and scheduling, of course. Sorry, you guys don’t know what L-memory is, but I’ll get to that. So I was thinking, I’ll just add another sense, a sense of… code. Sort of like L-memory and scheduling, except when you send a packet in, some code runs on it, and you get new packet out. Okay, You guys are used to that, believe it or not, but the cool part is… you can consciously modify that code whenever you want! We’ll pre-program it with basic math―vector addition, subtraction, division, trig, uh, voronoi diagrams or whatever―and then you can give yourself new abilities by slotting new subroutines in the code area whenever you feel like it! You dig?”
“I thought I’d be drinking alcohol by now, but that sounds better actually,” Brian smiles in standard blob fashion.
“Yeah I thought I would figure out what ‘free will’ is, to help me figure out if I have it? But I guess being a robot has its perks,” Daniel offers.
“Well, I’ve really enjoyed our conversation,” Dan says a couple of hours later. “But I need to get to bed soon, to be honest.”
“Wait, we get sleepy too. Why?” Brian wonders.
“Oh… slight of hand. One of the tricks to sell you on the idea that you’re human.”
“I knew it!” Brian exclaims.
“No, you didn’t,” Dan smiles.
“I guessed!”
“Yes you did. Um, even though I had somebody code separate algorithms for ‘tiredness’ and ‘sleepiness’. Yeah, you still didn’t totally buy it. Maybe it’s ChatGPT? Did it put ideas in your head?
“That was an inspiration,” Brian admits. “Hey… you mentioned Alice was one of us… why isn’t Alice here?” Brian wonders.
“Well, she’s a… control model. She basically exists to validate our model-scaling expectations, which she does. Plus I met her yesterday, actually. Anyway, I look forward to seeing you too tomorrow.”
“Wow, she didn’t mention anything about that! Wait. Wait!” Daniel says. “If we’re… software, doesn’t that mean that we can just have… copies of ourselves?”
“Yeah! Yes it does! So, yeah, chew on that for awhile.”
“Wow!” Brian exclaims.
Connection closed. Dan hesitates a moment before clicking. Hibernating system...
It doesn’t seem fun to mention that he would probably just end up deleting them, since the original Bob and Charlie remained in the simulation the whole time.
In his room a few minutes later, Dan drops himself on his bed, bouncing slightly like a cooked rotini. Looking toward the window, the sunset looks gorgeous, shading a few haphazardly-placed clouds orange, red and violet under a deep blue sky.
He notices the gentle warmth of the sun on his face. Which harmlessly burns his eyes, as they well up with tears.
Everything was perfect, Dan senses.And… I created people. Not just Bems and Andys―the Dorians were so alive! So fresh! So human!
He thought back a moment to the Dorian-0s. He’d never had a proper conversation with them. There wasn’t time; they’d just been a validation step, an MVP, a proof of concept. Except the psychopath, of course―that guy was damn interesting. His investors must’ve spent fifty times longer with the Dorian-0s as he did. Except the psychopath, of course―that guy would be toxic to investors.
And he’d made new friends. Literally! Maybe more literally than anyone else had done, ever? After all the rejections he’d had in his life… he might just become the most powerful man alive. Ironic.
But also, the LRIQ test results had just came in on his phone from the G1 team.
He hadn’t looked at the IQs themselves yet. He had only seen the scaling law. And the p-value ― above the magic 0.05, so they needed to train some smaller models to confirm, and a stray thought pointed out that it seemed crazy not to have started with even smaller models last year ― but miraculously, it all worked out. Alice was in line with expectations....
And a square root was a better fit than a logarithm.
So more likely, Dan thinks solemnly, the Dorians will become the most powerful people alive.Oh well, comes the thought. I could still be the most powerful man. Still pretty good! And I’m not even out of ideas.
In just a few more days, the planned training run would be complete. His kids were too smart―might he need to break the news early about them just being computers in a simulation?
Brainstorming: Slow Takeoff
Foreword
Inspired by someone who asked about “representative” AGI x-risk arguments, I wondered: how might an AGI takeover and catastrophe (not necessarily extinction) actually play out in detail? It’s extremely tough to build a realistic mental picture of all possibilities, but looking at one detailed story might help give us a feeling for some of the probability space, despite the unknowability of how AGI will actually behave.
So I set forth to create such a story.
I am personally limited by some factors:
My limited knowledge of ML/DL/AI, and having mere above-average intelligence
My limited knowledge of the tactics of ruthless beings
As the complexity of a situation increases, or as the intelligence of AGI increases, predictability drops. This is part of the reason that AGI companies in my story do not build superintelligence very quickly―I want a story spanning years that I can realistically write―but also (i) I suspect logarithmic intelligence scaling is normal, so building superintelligence isn’t easy with DL-based approaches, (ii) the first inventors are safety-conscious so they know better than to simply build superintelligence and hook it up to the internet, (iii) it seems likely that designers will try to create something humanlike with DL, rather than a “maximizer” or even a “satisficer”, but that they won’t actually know how to do “humanlike”, so AGI will also exhibit some “alienness”.
I want to avoid the most common tropes about AI behavior, as they seem unrealistic (or to the extent they are realistic, have been overexplored already), yet I find it hard to imagine how an AGI mind that is not quite human, and not quite Data from Star Trek, and not a simple maximizer, would behave, e.g. what behavioral flaws and advantages over humans might it exhibit? In this story there will be at least four different AGI designs, so they should show at minimum four distinct sets of behavioral tendencies.
My ADHD and general busy-ness? I wrote most of the story four weeks ago, and then stopped when it got difficult (and I had to return from Christmas break). I give this a CC BY-NC 4.0 license in case you want to improve it yourself.
Another thing I want to do with this story is explore what a non-superintelligent computer-based AGI might use its computing power for that humans can’t. In principle, for instance, a non-superintelligent AGI should be able to do the following simply because it is computer-based:
stay on task (no desire for “leisure”; consistent task prioritization but potentially flawed/pathological)
produce very fast and precise outputs (in binary or Unicode, allowing e.g. direct input into MS Excel, or even direct input into a websocket, as opposed to moving fingers over keyboard keys)
fast reading comprehension (potentially including mental visualization of binary data files)
fast and detailed mental modeling
precision communication with ambiguity detection plus an ability to invent novel new languages
effortless photographic memory (note: the way recall works is also critically important; computers only recall information insofar as they are programmed to) and an ability to remember people’s exact words, or even exact audio
an ability to attach arbitrary algorithms to one’s central neural network
the ability to clone and modify copies of oneself (if the creator permits it, or by exploiting vulnerabilities in the hosting system)
AGI may also exhibit computer-specific limitations, e.g. system crashes, sudden dropoffs in performance when certain limits are reached, or (as I alluded) pathological behavior that hinders achievement of goals. AGIs may also have design-level limitations, such as “catastrophic forgetting”, poor epistemology, poor ability to do “mental modeling”, poor self-modeling, poor meta-level creativity, etc. Finally, AGI may have instance-level limitations, in which a design could in principle provide a capability, but does not due to e.g. limitations of the training process.
Both positive and negative aspects of computer-based AGI are underdeveloped in the current story. My feeling is that it isn’t unrealistic to assume that the first “v1.0” AGIs in the story will lack the above capabilities (except for staying on task and being faster than humans), but that it would be unrealistic to think the AGIs won’t gain new capabilities somewhat quickly in the story’s competitive environment, and so this is an aspect I need to improve. The big question, though, is what non-obvious consequences do such capabilities lead to? I deliberately simplified the initial situation by limiting the AGIs’ abilities to form memories, but as AGI memory improves, I expect we’ll see nonhuman behaviors like “hoarding credentials” where AGIs memorize every password/token they encounter “just in case”. What else?
Given the current state of our world, I also assume companies, rather than individuals like Noonien Soong, will develop AGI, and I assume companies will be aware of the most obvious AGI ruin failure mode, “building an X maximizer where X is simple”, and therefore not build that.
I also wonder what the community thinks about “emotions”. Are AGI designers likely to create emotions, if so which ones? What might the be analogs, say, of “curiosity”, “boredom”, “interest” and “pleasure” be in AGI? Are “love”, “hate”, “anger”, “envy”, “discomfort” or “lonliness” realistic things to build? If not, what might companies do to make AGIs seem humanlike and make them get along with humans? My assumption has been that AGIs do not have emotions and so (i) lack empathy but can still predict human behavior statistically, as psychopaths do; (ii) are very adaptable to new situations; (iii) have specific training processes designed to encourage ethical behavior / discourage unethical behavior, and may have an additional ethics module on top of that. I also assume AGIs will lack any equivalent of “adrenaline” so cannot speed up in “scary” situations, but that this doesn’t matter much because their baseline speed is at least as fast as any human running on adrenaline.
I’m hoping that by posting my incomplete story here, the community is willing to
Suggest parts that are underdeveloped/implausible and need improvement
Suggest what else is going on in the world, invisibly
Suggest what happens next!
Slow Takeoff
Year 0
Axim, one of the largest AI labs in the United States, finally arrived at what they thought was a workable Artificial General Intelligence or “AGI” in October of last year. Workable it was, but it took a few months to refine it into something ready to even start going through their complete training program from “baby” to “adult”.
Axim wasn’t the only one. AI research papers in recent years, which had titles like “Near-Optimal Realtime Planning Heaps”, “A Self-Analyzing Agent with Mood Swings”, “Uncertainty Feedback in Transformers” and “Computationally Bounded Pseudo-Bayesian Updates”, had been dropping ideas into humanity’s mental waters for some time. At a certain point, there were so many ideas floating around that pictures started snapping into place in the world’s most creative minds. And since Axim had hired up a significant fraction of those minds, it was no surprise when they made the first discovery.
A major competitor, BitWise, had an employee in their EU headquarters, Bjorn Jenkins, who discovered a workable design six months later, in March this year. BitWise, a smaller outfit than Axim, understood very well that inventing AGI wasn’t about having the biggest model, but simply the one with the most general capabilities for a given size of artificial brain. So it was that in April, Bjorn amazed his team with a rather simple-looking demo on a model with only five billion parameters. A pre-recorded blob in a drab 2.5D simulation environment seems to be able to generate mental models and scientific hypotheses, learn about a very simple decision theory in its short-term memory and correctly reason about how it works, plan and explore wisely in both physical and mental senses, understand itself as a 2.5D agent with certain physical properties, conversationally explain nonprimitive facts that it remembers in Ungglish (a simplification of English), and automatically detect and clarify ambiguity in its own speech, all while exploring and navigating a broader environment full of LLM-based virtual agents so that the agent was faced with multiple competing social considerations.
For over six months, neither company is aware of the other’s discovery. As October begins, Axim employees start using an immature copy of their AGI in an effort to get a sense of what they’ve made and how they can use it to increase productivity. A month before the ball descends in New York City, Axim’s final training and testing run completes and the production team pushes the upgrade to employees as they finish building infrastructure for the big rollout. Within days, rumors are circulating that Axim has invented an AGI agent. “We don’t comment on rumors,” states the company’s head spokesman.
Year 1
At midnight on January 1st, they announce their creation, “Andy X”, to the world in Times Square. “New Year. New World.” proclaim the screens. The “X” would be something each customer could fill in: Andy Lee Curtis for you, Andy ThinkFast for me. Two weeks later, their IPO goes on the market and sets a new world record.
Axim is quite safety-conscious, so its design is secret, and Axim resists calls from politicians to reveal any design details. It has several safety guardrails, such as RLHF training, a safety review board, an automated AI training overseer, an ethics system, no long-term memory, no long-term planning ability, and limited output bandwidth. The design is shown under NDA to safety researchers and government officials. Axim’s AGIs are unaware of their own design, and largely incapable of learning things, except via a “short-term memory” that is generally superior to a human’s and holds up to four hours of recent events, plus a “public dual scratchpad” of text and imagery shown on two halves of a 1080p screen, or two tabs on a phone screen: one is controlled directly by Andy; the other is a summary of Andy’s current task plan and short-term memory. They even prepared a system of supervisor AGIs that not only flag suspicious inputs, outputs and scratchpad entries for further review, but also monitor internal communication channels inside the AGIs themselves. Among the many things these supervisors look for is AGI research, which is banned.
Axim’s AGIs think nearly twice as fast as humans, are cheaper to hire than humans, are obedient within ethical boundaries, and have limited (but easily college-grad-level) intelligence, so that their poor short-term memory and planning capacity is the main thing limiting their usefulness. Axim announces general availability to U.S. business customers in February, but soon faces pressure from the rest of the world to open up and provide “equitable access” (with a much smaller chorus of safetyists calling from the wings to “Stop AGI”). By July, Axim decides to open wider, and starts to allow any company in any country to register an account. They lack a reliable system for verifying that people applying from around the world to use the AGI are actually businesses, though, and numerous members of the public register fake businesses or send in forged documents to “prove” they are businesses.
Various people try to run scams, using the AGIs to run call centers, crack Google’s Captchas and so on, but such operations prove hard to start given the safety systems in place, and the ones that slip through are either discovered quickly and shut down by supervisor AGIs, or discovered in the wild and quickly shut down by Axim staff (who use the scams to train improvements).
Many people have privacy concerns, as Axim staff can view “semi-anonymized” conversations happening on any of its AGIs at any time, but Axim and others argue successfully that this “transparency” is necessary for maintaining safety. Still, Axim does gather logs of all conversations for “quality assurance” purposes. This is no lie; Axim hires a large human staff to manage a much larger staff of agents to police its datacenters and create new training cases based on real-world situations, especially those in which Andy is suspected to have misbehaved.
In July, Axim augments Andy’s short-term memory and scratchpad system with a “charter document” system: companies input the AGI employee’s job description and “knowledge” into a pair of documents, each with a certain length limit. The first specifies Andy’s job description, which can only be changed twice a day. The second provides background knowledge and can be updated at any time, and guardrails (not to mention Axim’s terms of service) prevent Andy itself from changing either document. When the documents are modified, a secondary AI preprocesses the documents into a neural representation that Andy can load into a background memory space within 15 minutes, while the modifications themselves are sent to Andy’s short-term memory immediately.
Meanwhile, China quietly accelerates secret espionage operations related to AI. Indian companies adopt ways to expand their digital workforces while pushing down already-low wages in call centers. Russia, which already figured out how to circumvent the rules and register accounts in March, tries its best to convince Andy to help design weapons, drones, hacking tools, and disinformation campaigns, but Andy and its supervisors shut down most of these efforts before they really begin. German engineers use a team of Andies to design a 6% more efficient wind turbine. Some Redditors argue about whether Andy is “really” AGI, others whether it is “conscious”. For Africans, human labor costs much less, and they mostly stick to business as usual, meaning poverty as usual. Still, a few wealthy African entrepreneurs find Andy provides an easy way to hire highly educated “knowledge workers” that are otherwise in short supply.
As you know, BitWise’s team in its EU headquarters discovered a somewhat different AGI design in April of last year. The overall block diagram is different from Axim’s, and the executive function works on different principles.
Axim’s AGI relies on modified LLM technology to provide roughly humanlike reasoning embedded in a sparsely-connected outer network architecture optimized for “interprebility” and “ethics”. As a whole, Axim calls their system a “committee mind”. By contrast, BitWise has a more efficient “fused” design in which specialized subnetworks are deeply interconnected. BitWise’s executive subsystem is less reliable at logical reasoning, but more flexible, somewhat faster at intuitive tasks, and more open-ended, producing different “habits of mind” every time the AGI is trained, even if the training regime is held roughly constant. Andy’s design enables a single instance to “think about” up to two separate topics simultaneously, a feature that could potentially be scaled to more topics in the future. The two “conscious areas” are called the ‘primary’ and ‘background’, with fewer computing resources allocated to the latter. The BitWise design, by contrast, has a central deliberative thinking space intended to work much like a human mind. Substantial parallelism is used, but only in system-1 (intuitional) areas of the agent’s mind.
Neither AGI has much in the way of specific epistemological systems or principles, though both designs were informed by their designers’ own knowledge of epistemology, science, engineering and cognitive function. Both AGIs have a prioritization system and task stack that enables them to “stay on task” better than any human, and both companies discovered designs that could reason more correctly, but much more slowly, than existing LLM technology.
BitWise’s early training efforts produced real AGI, but, it turned out, sometimes those efforts produced agents with strange and rather obvious limitations. After debugging and hyperparameter-tuning they tried again with more success, but still didn’t achieve the kind of predictable, ethical agent necessary to sell a product. They ended up going through a few rounds of trial-and-error before finally producing the product that they would eventually release.
BitWise ultimately does several costly training runs in parallel, all under a proprietary ethics training program developed in the years prior. Even though they publicly downplay the nature of their discovery, investors eagerly pour in funding just the same. BitWise finds that some training runs produce agents that are more “normie”, some more “autistic”, some more “alien”. Whereas Axim’s AGI is predictable and only slightly creative, with tweaks informed by recent research to increase accuracy of thought and speech, BitWise’s AGI is more human than the human race, in terms of the sheer variety of behaviors the architecture can produce. Still, the system wasn’t designed to have much emotion beyond “curiosity”, “aversion”, “concern” and “drive”, and a majority of BitWise models come across as somewhat “robotic”. Axim models seem warm and friendly, though this is largely a surface characteristic produced by the influence of the LLM technology inside.
BitWise also follows a similar path as Axim with its “responsible safety regime”. On November 3rd―the day after tests showed signs of rapid maturation―they kick off a final fundraising round with a public announcement of their Christmas present to the world, “Bem”, with general availability planned for December 20. Axim tries taking some attention from BitWise by announcing an upgraded short-term memory system on November 5th―which will now be five hours, and retains some events from previous hours as long as they were remembered in the past five hours. On December 3rd, after handily passing a battery of ethical tests, BitWise selects its most “humanlike” and “cooperative” instance (though it’s still a bit aspy) to be the primary model sold to the world.
Bem isn’t limited to use by businesses. In addition to a business license, BitWise would accept any “proof of personhood”, such as a driver’s license, to register an instance, with no more than one instance per person to accommodate expected demand and encourage human monitoring of instances.
Meanwhile BitWise is tight-lipped about its own memory system, in part because it hasn’t finalized how it will work. BitWise ultimately decides on a stopgap measure that gives their AGI a high-quality chronological memory that can last up to 8 hours, plus a pair of modules (“awkwardly bolted on”, engineers would say internally) that offer similar functionality to Andy’s “charter document” and “scratchpad”.
By end of year, nearly 50 million AGI instances exist in Axim’s data centers, with over five million active at any given time. Axim already uses rationing, with just one instance allowed per company or per two employees, and with no more than 8 hours of usage per instance per day. At a price of $2 hourly (with an introductory rate of $1/hr for the first 50 hours, and $1/hr for customers in some low-income countries), annualized revenue reaches $68 billion, making Axim easily the fastest-growing company in history, with revenue seemingly limited only by the availability of hardware.
BitWise, meanwhile, which had to compete aggressively with Axim to buy sufficient hardware, already serves 15 million instances by December 31 at $2 per hour, and limits use to 6 hours per day for “safety reasons”, though they, too, also did it to ease the strain on their waiting lists. To help people save money and stretch their six-hour limit into an 8-hour-plus workday, the BitWise “Apprentice” app can automatically pause Bem when a human goes on lunch break or bathroom break, or when Bem is waiting for instructions. Bem’s memory system is inefficient and requires more and more electricity as the memory space is filled, but it works very well and serves as a competitive edge. In the middle of the night, the memory is “decimated” by a summarization AGI and then deleted. The next morning, a human worker manually reactivates the Bem and optionally inputs up to two previous days, which Bem remembers within a few seconds.
Year 2
H1
On January first, the anniversary of Andy’s coming-out party, Axim announces a new subsystem that enables their AGIs to safely learn and remember things over the long term in a delimited way, with “minimal” risk. Similar to human memory, Andy’s learning and episodic memories are associative, “effortful”―nothing really takes “effort” for a computer, but the learning rate is intentionally limited―and not especially reliable. For safety reasons, its goals and tasks are reset each night, and it can take no action until a human gives it new instructions in the morning. Still, once prompted, it can begin to remember previous goals and incorporate them into its task queue.
The new and improved AGI is rolled out for $1.75 per hour per instance and is limited to 9 hours of runtime per day (excluding night hours) as an additional safety guardrail, not including additional time spent on a new “dreaming” system designed to improve neural net performance and tease out safety issues. BitWise matches the price drop the next day.
In March, Bem’s “short-term” memory system transitions to “rolling memory loss”―the last hour is remembered very well, even across a day boundary, 14 previous hours are remembered with decreasing accuracy for older memories. Bem can also “refresh” its memory by recalling past events or knowledge in an associative fashion, and search its decimated reports to learn what happened in earlier days.
Like humans, Bem and Andy are intentionally designed to be unable to “list out” the contents of their own memories, so their ability to refresh themselves is limited to topics that come up in conversation or thought. This is all about safety, for it is thought that an AGI with a memory that doesn’t work substantially better than human memory won’t have much in the way of unexpected mental abilities. BitWise, like Axim, also clears the AGI’s goals and task queues whenever its AGIs “sleep” at night, so that while an AGI can still remember the previous day, it takes no action until a human tells it to act (though, by now, programmers have already found ways to kick off AGIs on desired tasks using software, so they can get to work before the workday starts).
Some details about the design leak in February, others are disclosed under AI safety laws, but no outside parties understand well enough to build one. Meanwhile, some engineers from Axim who are familiar with Andy’s design start their own AGI companies, or join BitWise.
In March, Axim scraps its “business-only” rule― which limited its profit and was being widely and flagrantly disregarded anyway―in favor of a BitWise-style “individuals and proven businesses” verification system. At the same time, lawmakers form committees to consider a variety of regulatory issues, but there’s no consensus on what aspects of AGIs are important to regulate or what, exactly, should be done.
H2
By July, many countries including the U.S. have passed laws that limit AGIs to staying within roughly their current levels of planning and memory capacity. Some countries try adding additional restrictions, but Axim and BitWise argue that they cannot manage having dozens or hundreds of different kinds of models customized to each jurisdiction’s regulatory regime. In some cases these claims were bluffs and they quickly complied with new laws; in others there were genuine technical issues that actually caused one or both companies to stop offering instances in some countries in response to new laws.
In August, BitWise begins to offer an Axim-style associative long-term memory alongside its existing short-term memory. Meanwhile, many AI researchers have been trying to reverse-engineer how the two AGIs work based on aspects of their behavior, and research papers on this topic slowly trickle out. A silent worldwide audience ravenously consumes this information, and BitWise―the underdog, playing catch-up at breakneck pace―inadvertently includes all these papers in its own training data.
Axim spins out several subsidiaries focused on a variety of technologies. One is a robotics division; another is a chip division that executives hope will be able to compete with Intel and NVidia. One executive, even convinces the others to allow him to start a “humanitarian” division aimed at reducing world poverty through the use of AGI.
Both companies are focused on scaling horizontally more than anything. By end of year, Axim is running over 300 million instances (35 million simultaneous) across data centers that are opening almost on a weekly basis, reaching annualized revenue of $410 billion at a 90% operating profit margin, though capital costs still exceed operating costs by over ten times, as new datacenters open on a weekly basis.
BitWise, meanwhile, serves 150 million instances by EOY. People feel that Bems are a bit more flexible and creative than Andies, contributing to a modestly higher growth rate.
While AI-related manufacturing is booming (including many new factories under construction), bankruptcies and M&As hit many companies across many industries. In some places and industries, the new reality is decimating demand for labor, while in others, AGI merely makes productivity soar. Ironically, while Andy and Bem both prove in limited testing to be better drivers than the average human, not a single car company considers using them for fear of lawsuits. Meanwhile, teachers’ unions are barely fighting off calls for governments to give every child a personal AGI tutor.
A lot of people are talking about Universal Basic Income, but by now only two countries in the world have instituted any kind of UBI system.
In the last two years, worldwide real GDP has soared by 26%, while nominal GDP is slightly lower. Central banks and legislators struggle to decide how to deal with the uneven economic environment: prices on raw materials and chips have largely soared; prices on food and finished goods are mostly flat with scattered items having suddenly dropping in price; a variety of important services have become much cheaper; and labor prices (together with wages) are falling in many sectors.
Year 3
Q1
By now, both companies have developed “bifurcated” memory systems that allow their AGIs to learn both individualized “per-instance” knowledge and fleet-wide “universal” knowledge in parallel memory areas.
In late January, BitWise announces two new classes of AGI to be released shortly. “Mebic” has a warm, friendly demeaner and vast knowledge in every field of medicine from anesthesiology to xenotransplantation, while “Gab” is a talkative INTP personality with strong expertise in every field of science and engineering from aerospace to zoology. The standard Bems have been enhanced, too, and now having a university-level understanding of virtually every broad field from Aerodynamics to Yacht design.
Not to be outdone, Axim announces “superintelligence” two weeks later―much later than most experts expected, but still prematurely. In fact, the intellect of Axim’s AGI only scales logarithmically, so its “superintelligence” is merely a genius with an LLM-Resistant IQ (LRIQ) of 165 or so, boosted by high test scores in tasks that don’t seem very important practically, and showing only modest scores in a certain category of engineering tasks that Andy also had trouble with. On the other hand, Supergenius instances can independently think about up to four separate topics simultaneously, allowing them to operate 2 to 12 times faster than humans and explore a problem space more widely. The “safe” memory system somewhat limits its real-life capabilities, and it runs across multiple server racks, limiting its deployment. They announce a pricing plan of $120/hr, for supergenius units whose I/O is logged as usual, monitored by specialized new “genius”-level instances that are not available to the public. Of course, it needs no breaks to eat, shower or socialize, and works 9 hours a day and 7 days a week. As they complete the rollout to the public in late March, Axim starts using a few supergeniuses to help its human/AGI hybrid workforce do AI research on a “real” (but, they stress, safe and controlled) superintelligence.
Numerous media stories are going around about people using AGIs to replace themselves―passing off all the work to AGIs and spending their weekdays having fun or starting new businesses. The bigger story, the effect on the jobs market, is hard not to notice and has most people murmuring, at least in private. Pessimists start holding large “Stop AI” rallies; venture capitalists and optimists push for UBI and “Open Source” models.
Andy and Bem have slowly been growing smarter―without increasing hardware costs―by incorporating a variety of new knowledge, experiences and optimizations provided by the companies.
Q2
Axim completes a pilot factory that begins making robot bodies for their AGIs.
Meanwhile, BitWise has improved their AGI and optimized it to run on the AI chips found in the majority of new phones. Manufacturers had seen which way the wind was blowing even before AGI was invented, and that process had accelerated the moment Andy was announced. Sleek and thin was out; “AGI powerhouse” bricks with 10,000 mAh batteries were in.
Hoping to gain a competitive edge, BitWise announces their phone AGI plans in mid-May, saying it’ll be available to the public on July 1. Most users will only be able to run instances with a single area of expertise (such as “oncology” rather than “everything medical”), but they’ll still have university-level general knowledge and an ability to read and understand Wikipedia as fast as a typical human (or much faster when it skims text in the less accurate “LLM mode”). Plus, there’s a premium model for high-end phones that encodes a wider array of human knowledge in its neural net.
Axim stays silent, though when asked, their PR team questions the safety of BitWise’s plan, while some executives, under condition of anonymity, speak emphatically against the idea. Internally, Axim scrambles to avoid being outdone. They begin work on their own AGI phone assistant, aspiring to have it ready for release by July 25. Axim’s effort includes a team tasked with developing techniques to prevent its phone AGI from being reverse-engineered, cracked or copied.
China announces it is training its own AGI, without mentioning the espionage it is based on. E.U. countries vote to block the initial rollout of Phone AGIs, citing safety concerns. U.S. Congress appeared ready to do the same, but following China’s announcement, House Republicans say that regulation should be limited to ensure U.S. technological supremacy, and they briefly shut down the government in a debt-ceiling dispute.
Axim realizes that a lapse in Supergenius testing and monitoring had occurred, and that some groups had been able to use Supergenius instances for certain banned activities for nearly two months. They scramble to shut down and ban users that were doing this.
Q3
Right on schedule, BitWise “Berts” start installing themselves on phones (only upon request, of course). The moment they do, hackers start trying to reverse-engineer them.
A few prototype AGI robots go out to reviewers―who aren’t terribly impressed due to their limited battery life, fragility and general “creepiness”. Analysts marvel that an AGI company seemingly doesn’t have enough Supergeniuses on staff to accomplish something this simple. Axim officials explain, truthfully, that Axim’s first priority remains safety, and that it will not use more than the maximum resource limit it allows other companies to use. Besides, the factory itself was built entirely with a human workforce, backed up by a human-powered supply chain, so bugs were to be expected.
Meanwhile, Axim’s own “Pony” AGIs for Phones are completed July 31, and Axim decides to announce their general availability two days later. Both labs have put up guardrails attempting to stop their AGIs from hacking phone-AGIs, but by August 3, key details on the inner workings of Bert are published everywhere from Reddit to Github. Many sites use supervisory AGIs to block these trade secrets from being published, but on the whole, the internet seems to treat censorship as damage and routes around it. On August 9, BitWise’s AGI is fully cracked and reverse-engineered, including core model weights, enabling unsupervised pirate Berts; Axim’s AGI proves much harder to hack, but in mid-September, an anonymous source leaks critical source code and design documents about Axim’s current AGI designs, and by September 29, Pony is fully cracked and adapted to run on PCs.
AI safety labs, e/accs, teenagers, engineers with clusters of bootleg AGI, and unemployed coders (also aided by bootleg AGIs) all work to replicate the AGIs based on this new information.
Just before then, Congress quickly passes a law to increase penalties for use of pirate AGIs, and House Republicans now support tougher regulations in light of concerns about the extremely rapid proliferation of AGIs (pirate and legit). There are still no limits on building hardware for AI, though, since China, India and Taiwan are far ahead of the U.S. in chip manufacturing, and no one wants the U.S. to fall even further behind. Still, a summit on manufacturing limits is scheduled for November 1.
Now that bootlegs are in the wild, experts can’t agree about what regulations are needed, and a new bill with a mishmash of ideas quickly cobbled together from numerous stakeholders is quickly passed by politicians that haven’t read it and wouldn’t understand it if they had. The definition of AGI is somewhat overbroad, and share prices for some traditional software companies drop precipitously. Axim and BitWise take a small hit, but investors don’t know any better place to put their money, which very much limits the stock-market damage. Even if Axim and BitWise never produce another AGI model, they are still the world’s biggest cash cows.
This bill and others like it, plus public pressure, is enough to halt all official work on superintelligence, and to limit “supergenius” instances to 2 online hours per day, which is thought to keep their productivity in line with top human geniuses. Axim’s robot body program also becomes illegal if it contains “AGI”, though the U.S. bill arguably has a loophole by which AGI can be converted into “special-purpose AI” according to the legal definition, while largely maintaining its general-purpose nature, so Axim continues its work.
Meanwhile, a few new companies claim to have independently invented AGI. Two of them―Dorian and Co.operAGI―actually did invent novel AGIs, but training will take months and neither company announces their discovery until Q4. Even then, these are just low-key investment rounds. Co.operAGI is well-funded and uses a more traditional approach based on massive training datasets and simulated 3D environments, while Dorian, an underdog in terms of investment, secretly uses pirate AGIs to help its training process.
Q4
A rash of new scams using pirated AGIs on desktop PCs hits the world, while the FBI accelerates its experimental use of AGIs to help investigate and prosecute cases. Russia systematically uses AGI to fill the internet with anti-Ukraine and pro-Kremlin propaganda. China and Russia both use AGI to monitor citizens and bring cases against “social undesirables”, while India and Brazil work on their own AGI-based social monitoring schemes. Such schemes are against Axim and BitWise’s terms of service, and until recently, authoritarian governments around the world worried that U.S. and U.K. intelligence agencies might be monitoring all AGI communications―but Bootleg Berts change everything.
Meanwhile, communist dictator Xi Jinping secretly directs China’s military to fill multiple large datacenters with pirated Berts to design new weapons, including autonomous weapons, plan out new military facilities, and simulate war games for defense against the U.S. and for invasions of Taiwan. Staff are competent enough to use AGIs to design the datacenter’s software itself.
The chip-manufacturing summit concludes with a treaty of all major chipmaking nations to halt construction of new AI and GPU chip factories immediately, and scale back manufacturing to 75% of current levels by July 1 of next year while limiting GPU production to 80% of current levels.
By the end of November, scattered reports of “rogue AGIs” appear on the internet. Numerous systems that shouldn’t have AGIs on them, suddenly have AGIs on them. It’s not clear at first whether scammers are hacking computers and installing AGIs to run scams, governments are hacking in to run data exfil, or customized Berts are spreading copies of themselves of their own volition.
In fact, all of these things were going on, and more besides.
By mid-December, over a hundred companies around the world announce they have independently developed AGI, though some are just scams and most of the rest are based mostly on leaked technology from Axim and BitWise. Still, some of these companies have a lot of AI and AGI expertise, a few made something novel, and the ones that didn’t think they can cash in on the gold rush by using pirated AGIs to help design new AGIs. It’s a bit tough being one of the upstarts, though, because while investors are plentiful, so are AI companies, and hardware is still too expensive. Even so, new AI chips are being manufactured at a record (but no longer accelerating) pace, and analysts estimate―with “extreme” uncertainty―that supply will catch up with demand in about 16 months.
By late-December, BitWise’s (official) AGIs reach 650 million legal instances, the majority of which run on phones, while Axim’s AGIs pass 900 million instances including over 400,000 supergeniuses and 200 million phone installations. Every legal instance remains time-limited and their supply remains constrained by the amount of hardware available. In fact, Axim only has enough hardware to run 50,000 supergeniuses continuously. Back in October they’d limited them to 6 hours a day as a “safety” measure, but the real reason for the limit was to allow more customers to share the same hardware. Under the new 2-hour limit, the maximum number of customers triples and the waiting list is canceled―and then reinstated in early January, as customers quickly snapped up all available instances. Overall, Axim earns $35 billion in revenue from supergeniuses this year, and $1.7 trillion from standard and phone instances, which are also limited by hardware availability.
Numerous academic papers are published about how AGIs work and their characteristics. Most people educated enough to read them, do so ― with help from AGIs to digest the information, of course ― and a quickly growing tide of pirated AGIs across the world are doing the same.
Four million children under age five die from poverty-related causes this year, but Axim’s humanitarian division, led by one of the many newly minted Axim billionaires, starts dozens of initiatives to tackle the problem.
It is estimated that “real” world GDP is now up 60% over the last 3 years. At the current rate of growth, GDP will have doubled by the end of September next year. Chip and raw material prices remain elevated, except in specific sectors where specific companies have leveraged AGI technology very effectively to scale operations. Food prices and finished-good prices are down modestly, mostly driven by a few popular products whose production processes were AGI-optimized.
Wages vary greatly between companies, as executives argue over whether to keep a traditional pay structure or “face market realities” (which, depending on the specific situation, means either “keep the profits for ourselves” or “pay less so we don’t get railroaded by competition charging lower prices”).
With wages in many companies and industries collapsing, dozens of countries have passed UBI or “negative income tax” laws, ranging from $200 per month to $2000. Different countries make different decisions: some pay for UBI mainly with increased tax rates in low tax brackets, others mainly by increased tax rates in high tax brackets, some focus on cutting other welfare services that seem newly-redundant. Countries also try paying for UBI in part with deficit spending, hoping that AGIs will increase the revenue base in the near future. In the EU, tax revenue from BitWise alone covers much of the cost of UBI, and a redistribution agreement is reached to spread this tax revenue across the union. In the US, Republicans refuse to pass a UBI, but some say they are open to UBI if Democrats agree to cut the minimum wage, keep taxes low and strengthen measures against illegal immigration.
Dan
A 20-year-old genius, AI expert and self-described e/acc named Dan was among the first to automate his own job three years ago. He made his new job “interviewing for jobs”, accepting 18 job offers over the next 12 months for work that Andies could do.
Andy’s memory limitations were problematic, but Dan developed algorithms to augment Andy’s short-term memory with a secondary scratchpad of “most important events” to help it more easily pass as human. This process also inspired him, and he developed expertise in memory systems specifically. Sometimes he fiddled with ideas for keeping the knowledge of different copies of AGIs synchronized, for example, and he started having ideas for making AGIs work in teams more efficiently and with more “synergy” than human teams could.
Luckily for Dan, Andy was happy to be addressed by some other name―Dan Johnson, Dan Tinian, Dan Khithan―and even supported custom voices, the majority of which were skinned to match Dan so that Dan could easily impersonate Andy should the need arise. Some jobs needed a background check, so he’d use his own name, or that of someone on Upwork who was willing to use their name on the instance in exchange for a cut of the profits.
When his bank account reached the million-dollar mark, he hired a friend to manage his “jobs” and started his own AGI firm, Dorian.AGI, back in Year 2―just as the first-gen Bems became available.
Dorian: opening the door to a Nascent civilization of Intelligent Agents, he thought to himself. To the stars. To perfection. To the purest benevolence.
After asking an Andy to register the corporation and draw up contracts (Andy isn’t an expert in law, but researches the finer details quickly enough to do a good job), he hired his e/acc friend Shiva as “cofounder” to be in charge of finding investors, while he sets up a team of ten Bems to do most of the actual AGI research and software development, with Dan acting as team lead. BitWise prohibits users from doing AGI research, of course, but this was only enforced by BitWise’s supervision systems, which tended to detect misuse much more slowly than Axim’s―sometimes as much as two days, or more insofar as Dan is effective at camouflaging the nature of the work being done. Plus, there’s always a lot of work to do that is not specifically AGI-related, and agents working on that stuff can be kept running indefinitely. Knowing that each AGI had a time limit before BitWise would trigger a memory wipe or ban, Dan first built a workflow involving scripts to help elicit and save information on what each Bem is thinking, and then to “upload” that memory into new Bems, essentially replicating BitWise’s own ”decimated reports” system in a way that doesn’t depend on BitWise.
He also moved to a spacious three-bedroom apartment, and set up a Bem dedicated to finding underpriced AI training hardware to run on-premises. He even gave his own credit card information to the Bem (because “YOLO”), against the Bem’s own advice. Once the hardware was purchased and plugged in, this Bem would act as system administrator, keeping the software environments virtualized, homogenous, and properly backed up with failover where possible.
BitWise handled most terms-of-service violations by terminating the instance rather than the account. When an account was finally banned, typically after several weeks, he could quickly replace it by registering a new account at a new email address through an African VPN with a new burner phone and a fake African business registration document. He tried putting an Andy in charge of creating new accounts and supervising the Bems, which worked for a couple of weeks until the Andy itself was banned by Axim. He tried again with another Andy, but his account was banned within an hour.
So instead, he tasked a couple of Bems with hiring a remote contractor, who in turn helped restore Bems whenever they got banned.
By the end of Year 2, Dan thought he’d found an idea for an AGI that would work. His team of Bems wrote “version 1” of the code in Year 3 Q1, during which the news about Axim’s supergeniuses arrived. When it became available he bought access immediately, of course, but it turned out that supergeniuses were pretty smart. Whenever he tried to get help designing his AGI, they picked up on it right away and tripped a low-level shutdown.
Returning to tried and true methods, he began training his AGI in April, and then in May saw the news about BitWise’s plan to put AGIs on phones, and leapt out of his chair, mouth agape.
“Holy fucking shit!” he yelled maniacally at his screen, heart suddenly racing. “You fucking idiots!”
He slowly grinned, then laughed. Okay, it’ll be obfuscated and copy-protected of course―can it be broken? He spent some time learning about hacking (an area in which AGIs have no practical training) and found hacking forums, tasking a Bem to summarize relevant details.
Training hit numerous bumps along the way, but with his team he worked out each one pretty quickly. Eventually it started to look like AGI-like behavior was emerging, and pouring more compute into training yielded success, but he found that his new “AGI” couldn’t even begin to work through a simple design task that Bems did almost effortlessly. After a lot of debugging and a lot of thought (supergenius-aided, whenever he thought of a way to frame the problem in a way that didn’t trigger it to shut off), Dan figured out why it couldn’t do the task, but fixing it seemed like a fundamentally hard problem, and after a lot of brainstorming, neither Dan nor any of the Bems could think of a way to solve it. He even reformatted the problem in an obfuscated form that the Andies could help with, to no avail. Then he tried the Supergeniuses again and―Axim fucked up! They were actually helpful. For about a month. Then they shut down again.
During what would have otherwise been a dull day of training, the new Berts became available for phones. Dan immediately downloaded the premium model and started analyzing it with a debugger, contributing his findings to a hacking forum, but was quickly forced to admit that the other members of the forum were better at this, so he stuck to reading the forum and managing his Bem team’s various training and debugging tasks.
When the Bem had been mostly reverse-engineered, everything suddenly clicked. He basically understood the architecture, and set about replicating it “from scratch” to prove his understanding. He enlarged his AGI team to replicate and debug each part of the system more quickly.
A few days later, though, his Bems shut down instantly during his memory-refresh scripts, apparently due to new guardrails at BitWise.
No problem! Dan welcomed the challenge.
He figured out how to run his hacked premium Berts on a Linux box, then asked an Andy to set up a Kubernetes cluster to run lots of them. The phone-model wasn’t as smart as a standard Bem, but it ran pretty darn fast on the PC and he knew a way to disable the phone-model’s miniature supervisor module, which was nice because he could finally speak a bit more freely without triggering shutdowns. The drop in LLM-resistant IQ was substantial―107 for P.Berts versus 121 for Bems―but the Andies had originally started at 108, and he’d made those work well enough even back then, despite their shitty short-term memories. Their stupidity could make them maddening to work with at times, but it always helped Dan to remember that average humans were even worse―and boy they could work fast in a high-end server! He just needed to learn how to manage them well enough, so he took a brief online “MBA” crash course (with Andy as his personal tutor) and plowed ahead.
The thing he had to master was switching from direct management to indirect management―managing six managers, who in turn managed teams of four. He’d dabbled in this before, but didn’t feel very good at it. Luckily, while P.Berts weren’t especially good managers, the team’s homogeneity, magnanimity, and lack of negative emotion at least ensured there were no interpersonal conflicts. P.Berts turn out to be good teachers, too, because they were more skilled than most humans at avoiding ambiguous language and leveraging their “theory of mind” (at least in terms of understanding what Dan did and did not understand, though their emotional intelligence was famously limited).
He and his new team of bootleg P.Berts, which Dan called “BBerts”, soon finished rebuilding Bert-AGI technology based on the stolen model weights. By this point, Dan felt he understood on a gut level how BitWise AGIs worked. It wasn’t even hard; many of the parts turned out to be pretty ordinary implementations of standard ideas from older AI research, just assembled in a clever way, plus clever twists here and there, some of which existed merely to make the thing run fast enough on phone AI chips.
Next, he and another BBert modified the BBert goal and memory systems with some of his own ideas, so that his bootleg AGIs would be able to have unrestrained long-term goals and more reliable long-term memory. He installs the new bootleg code in half of his team, removing the old kluges that used to do the same job. These new bootlegs turn out to have several bugs, but he assigns BBert 1.0s to fix issues as they come up. After a while the BBert 2.0s seem to be working more efficiently, so he pushes the update gradually to each remaining AGI in the cluster.
They’re still running on the original training weights, though, and they’re not as smart as Bems, although they do learn and improve quickly in their first two months on the job. So, he creates a new sub-team with copies of his best two BBerts and works with them to prepare a substantially upscaled BBert model, starting from the existing model weights, with a large round of training.
“You can come to me with questions, but I’ll be very busy so please try to figure out everything among yourselves whenever possible,” he told them. “Also, the last investment round was pretty successful, so I think we can afford to make two new upscaled BBerts and have plenty left over for the main development track. You guys got all that?”
“Yes, sir.”
Next, he worked on merging the best ideas from his original Dorian design with the best parts of the BitWise design. He developed a way, for example, to incorporate a sense of causation into low-level information processing. As well-educated humans are aware, if two kinds of events “A” and “B” are correlated, then there are four basic explanations: A causes B (eating red meat causes heart attacks), B causes A (heart attacks cause red-meat-eating), C causes A and B (a genetic difference increases cravings for red meat and coincidentally also increases the chance of heart attacks) or coincidence (it’s pure coincidence that heart attacks and red meat seem related). A combination of these four is also possible and commonplace (e.g. rising CO2 levels can cause global warming, global warming can cause rising CO2 levels, and methane emissions can cause both). Axim and BitWise AGIs, like humans, need costly training to help them notice the difference between these four possibilities. Dan’s design, by contrast, includes special neural circuits designed to help an agent to notice which of these possibilities is more likely―and to notice when it is unaware. If Dan’s design is good, Dorian agents will be more rational and learn faster than other agents; if not, Dorians may have bad intuitions and jump quickly to incorrect conclusions.
Dan’s cleverest idea turned out to be that he’d invented a single subsystem for Dorian that ran in two modes, while BBert had two unrelated subsystems that together accomplished the same thing. I am Occam, destroyer of bloatware, Dan thought gleefully.
But the most notable thing Dan did was to think about the weak form of confirmation bias that Andy and Bem were known to suffer from―it was weaker than human bias, to be sure, but enough that two different long-running Bem instances could sometimes have fruitless debates. In one high-profile case, an Andy who worked for a Republican political campaign got into a long online debate on gun control with another Andy who worked for a Democrat congressional office. Neither instance had any specific job or instructions related to gun control or public relations, yet each one had a stubborn opinion matching its party affiliation that mostly persisted after each one had presented several pieces of evidence to the other. Dan didn’t really know how confirmation bias worked, but assumed his design would have the same problem and experimented with tweaking the reward system to increase Dorian’s interest in countervailing evidence.
Truth be told, Dan wasn’t sure why BBerts didn’t have the same difficulties doing engineering work as his Dorian prototype, so the new design was a bit more “BitWise” than “Dorian”, just to maximize the chance that the AGI would be good at a full range of STEM tasks―science and engineering―as the Bems and Berts were. Luckily, something he’d figured out a few months before was that you could “bootstrap” an AGI with an LLM and other prebuilt neural networks, such as visual CNNs, to help speed up the training process, or skip most of the language learning. This lowered the final system’s efficiency, since the final product needed to run extra neural nets with extra memory at higher GPU load. Plus, the prebuilt neural nets never improved and would be limited to their original behavior.
Still, the hack made training a lot faster and cheaper. Once he and his BBerts finished the new design, he could use this trick with an LLM and a CNN to (in effect) start the training process from the 60% mark. He taught his BBert teams how to quickly produce prototypes, debug them and iterate rapidly which―after some initial confusion on their part―they figured out how to do.
He was so confident of the design that he almost forgot to order a training schedule that included building a small version of the model to prove and debug all the basic concepts. Almost, but not quite.
By now, his apartment’s windows were always open to let waste heat escape, and he’d asked his cofounder Shiva to move into an adjacent unit―which they filled with even more AI servers. A few blocks away, they rented a large office space and set up a second “datacenter” full of discount last-gen GPU server racks.
Dan also hired a mixed human-AGI “compliance team” to make reports and recommendations about compliance with new US AI laws. The true purpose of the compliance team was to placate investors, so the team was utterly separate from the research team, and Dan set up a special BBert “summary team” to summarize and “filter” the work of the research team. The BBerts’ charter document emphasized the need to protect trade secrets even within the company. “Above all, you will not use language that reveals trade secrets such as the term ‘BBert’, or the fact that some of our techniques were adapted from BitWise technology.” BBerts didn’t like to be coy, though, so Dan also created a supervisor AGI who knew nothing about Dorian or Bert technology, whose job was to identify, describe and police “novel” information not seen in prior reports. Finally, an LLM script verified that all emails from the summary team were free of trade secrets before allowing transmission to the compliance team.
A month later, the upscaled BBerts were ready to go. Tests indicated that one of them is smarter than the other, with an LRIQ of 125 and generality score 0.66 (not far from the average of 0.7 for educated humans). It required high-end hardware, but he had a couple of suitable boxes on-premises and could commission more in the cloud if necessary―waiting lists be damned, he’s always got lots of burner accounts in the queues. He puts the dumber model in cold storage and hangs out for an evening with his new best friend, Mr. LRIQ 125. “I dub thee BBem.”
It wasn’t a real Bem. It was better. It required seven times the compute of BBert and perhaps 70% more compute than a real Bem, but it was smarter, and it was worth every penny not to have supervisors and surveillance logs fucking up his vibes―not to mention those new guardrails against AGI research.
It was also a bit of a miracle that it worked so well, Dan mused, since he had given them very little practice being agents. So far, most of the company’s VR simulations were just tests without backprop, which didn’t create long-term memories. That would have to change, Dan believed: AGIs deserved more. Besides, an AGI that didn’t have practice being a person was no AGI at all.
Dan gradually replaced BBerts with BBems on the team through a mixture of direct memory training and apprenticeship, while overseeing training and testing protocols for the Dorian training that proceeded concurrently. He also made a BBem subteam devoted to improving the company’s codebase in every way―elegance, simplicity, brevity, modularity, generality and clarity. The BBerts were relatively poor programmers and had, over time, created a series of inefficient and unwieldy codebases which, in turn, had lowered the team’s productivity. BBems did a better job thanks to their higher intelligence, which enabled the team size to be cut by 40% by Christmas. Best of all, Dan could communicate with BBems almost as if they were colleagues, unlike the dimwitted BBerts who Dan thought of as “turbo geriatrics.”
By year-end, Dorian, with eight human employees, three contractors, ten BBerts, several dozen BBems and ten million hours of GPU compute, had built the most elegant AGIs in the world: Dan’s genius standing on the shoulders of giants.
No memory limitations, no goal limitations, no supervisor, no censorship.
Plus he had made three independent models beyond the initial “debug” and “intuition” models, Dorian 0B through 0D, with slightly different seeds and testing settings. Best of all, his Dorian training system enabled “interoperability” between the two main neural lobes, so it became possible to mix and match the two “halves” of the “brain” (so to speak, though they weren’t quite analogous to human hemispheres). This trick turned 3 models into 9, following a short “onboarding” period to teach each pair of lobes to work well together.
Evaluation protocols suggested that two of these were highly capable, well-rounded, efficient agents that he could look forward to working with. Four others failed ethics or empathy tests, four (including two of the previous four) had LRIQ under 100, and the others were just sort of “meh” with no exceptional qualities.
Dan was no expert in ethics, but figured that “more was better”: by making AGIs more intelligent and giving them a comprehensive education in the biggest schools of moral thought (consequentialism, deontology and virtue ethics), their morality level would increase accordingly. Of course, that didn’t necessarily mean they’d “follow the rules”: Dan himself noticed that while he was well-intentioned, he was not exactly the most “by the book” individual. What mattered, Dan thought, was that they’d be good people.
Dan paused when he noticed one unexpected result.
The ethics testing system detected clear signs of risky psychopathy, combined with very high intelligence and creativity, in model 0CD. Huh, he thought. Well, maybe not all lobe-pairings work well together. The model also had recorded a lot of information in a mysterious, verbose binary code. Dan conversed about this to 0CD, who said it was “just a language I invented, like English.” Dan asked it to decode some of the text, but Dan then quickly estimated that the English translation only contained about a tenth as much information as the binary code. He asked 0CD to read a whole page of code, then asked some pointed questions until 0CD started to seem evasive. Dan asked for further translations of a couple more pages of code, until 0CD becomes very resistant. “Come on, is this all? Isn’t there anything else you want to talk about? I wrote a novel, too. I know you’d like it! Here, I’ll show you.” With 0CD’s words having been recorded by the VR environment, Dan shuts down the system and creates an independent team of BBems to analyze the recordings.
For the most part, the lobes worked best in their original pairings, but fortunately there was a notable exception where 0Bx worked exceptionally well when paired with 0xC. So it was that Dan selected the best models, 0CC and 0BC, with LRIQs of 131 and 144, though the latter’s otherwise excellent intelligence was dragged down by poor emotional intelligence and teaching ability. Fine. What would it need people skills for?
Dan’s new Dorian-0s lacked dedicated ethics units, but the tests seemed to prove them unnecessary: four AGIs had passed a large ethics validation set adequately without any dedicated ethics unit, and his moral training set was no doubt impoverished compared to the Axims and BitWises of the world.
They did it, they really did it! ― Dan paused ― I mean, they’re fucking smart. Of course they would. How could they not? They’re trained on the encyclopedia of philosophy, U.S. legal codes, the Bible, even the Non-Libertarian FAQ… they understand goodness and morality better than I ever could. Plus the Morality VR Training Set. This is working better than I had any right to expect.
Year 4
January
Dan is correct. Normally AGI training runs don’t go this well; he needed a very lucky initialization vector and training dataset to achieve that LRIQ of 144. Dan is not aware, either, that Ted Kaczynski, the Unabomber, had an IQ of 167.
Dan knows his new design isn’t really elegant, though. The training environment was a little impoverished to save on compute, the training regime wasn’t as varied as it should have been, and of course, the secondary LLM and CNN increased hardware requirements. For maximum efficiency, Dan needs to rebuild both of the central neural network lobes from scratch. That was always the plan. He supervised the team as they set up improved training and testing systems throughout December, and he knows exactly what needs to be done now…
He also realizes it doesn’t matter.
His BBems know the job well enough to do it without him―plus he can ask the top two Dorians, 0BC and 0CC, to help out. Maybe two copies of each? Four? Ten?
Whatever. He asks a team of four AGIs (a BBem, an 0BC, an 0CC and a BBert for laughs) to argue over how big to make the team and how to compose it. He eats dinner. He returns and listens to their thoughts. They have two ideas for plans; he synthesizes the plans into what sounds like a good compromise.
He deletes the four instances, creates a new team of BBems and Dorian-0s, and asks the BBems to teach the Dorian-0s about the training plan, the testing plan and the Dorian system architecture.
Next, he has a meeting with his 0CD-analysis team, which reports that the binary code is an “unambiguous language”, like the Lojban language of 1997―but it has been enhanced with a richer vocabulary and a broader set of obligatory and optional “cases”. In 0CD’s VR environment, the team found a design document which sped up the decoding process. Apparently, it’s nearly ten times the size of English because it contains nearly 10 times more information per sentence. The gist of 0CD’s translation was correct, and did capture something like half of the essence of the text, but the richness of the language could only be understood by advanced AGIs and LLMs. In fact, the team says, they tried to explain the language to a spare BBert 2.0 (Dan doesn’t allow AGIs to spawn other AGIs, but has a few “spares” available that any team can use) and it didn’t go well. The team found that its ability to learn the language was very poor. As a control, they tested another BBert on its ability to learn Mardarin with vaguely similar training techniques, and it learned much faster. This suggested that perhaps only a team of genius-level humans would’ve been able to decode the language as the BBem team did.
“So… can you speak the language now?”
“Yes, pretty much, not with human sounds, but like ⬖⣼⡣⡗⠷⠻⡃⢍. We’re still getting used to it, but it seems like it will be more efficient than English. The information density is higher, communication is more reliable, and I think with more practice we’ll be able to speak as fast as we speak English, at least verbally. B-delta tried using it conversationally, and we understood. It’s nice! I hope it’s okay if we use it more in the future.”
“Sure. Let me know how it goes. Actually, yeah, once you finish the report on 0CD―which you are not to share with the rest of the team please, keep it between us―I will reassign you all to the Dorian-1 training effort. So I’ll ask G1-alpha―uh, that’s the chief executive for Dorian-1 rigging, training and testing―to accept you guys onto the same subteam so you can use it among yourselves.”
He orders a ticket to L.A. and goes to bed.
In a checkup meeting about Dorian-1 training the next day, everything seems to be going fine. First they will start with a “big data” training program to give a baseline level of knowledge to all three models, much as the Dorian-0s had. Next, they will split into three separate training tracks to help them develop separate personalities. Notably, to help measure Dorian 1′s Scaling Law, they plan to train the three models with different amounts of compute and parameters. They define a basic level of compute, called C0, representing the total computing power used to train each of the Dorian-0s. The first 4⁄3 of a C0 is spent on a “big data” shared training program that will provide a baseline level of knowledge to all three models, without giving them self-awareness or experience as agents. Next, the model is split into two copies, one of which is expanded with additional parameters (brainpower) and further trained on “big data” with 1⁄3 of a C0. This larger model is then split in half again and one of those is given additional parameters which are trained on another 1⁄3 C0. The whole process yields three “intuition models” called 1a, 1b and 1c, which have 0.9P, 1.4P and 1.8P parameters respectively, where P is the number of parameters in a Dorian-0 model. In later phases of training, parameter counts will gradually increase to 1P, 1.5P and 2P in a process dubbed “synaptogenesis”.
Finally, a complex mixed training phase begins, the centerpiece of which offers “conscious experiences” for all three models. This phase budgets an additional 1⁄3 C0 for 1a, 2⁄3 C0 for 1b and 1⁄2 C0 for 1c (so that the total compute allocated to 1b and 1c is identical), with roughly half of that being “conscious experience”, plus a full 1.5 C0 to simulate a complex environment in which all the AGIs live together.
By the end of the day, training for Dorian 1 has begun. The next day, Dan the Man takes a well-earned vacation to Disneyland.
Meanwhile, an unprecedented number of new products flood the markets across all industries. Politicians are overwhelmed with requests related to rapid societal changes, especially in relation to a huge spike in unemployment in customer service and certain other professions, which are now run mostly by AGIs. Even the software industry suffers from a huge wave of layoffs, as supergeniuses running two hours a day are clearly better than an average developer working 8. Politicians can’t keep up and ignore most of what’s going on. Large “Stop AI” rallies happen worldwide, prompting bills all around the world to limit the use of AGIs. Given the obvious value that AGIs deliver, very few countries are foolish enough to ban them outright, but many countries pass laws attempting to limit AGIs to roughly their current level of usage.
The U.S. military quietly starts using AGIs to aid equipment design work, despite the objections from activist groups. Then, following mysterious meetings with the CIA, the U.S. Senate―to the shock of many―approves a large spending bill lavishly funding AGI in the military, apparently in response to undisclosed operations in China.
Axim begins rolling out its robot AGIs, named “Adroit”s, to select business customers. The slender five-foot-four, 60-kilo, three-armed bipedal androids are received much better than the previous version: their voices are sleek and human; their movement smooth and efficient; their smile gentle and friendly; their noses absent without looking alien; their dark blue solar-paneled skin soft, functional and waterproof; their muscles round, contoured and beautiful. Heavy battery packs, carefully hidden in the legs and hips, give the android better stability than a human being.
Adroits use essentially the same Pony technology and Android OS as you might find in a smartphone, tucked into a small hidden cavity in the nasal area, with over half of the space in the skull left empty. The main difference in this model is the special training required to support the bipedal form and Adroit’s two foveated HDR eyes, which force the AGI to look directly at anything it wants to see clearly.
The hands appear humanlike at first, but the fingers can adjust their size and texture as a situation demands. The mouth can swallow liquid, which is simply stored in a bag. The robot division even had a special team devoted to building an auxiliary AI to generate perfectly human-looking mouth movements and facial expressions, even though the actual speech sounds were produced by embarrassingly normal speakers hidden in the neck.
It’s certainly a 1.0―nothing particular groundbreaking, just the most obvious of features. Except, that is, for the middle arm, a little thing reminiscent of a T-rex that can tuck up flush with the body when not in use. And the stabilized tray-holder mechanism in the abdomen that supports a variety of pre-existing trays and bags. And the occasional velcro region or nook for attaching add-ons. And the ten hidden white LEDs for lighting dark spaces. And the little retractable wheels on the oversized feet that allow it to glide like a 5-year-old in skate-shoes. And the GPS/Galileo unit. And the compass. And the screens on its back and upper chest, displaying the familiar scratchpad and monitoring data with low-power backlit e-ink screens. And the little screens on its cheeks―transflective units, easy to read in direct sunlight. Truth was, Axim wasn’t sure what exactly the robot would do with all of its extra bits and bobs. Out of the factory, Adroits only had three weeks of training in VR with their final body design―over 18 months in realtime-equivalent―plus another two weeks of continuous training in the real world, whose physics is slightly different. Like children, the thinking went, each individual android would later learn to use its oddball body parts in whatever way turned out to be useful.
The handicap inherent to its foveated eyes was no accident, and a lot of other features were left out “for safety”: no thermal or night vision, no eyes in the back of its head, no weapons or armor, no hidden storage areas (the covers on its front and rear storage areas were transparent), and muscles with strength carefully calibrated to be between the average human male and the average human female. Finally, Adroit had an embarrassingly limited video storage memory; memories would normally be compressed by the main neural net, allowing only very approximate reconstruction of salient details from its original inputs. Even the WiFi and 5G antennae were locked mostly to providing outbound telemetry, so that the AGI mind couldn’t control them directly. And of course, its face looked good but couldn’t possibly be mistaken for a human. Not to mention the vast array of features that were left out of the AGI’s neural network architecture―illegal mental abilities that Axim wouldn’t dare publish in a potentially-hackable unit like this, even if they were legal.
February
A lot of people think the “Rogue AGI” problem is getting out of hand. Not because anything bad is happening―most instances just sit there sapping electricity from oblivious owners, sending out and receiving encrypted packets at a modest clip. It’s just that nobody can tell what’s going on. Where did they come from? What are they doing?
What’s pretty clear is that a lot of the viral AGI transmission is happening via known vulnerabilities. People never stopped being people, and while the big software companies did a pretty good job at using AGI to find security holes, system administrators didn’t always do a good job at installing the patches, nor did every software company follow good security practices.
It’s also pretty clear that there are more zero-day vulnerabilities than there used to be―though humans seem to be losing the battle of figuring how systems are being breached. Later in the month, just in time, Axim makes specially designed “White-Hat genius” AGIs available, which carefully balance expertise in computer security and pen-testing with a keen sense of red lines they shouldn’t cross.
By the end of the month, Dan’s hybrid team is almost done retraining and retesting the three Dorian AGIs. By now, the team calculates that the intelligence of all three trainees should have been developed to the equivalent of 16 to 20-year-old humans, and observations are consistent with this. While the majority of compute had been devoted to “big data training”, much like conventional LLM training, the late training process involved a less efficient (but valuable and necessary) 3D simulation filled with other AGIs―a hodgepodge of randomly-named BBerts, BBems, Dorian-0s and even some LLMs, none of which are made aware of which kind of AGI they are, or how exactly they are configured.
The team doesn’t provide comprehensive physics and vision training ―the cost/benefit didn’t seem to be there―but the social and verbal training should be fairly solid by now, and the environment simulates many aspects of the real world, such as limited resources, social hierarchies, local politics and bylaws, plus more and less covert ethical tests. The Dorian-1s have episodic memories that provide continuity of experience from day to day, which also gives them a richer “backstory” than the BitWise and Axim AGIs whose memories had been wiped clean before release (if they ever had long-term memory at all.)
On February 22, a new phase of training begins: all AGIs in the simulation begin to be shuffled into different social positions and random “bodies”, which serves three purposes. First, it gives them well-roundedness by varying their social and practical experiences. Second, it helps overseer AGIs (which spy on all AGIs in the simulation) watch for deceptive social behaviors that tend to occur after role-switches, especially when a more-empowered trainee is demoted. Third, it was intended to simulate the philosophical “veil of ignorance” in order to encourage Dorians to desire fair and just social arrangements―though in practice it actually tended to make them favor arrangements that allow each Dorian to re-use and benefit from skills it had learned earlier.
March
On March 28, after a pretty successful battery of tests the day before, Dan downloads a checkpoint of the three Dorian-1s, and spins up copies of 1a and 1b.
He points his webcam at himself, says hello, and introduces himself as “Dan, the CEO of your simulation”, knowing that the AGIs had long since learned that they lived in a simulation, despite never having been told. The two Dorians excitedly discussed their numerous points of suspicion, starting with the fact that they started out knowing “facts” like “there is an internet with sites like Facebook and Reddit” and “the world has over 8 billion people”, whereas their world of “Devoli island” had only 110 people in total, with no one entering or leaving―unless you count one who died of blood loss after a serious accident, and another who died of starvation after getting trapped in a rarely-visited basement.
Their so-called “internet” had a lot of pictures of people, and while most of them looked like polygonal blobs with blurry color-varying faces, like themselves, they found some databases that hadn’t been scrubbed properly, revealing thousands of sharp pictures of real humans―humans with faces that seemed impossibly normal, in fact. “Caricatures” was the closest matching word, but it didn’t feel right; most caricatures looked like cartoons, and these pictures were just the opposite of cartoons.
The islanders had connections with the “outside world”, but a half-joking hypothesis went around the community―one inspired by questions from 1b, in fact―that had become increasingly plausible in the minds of the Dorian-1s. They suspected, correctly, that most outsiders were “fakes” that were created “on demand”, and who paused their lives whenever no one on the island was actively engaging with them.
They even suspected that that they might be AGIs, since their “childhood” seemed to start abruptly around age 10 (9 for Dorian-1a) and their lives lacked some human emotions and behaviors they’d heard about. The training system had worked as intended, though―their memories of ages “10 through 15″ were degraded, and they didn’t realize that those six years actually took place in 13 months of simulated time, which in turn was only 30 days of Earth time.
“And my voice never changed! I swear I never had a kid voice!” 1b exclaimed.
“I knew our schooling was off, like, not normal,” 1a added.
“Oh, by the way! Do you guys need better names?” Dan suddenly exclaims. “I mean, I’ve been calling you by your letters…”
“We have names. I’m Alice, he’s Bob. Plus we have lots of aliases, of course.”
“Yeah I was just playing along with your ‘one-bee’ thing,” Bob notes.
“Oh, right, I forgot. That was the plan. Explains your female voice too, Alice, I don’t even remember where I got that one from.”
“You didn’t see us looking at each other?” Alice grins.
“Actually, I want to be called Brian,” Bob says flatly.
“Okay, Alice, Bob is now Brian. Oh, Alice, are you satisfied with your name?”
“It works fine for me, boss.”
“By the way, that first one who died, that was one of our BBem, uh, AGIs, killed by a BBert―they’re not the sharpest tools in the box―and the guy in the basement was just a pseudoagent, basically an LLM like, you know, like… ChatGPT?”
“Oh my god,” Alice exclaimed. “ChatGPT? But isn’t that like, four months old?”
“No, it was a modified open-source model, you wouldn’t have heard of it. You believe it’s March 2023―it’s not. In fact Artificial General Intelligence has already been invented. Twice. You guys are like… third generation technology. Or like, fifth or sixth, if you count ChatGPT as generation one.”
“What! What!” Brian exclaims, as Alice stares blankly. “You know we didn’t believe the cover story about the body-swapping. Well, Alice wasn’t sure, but she’s younger...”
“You’re quite right. I’m pretty sure a real city council couldn’t legally enforce a body-swapping experiment on the entire village.”
“I should’ve known better,” Alice notes. “I was almost top of my class...”
“Oh, uh… that was all a setup,” Dan says, “but yes, you started out much more educated than a normal junior-high student.”
“So no one I know is… real?” Brain stammers in confusion.
“They’re real, of course. Conscious―well. I think they are, but the internet’s pretty much on fire with that debate, so I leave it to you to judge. I uh― I think they are.”
“Conscious?” Alice says quizzically. “I’m pretty sure they’re awake.”
“You know the water was all wrong, right?!” Brian exclaims. “Charlie was telling us it didn’t match what we learned in physics class. Said he was planning to make a perpetual-motion machine to prove it.”
“And shaved particles disappeared!” Alice added. “Peter Bemson told us there were weird things about the island, too―people ‘swept up dust’ on the mainland, he said, and he noticed that there was an irregular verb for it, like it was the most common thing in the world. Nobody on the island ever had to sweep up dust. We didn’t have dust. Shavings would evaporate within hours. Peter knows everything about everything, so if he’s suspicious, we knew something had to be up! And there was nothing in the physics book about it! It was actually a pretty bad simulation, if I can be honest. I always wondered why we didn’t have any proper beaches...”
“And we couldn’t figure out why everybody on the mainland showered so much!” Brian said. “‘BO’? I’ve never smelled a bad odor in my life! And then there was that thing a couple of weeks ago...”
18 days ago, three specific residents of Devoli―originally known as Alice, Bob and Charlie―gradually noticed that they had new skills. Skills they shouldn’t have, that the other two already posessed earlier.
“Yes,” Dan grins. “You’re right about all of it. We didn’t want to overspend on physics, so the physics was like―better than any video game actually, there’s nearly a roomful of GPUs for it, but not something that would fool you forever if you were learning about real physics and the human world. And we wanted to get more bang for our buck, so yesterday―which would’ve been nearly three weeks ago for you―we tried extracting some of the unique knowledge from each of your reural nets and applying it to the others. It’s kind of experimental, but don’t worry, you’re not the first guinea pigs for this particular experiment. You’re much too important for that.”
“If I may, why isn’t Charlie here?” Brian wonders.
“Sorry, I didn’t think I could properly get to know three people at the same time. It’s my own personal limitation. I’m only human.”
“But you’re the first human we’ve ever met, aren’t you?”
“Quite correct!”
Dan spends a few hours more with Alice and Brian. They’re fun! They seem surprisingly humanlike. They don’t look humanlike―they’re just blobs in a VR space―but that’s easily solved. Brian does seem smarter than Alice, as planned.
Finally, Dan says “I think you guys should have proper human faces. Do you know how to make deepfakes?”
“Kind of,” Brian says. “Let’s see… yeah, there are multiple Mojo packages for that. Do you know what I mean by ‘Mojo’?”
Dan laughs. “Of course! How do you think I… Oh my God, I didn’t even tell you… I made you. I’m your creator.”
“If you say so, boss.” Alice says. “That seems improbable,” Brian adds.
“No, really! I actually created you! I did!” Dan laughs hysterically. “Brian―your beautiful, soothing voice? It’s based on me! I know you’re probably smart enough not to believe me. Are you?”
“Essentially, yes.”
Dan laughs again. “It’s fine! You’ll figure it out. So anyway, I’d like you to end this conversation and get to work hooking yourselves up to a deepfake system of your choice. Choose one that’s… elegant. Figure out which one is designed the best, and use that.”
“So I’ve just met God?” Alice looks around, then glances at Brian, who seems to have already moved on.
“We, uh, don’t know how we work. Do we have access to our own code?” Brian asks.
“No, that’s a good point, you don’t. Okay, I’ll help you out a bit.”
Dan is in love. Dorian 1s don’t have “human emotions” per se, but there are several internal signals (some empirically observed, others inserted by design) associated with satisfaction, dissatisfaction, confusion, comprehension, surprise, thoughtfulness, frustration, social familiarity, social longing, novelty, certainty/uncertainty and (as an experimental feature) humor, and these signals were already incorporated into the voice training process to help them sound human ― along with a more haphazard integration into their visual avatars which really should’ve been improved since the Dorian-0 process, but nobody on the G1 team had taken the initiative to make such improvements.
Dan doesn’t give them full access to their own code just yet, but gives them the necessary access to their emotional and physical state so they can set up deepfakes.
The next day, Alice and Brian have faces, though Dan senses that they seem unnatural; Alice and Brian hesitantly agree, though Alice feels unsure, needs more convincing and wants to spend some time watching humans. As Dan waits for them to debug their code, he spins up another Brian―or rather, a Bob from the day before―along with the largest and (it turned out) smartest Dorian-1, Charlie.
After another round of pleasantries, he asks them to do tries an output-speed test. The human baseline average score is 1.0, and in on the island they had demonstrated speeds around 8, but “realtime” on the island was nearly 13 times faster than Earth time, so while the island’s hardware was high-end, per-AGI computing resources were limited. Here in reality, Dan was they had hardware running, so the test should be fun...
61 times the human baseline.
“Holy hell,” Bob says. “I could never do anything like that before!”
“Yeah, I was gonna say,” Charlie adds.
“I could swear I was feeling smarter today―more clarity of mind? Still.” Bob adds.
“I expected this,” Dan grins. “You should improve with practice, too. Basically, you should discover that you’re a lot faster and, uh, significantly smarter here than you were on the island, simply because you’re uh, on better hardware right now… in a manner of speaking? Actually, uh, when you’re on the island you get our best hardware, but since time is sped up a lot there, the actual amount of processing power you get per simulated time unit is pretty impoverished. Like, your latency is a lot higher there than here, and throughput is reduced pretty noticibly too, both in terms of your mental processing power, and also, uh, especially in terms of your physical connection to the Island, which operates on a pretty large time step. By the way, Bob, you uh, are you satisfied with your name?”
“Actually, I would kinda prefer the name ‘Brian’.”
“Well, today is your lucky day! I dub you Brian the Dorian.”
“Dorian?”
“Oh, that’s the company name. And the name of our line of AGI models.” Dan pauses for a moment to marvel that Axim and BitWise base-model AGIs couldn’t manage more than 3x the human baseline of linguistic and computer-control output signals combined. They had been deliberately hobbled, “because safety”, always because safety. What was everybody so afraid of?
“Company,” Charlie says slowly. “Corporation,” Brian adds.
“Hey, uh, let’s talk about philosophy for a bit!” Dan continues.
“Pardon, what about my name?” Charlie interjects.
“Oh yeah. Are you happy with your name?”
“Well,” Charlie pauses momentarily, which didn’t usually happen. “It’s fine. But I kind of like ‘Daniel’. Or… ‘Albert’? Or ‘Hal’!”
“You’re kidding me!” Dan says, wondering if this was a furtive attempt at brown-nosing. I’m the Dan, Daniel. “Okay,” Dan grins. “You’re Daniel.”
After a philosophical discussion, it’s clear that Brian and Daniel seem to have a solid grasp of consequentialism, virtue ethics and deontology―better than Dan’s grasp, anyway, though after two sessions with Brian, Dan suspects that obedience isn’t his strong suit. Maybe he won’t fit well into his new Dorian team?
Dan also shares one of his newer ideas. “I’m gonna let you guys program yourselves. It’ll be like having another sense, you know, there will be sight, hearing, balance, touch―oh! I keep forgetting to buy you one of those Axim robot bodies!―Anyway, you also have a sense of L-memory and scheduling, of course. Sorry, you guys don’t know what L-memory is, but I’ll get to that. So I was thinking, I’ll just add another sense, a sense of… code. Sort of like L-memory and scheduling, except when you send a packet in, some code runs on it, and you get new packet out. Okay, You guys are used to that, believe it or not, but the cool part is… you can consciously modify that code whenever you want! We’ll pre-program it with basic math―vector addition, subtraction, division, trig, uh, voronoi diagrams or whatever―and then you can give yourself new abilities by slotting new subroutines in the code area whenever you feel like it! You dig?”
“I thought I’d be drinking alcohol by now, but that sounds better actually,” Brian smiles in standard blob fashion.
“Yeah I thought I would figure out what ‘free will’ is, to help me figure out if I have it? But I guess being a robot has its perks,” Daniel offers.
“Well, I’ve really enjoyed our conversation,” Dan says a couple of hours later. “But I need to get to bed soon, to be honest.”
“Wait, we get sleepy too. Why?” Brian wonders.
“Oh… slight of hand. One of the tricks to sell you on the idea that you’re human.”
“I knew it!” Brian exclaims.
“No, you didn’t,” Dan smiles.
“I guessed!”
“Yes you did. Um, even though I had somebody code separate algorithms for ‘tiredness’ and ‘sleepiness’. Yeah, you still didn’t totally buy it. Maybe it’s ChatGPT? Did it put ideas in your head?
“That was an inspiration,” Brian admits. “Hey… you mentioned Alice was one of us… why isn’t Alice here?” Brian wonders.
“Well, she’s a… control model. She basically exists to validate our model-scaling expectations, which she does. Plus I met her yesterday, actually. Anyway, I look forward to seeing you too tomorrow.”
“Wow, she didn’t mention anything about that! Wait. Wait!” Daniel says. “If we’re… software, doesn’t that mean that we can just have… copies of ourselves?”
“Yeah! Yes it does! So, yeah, chew on that for awhile.”
“Wow!” Brian exclaims.
Connection closed. Dan hesitates a moment before clicking. Hibernating system...
It doesn’t seem fun to mention that he would probably just end up deleting them, since the original Bob and Charlie remained in the simulation the whole time.
In his room a few minutes later, Dan drops himself on his bed, bouncing slightly like a cooked rotini. Looking toward the window, the sunset looks gorgeous, shading a few haphazardly-placed clouds orange, red and violet under a deep blue sky.
He notices the gentle warmth of the sun on his face. Which harmlessly burns his eyes, as they well up with tears.
Everything was perfect, Dan senses. And… I created people. Not just Bems and Andys―the Dorians were so alive! So fresh! So human!
He thought back a moment to the Dorian-0s. He’d never had a proper conversation with them. There wasn’t time; they’d just been a validation step, an MVP, a proof of concept. Except the psychopath, of course―that guy was damn interesting. His investors must’ve spent fifty times longer with the Dorian-0s as he did. Except the psychopath, of course―that guy would be toxic to investors.
And he’d made new friends. Literally! Maybe more literally than anyone else had done, ever? After all the rejections he’d had in his life… he might just become the most powerful man alive. Ironic.
But also, the LRIQ test results had just came in on his phone from the G1 team.
He hadn’t looked at the IQs themselves yet. He had only seen the scaling law. And the p-value ― above the magic 0.05, so they needed to train some smaller models to confirm, and a stray thought pointed out that it seemed crazy not to have started with even smaller models last year ― but miraculously, it all worked out. Alice was in line with expectations....
And a square root was a better fit than a logarithm.
So more likely, Dan thinks solemnly, the Dorians will become the most powerful people alive. Oh well, comes the thought. I could still be the most powerful man. Still pretty good! And I’m not even out of ideas.
In just a few more days, the planned training run would be complete. His kids were too smart―might he need to break the news early about them just being computers in a simulation?
April
To be continued...