A second thing that seems worrisome is the suggestion that CPU power might be being artificially throttled; that someone has enough power (political or economic) to sabotage any effort to produce a faster CPU and can exercise that power without being noticed would be the stuff of conspiracy theories.
Which is … a bad thing?
I for one would be very reassured to learn there was someone in our world looking after this stuff. Heck, civilizational incompetence is my main counterargument to conspiracy theories.
I know that they’re probably human, probably more competent than our current observed rulers (and thus are rational enough to realize they needed to prevent X-risks, at the very least, and likely other Important Things), and are currently engaged in saving the world.
So I don’t actually know nothing about their goals and values, and what I do know seems promising. Heck, if we could actually pull it off, I can see someone with my exact goals and values behaving in precisely this manner.
Of course, they might be aliens. They might actually be quite incompetent and easily uncovered, having succeeded through luck and collective apathy. And they might have some amazingly stupid other reason for suppressing dangerous technological advancements. But the prior seems relatively low.
If all you know is that some unknown entity holds back technological progress, your conclusion seem unjustified. This entity might as well be preparing the humanity for a culling and doesn’t want to prey to kick too much.
Lumifer’s got most of the disquieting part; I know nothing of the values or goals of the people behind the CPU throttling. I also know nothing of their methods, aside from that they are effective; the character only noticed the CPU throttling after having seen our universe and comparing.
So what do they do if John Smith invents a new, faster CPU, all unknowing of any trouble? Does John Smith suddenly vanish mysteriously one day, reduced to nothing more than a newspaper headline, a few disinterested search parties, and some grieving relatives? Does he turn up suddenly cryopreserved in some facility, with all his notes lost in a mysterious fire?
Or does someone just give him a call one day and say ‘Please don’t do that’? And what if he’s stubborn, and goes ahead with it anyway?
If I visit dath ilan, what other apparently helpful or innocuous action(s) might get me in similar trouble?
And why are the CPUs being throttled, in any case? The story suggests it may be an anti-UFAI measure, but one would observe the same results if it were led by, say, a greedy CPU manufacturer who sabotages everyone who makes a better CPU than he can.
I don’t have any problem, in theory, with the idea of CPU power being throttled. What bothers me is the secrecy around the throttling; if the throttling was widely known, and the reasons were also widely known, then I wouldn’t be nearly as disquieted by it. (I might or might not disagree with the reasons, of course, but that’s seperate).
More specialized, highly-trained professionals, stronger formal oversight and central planning, easier choice of neighbors, and no economic room for amateurs to compete, all of which means less informal interdisciplinary coordination.
If John Smith in dath ilan invents a new, faster CPU and tries to make money with it, he does so by checking around for existing prizes, not by doing the marketing himself. Whoever’s offering the prize is, presumably, in on the conspiracy, and sets up conditions for collecting the money which bring John in on it too. If he doesn’t try to make money? Nobody cares, outside maybe his immediate circle of friends.
Professional chip designers, as individuals who share a common interest, would all live together and know each other. If some conspiracy can be made to appeal to most of them, dissenters face ostracism and loss of formal qualifications, possibly starting over in a completely new career which decades of ultraspecialized training would not have adequately prepared them for. Adapting to those circumstances would be virtuous, of course, but that doesn’t make it easy, and obsessively rapid iteration means low tolerance for stubborn people in general.
If John Smith in dath ilan invents a new, faster CPU and tries to make money with it, he does so by checking around for existing prizes, not by doing the marketing himself.
Why not create his new CPU, and then offer his own prize for someone to market it, instead?
If he doesn’t try to make money?
Arguably, that could be worse. What if he makes the designs widely available at no cost? You could have a dozen unrelated small groups around the world start manufacturing the new chips within a week.
Professional chip designers, as individuals who share a common interest, would all live together and know each other.
That is true. It is quite possible that the CPU throttling is a conspiracy among professionals. However, the fact that the CPUs are being throttled leaves a clear opening for a gifted amateur—one not (yet?) inducted in the conspiracy—to create a design that grabs the low-hanging fruit left purposely untouched by the professionals.
It also leaves an opening for one of the professionals to defect, and try to gain income and fame by creating a faster CPU despite being a member of the conspiracy.
Why would there be a dozen unrelated chip-fabricating facilities sitting idle in different corners of the world, when everyone lives in the same city, every group with shared interests carves out their own neighborhood, and there’s an overwhelming social norm to coordinate and standardize for increased efficiency? If there aren’t such factories, whoever tries to build one will attract the conspiracy’s attention during the permit-application process. If the factories exist but aren’t idle, newcomers will be competing with better-coordinated conspiracy members for production time before they can even test prototypes.
If an actual shutout fails, microtech in production seems like it would be absurdly easy to sabotage. Contaminate the raw materials, introduce a single-bit error into quality control so it rejects flawless chips and passes garbage, tweak the blueprints and/or assembly mechanisms so layers are misaligned, overheat it to denature the semiconductors. Overseer assesses the probability that a whole sequence of factory techs each made some grossly negligent or malicious mistake with this one model of chip, but no others before or since, concludes that the design itself is flawed.
Rapid iteration means the design is then thrown out, and the designers bluntly exhorted to either learn from those with a proven record (conspirators, all) or move on to some line of work that better suits their talents.
What bothers me is the secrecy around the throttling; if the throttling was widely known, and the reasons were also widely known, then I wouldn’t be nearly as disquieted by it.
Keeping it secret from the humans is just a side effect of keeping it secret from the nascent AIs.
Because if they knew it was happening they would build their own faster CPUs. And they would do this in secret so they now had access to CPUs whos exponential growth in capacity was faster than the CPUs available to humans. And that would suck for humans who are trying to avoid powerful UAIs from developing.
Which is … a bad thing?
I for one would be very reassured to learn there was someone in our world looking after this stuff. Heck, civilizational incompetence is my main counterargument to conspiracy theories.
Given that you know nothing about this someone’s goals and values, I don’t see much reassurance here.
I know that they’re probably human, probably more competent than our current observed rulers (and thus are rational enough to realize they needed to prevent X-risks, at the very least, and likely other Important Things), and are currently engaged in saving the world.
So I don’t actually know nothing about their goals and values, and what I do know seems promising. Heck, if we could actually pull it off, I can see someone with my exact goals and values behaving in precisely this manner.
Of course, they might be aliens. They might actually be quite incompetent and easily uncovered, having succeeded through luck and collective apathy. And they might have some amazingly stupid other reason for suppressing dangerous technological advancements. But the prior seems relatively low.
If all you know is that some unknown entity holds back technological progress, your conclusion seem unjustified. This entity might as well be preparing the humanity for a culling and doesn’t want to prey to kick too much.
Lumifer’s got most of the disquieting part; I know nothing of the values or goals of the people behind the CPU throttling. I also know nothing of their methods, aside from that they are effective; the character only noticed the CPU throttling after having seen our universe and comparing.
So what do they do if John Smith invents a new, faster CPU, all unknowing of any trouble? Does John Smith suddenly vanish mysteriously one day, reduced to nothing more than a newspaper headline, a few disinterested search parties, and some grieving relatives? Does he turn up suddenly cryopreserved in some facility, with all his notes lost in a mysterious fire?
Or does someone just give him a call one day and say ‘Please don’t do that’? And what if he’s stubborn, and goes ahead with it anyway?
If I visit dath ilan, what other apparently helpful or innocuous action(s) might get me in similar trouble?
And why are the CPUs being throttled, in any case? The story suggests it may be an anti-UFAI measure, but one would observe the same results if it were led by, say, a greedy CPU manufacturer who sabotages everyone who makes a better CPU than he can.
I don’t have any problem, in theory, with the idea of CPU power being throttled. What bothers me is the secrecy around the throttling; if the throttling was widely known, and the reasons were also widely known, then I wouldn’t be nearly as disquieted by it. (I might or might not disagree with the reasons, of course, but that’s seperate).
More specialized, highly-trained professionals, stronger formal oversight and central planning, easier choice of neighbors, and no economic room for amateurs to compete, all of which means less informal interdisciplinary coordination.
If John Smith in dath ilan invents a new, faster CPU and tries to make money with it, he does so by checking around for existing prizes, not by doing the marketing himself. Whoever’s offering the prize is, presumably, in on the conspiracy, and sets up conditions for collecting the money which bring John in on it too. If he doesn’t try to make money? Nobody cares, outside maybe his immediate circle of friends.
Professional chip designers, as individuals who share a common interest, would all live together and know each other. If some conspiracy can be made to appeal to most of them, dissenters face ostracism and loss of formal qualifications, possibly starting over in a completely new career which decades of ultraspecialized training would not have adequately prepared them for. Adapting to those circumstances would be virtuous, of course, but that doesn’t make it easy, and obsessively rapid iteration means low tolerance for stubborn people in general.
Why not create his new CPU, and then offer his own prize for someone to market it, instead?
Arguably, that could be worse. What if he makes the designs widely available at no cost? You could have a dozen unrelated small groups around the world start manufacturing the new chips within a week.
That is true. It is quite possible that the CPU throttling is a conspiracy among professionals. However, the fact that the CPUs are being throttled leaves a clear opening for a gifted amateur—one not (yet?) inducted in the conspiracy—to create a design that grabs the low-hanging fruit left purposely untouched by the professionals.
It also leaves an opening for one of the professionals to defect, and try to gain income and fame by creating a faster CPU despite being a member of the conspiracy.
Why would there be a dozen unrelated chip-fabricating facilities sitting idle in different corners of the world, when everyone lives in the same city, every group with shared interests carves out their own neighborhood, and there’s an overwhelming social norm to coordinate and standardize for increased efficiency? If there aren’t such factories, whoever tries to build one will attract the conspiracy’s attention during the permit-application process. If the factories exist but aren’t idle, newcomers will be competing with better-coordinated conspiracy members for production time before they can even test prototypes.
If an actual shutout fails, microtech in production seems like it would be absurdly easy to sabotage. Contaminate the raw materials, introduce a single-bit error into quality control so it rejects flawless chips and passes garbage, tweak the blueprints and/or assembly mechanisms so layers are misaligned, overheat it to denature the semiconductors. Overseer assesses the probability that a whole sequence of factory techs each made some grossly negligent or malicious mistake with this one model of chip, but no others before or since, concludes that the design itself is flawed.
Rapid iteration means the design is then thrown out, and the designers bluntly exhorted to either learn from those with a proven record (conspirators, all) or move on to some line of work that better suits their talents.
Keeping it secret from the humans is just a side effect of keeping it secret from the nascent AIs.
Why is it necessary to keep it secret from the nascent AIs?
Because if they knew it was happening they would build their own faster CPUs. And they would do this in secret so they now had access to CPUs whos exponential growth in capacity was faster than the CPUs available to humans. And that would suck for humans who are trying to avoid powerful UAIs from developing.