You may live in a place where arguments about the color of the sky are really arguments about tax policy. I don’t think I live there? I’m reading your article saying “If Blue-Sky-ism is to stand a chance against the gravitational pull of Green-Sky-ism, it must offer more than talk of a redistributionist tax system” and thinking ”...what on earth...?”. This might be perceptive cultural insight about somewhere, but I do not understand the context. [This my guess as to why are you are being voted down]
robo
Karma: 280
You might be[1] overestimating the popularity of “they are playing god” in the same way you might overestimate the popularity of woke messaging. Loud moralizers aren’t normal people either. Messages that appeal to them won’t have the support you’d expect given their volume.
Compare, “It’s going to take your job, personally”. Could happen, maybe soon, for technophile programmers! Don’t count them out yet.
- ^
Not rhetorical—I really don’t know
- ^
Eliezer Yudkowsky wrote a story Kindness to Kin about aliens who love(?) their family members proportionally to the Hamilton’s “I’d lay down my life for two brothers or eight cousins” rule. It gives an idea to how alien it is.
Then again, Proto-Indo-European had detailed family words that correspond rather well to confidence of genetic kinship, so maybe it’s a cultural thing.
Sure, I think that’s a fair objection! Maybe, for a business, it may be worth paying the marginal security costs of giving 20 new people admin accounts, but for the federal government that security cost is too high. Is that what people are objecting to? I’m reading comments like this:
Yeah, that’s beyond unusual. It’s not even slightly normal. And it is in fact very coup-like behavior if you look at coups in other countries.
And, I just don’t think that’s the case. I think this is pretty-darn-usual and very normal in the management consulting / private equity world.
I don’t think foreign coups are a very good model for this? Coups don’t tend to start by bringing in data scientists.
What I’m finding weird is...this was the action people thought worrying enough to make it to the LessWrong discussion. Cutting red tape to unblock data scientists in cost-cutting shakeups—that sometimes works well! Assembling lists of all CIA officers and sending them emails, or trying to own the Gaza strip, or <take your pick>. I’m far mode on these, have less direct experience, but they seem much more worrying. Why did this make the threshold?
Huh, I came at this with the background of doing data analysis in large organizations and had a very different take.
You’re a data scientist. You want to analyze what this huge organization (US government) is spending its money on in concrete terms. That information is spread across 400 mutually incompatible ancient payment systems. I’m not sure if you’ve viscerally felt the frustration of being blocked, spending all your time trying to get permission to read from 5 incompatible systems, let alone 400. But it would take months or years.
Fortunately, your boss is exceptionally good at Getting Things Done. You tell him that there’s one system (BFS) that has all the data you need in one place. But BFS is protected by an army of bureaucrats, most of whom are named Florence, who are Very Particular, are Very Good at their job, Will Not let this system go down, Will Not let you potentially expose personally identifiably information by violating Section 3 subparagraph 2 of code 5, Will Not let you sweet talk her into bypassing the safety systems she has spent the past 30 years setting up to protect oh-just-$6.13 trillion from fraud, embezzlement, and abuse, and if you manage somehow manage to get around these barriers she will Stop You.
Your boss Gets Things Done and threatens Florence’s boss Mervin that if he does not give you absolutely all the permissions you ask for, Mervin will become the particular object of attention of two people named Elon Musk and Donald Trump.
You get absolutely all the permissions you want and go on with your day.
Ah, to have a boss like that!
EDIT TL/DR: I think this looks weirder in Far mode? Near mode (near to data science, not near government), giving outside consultant data scientists admin permissions for important databases does not seem weird or nefarious. It’s the sort of thing that happens when the data scientist’s boss is intimidatingly high in an organization, like the President/CEO hiring a management consultant.
Checking my understanding: for the case of training a neural network, would S be the parameters of the model (along with perhaps buffers/state like moment estimates in Adam)? And would the evolution of the state space be local in S space? In other words, for neural network training, would S be a good choice for H?
In a recurrent neural networks doing in-context learning, would S be something like the residual stream at a particular token?
I’ll conjecture the following in a VERY SPECULATIVE, inflammatory, riff-on-vibes statements:
Gradient descent solves problem in the complexity class P[1]. It is P-Complete.
Learning theory (and complexity theory) have for decades been pushing two analogous bad narratives about the weakness of gradient descent (and P).
These narratives dominate because it is easy prove impossibility results like “Problem X can’t be solved by gradient descent” (or “Problem Y is NP-Hard”). It’s academically fecund—it’s a subject aspiring academics can write a lot of papers about. Results about what gradient descent (and polynomial time) can’t do compose a fair portion of the academic canon
In practice, these impossible results are corner cases cases don’t actually come up. The “vibes” of these impossibility results run counter to the “vibes” of reality
Example, gradient descent solves most problems, even though it theoretically it gets trapped in local minima. (SAT is in practice fast to solve, even though in theory it’s theoretical computer science’s canonical Hard-Problem-You-Say-Is-Impossible-To-Solve-Quickly)
The vibe of reality is “local (greedy) algorithms usually work”
- ^
Stoner-vibes based reason: I’m guessing you can reduce a problem like Horn Satisfiability[2] to gradient descent. Horn Satisfiability is a P-compete problem—you can transform any polynomial-time decision problem in a Horn Satisfiability problem using a log-space transformation. Therefore, gradient descent is “at least as big as P” (P-hard). And I’m guessing you can your formalization of gradient descent in P as well (hence “P-Complete”). That would mean gradient descent is not be able to solve harder problems in e.g. NP unless P=NP
- ^
Horn Satisfiability is about finding true/false values that satisfy a bunch of logic clauses of the form . or (that second clause means “don’t set both and to true—at least one of them has to be false” ). In the algorithm for solving it, you figure out a variable that must be set to true or false, then propagate that information forward to other clauses. I bet you can do this with a loss function turning into a greedy search on a hypercube.
Thanks! I’m not a GPU expert either. The reason I want to spread the toll units inside GPU itself isn’t to turn the GPU off—it’s to stop replay attacks. If the toll thing is in a separate chip, then the toll unit must have some way to tell the GPU “GPU, you are cleared to run”. To hack the GPU, you just copy that “cleared to run” signal and send it to the GPU. The same “cleared to run” signal must always make the GPU work, unless there is something inside the GPU to make sure won’t accept the same “cleared to run” signal twice. That the point of the mechanism I outline—a way to make it so the same “cleared to run” signal for the GPU won’t work twice.
Bonus: Instead of writing the entire logic (challenge response and so on) in advance, I think it would be better to run actual code, but only if it’s signed (for example, by Nvidia), in which case they can send software updates with new creative limitations, and we don’t need to consider all our ideas (limit bandwidth? limit gps location?) in advance.
Hmm okay, but why do I let Nvidia send me new restrictive software updates? Why don’t I run my GPUs in an underground bunker, using the old most broken firmware?
I used to assume disabling a GPU in my physical possession would be impossible, but now I’m not so sure. There might be ways to make bypassing GPU lockouts on the order of difficulty of manufacturing the GPU (requiring nanoscale silicon surgery). Here’s an example scheme:
Nvidia changes their business models from selling GPUs to renting them. The GPU is free, but to use your GPU you must buy Nvidia Dollars from Nvidia. Your GPU will periodically call Nvidia headquarters and get an authorization code to do 10^15 more floating point operations. This rental model is actually kinda nice for the AI companies, who are much more capital constrained than Nvidia. (Lots of industries have moved from this buy to rent model, e.g. airplane engines)
Question: “But I’m an engineer. How (the hell) could Nvidia keep me from hacking a GPU in my physical possession to bypass that Nvidia dollar rental bullshit?”
Answer: through public key cryptography and the fact that semiconductor parts are very small and modifying them is hard.
In dozens to hundreds or thousands of places on the GPU, NVidia places toll units that block signal lines (like ones that pipe floating point numbers around) unless the toll units believe they have been paid with enough Nvidia dollars.The toll units have within them a random number generator, a public key ROM unique to that toll unit, a 128 bit register for a secret challenge word, elliptic curve cryptography circuitry, and a $$$ counter which decrements every time the clock or signal line changes.
If the $ $ $ counter is positive, the toll unit is happy and will let signals through unabated. But if the $ $ $ counter reaches zero,[1] the toll unit is unhappy and will block those signals.
To add to the $$$ counter, the toll unit (1) generates a random secret <challenge word>, (2) encrypts the secret using that toll unit’s public key (3) sends <encrypted secret challenge word> to a non-secure parts of the GPU,[2] which (4) through driver software and the internet, phones NVidia saying “toll unit <id> challenges you with <encrypted secret challenge word>” (5) Nvidia looks up the private key for toll unit <id> and replies to the GPU “toll unit <id>, as proof that I Nvidia know your private key, I decrypted your challenge word: <challenge word>”, (6) after getting this challenge word back, the toll unit adds 10^15 or whatever to the $$$ counter.
There are a lot of ways to bypass this kind of toll unit (fix the random number generator or $$$ counter to a constant, just connect wires to route around it). But the point is to make it so you can’t break a toll unit without doing surgery to delicate silicon parts which are distributed in dozens to hundreds of places around the GPU chip.
- ^
Implementation note: it’s best if disabling the toll unit takes nanoscale precision, rather than micrometer scale precision. The way I’ve written things here, you might be able to smudge a bit of solder over the whole $$$ counter and permanently tie the whole thing to high voltage, so the counter never goes down. I think you can get around these issues (make it so any “blob” of high or low voltage spanning multiple parts of the toll circuit will block the GPU) but it takes care.
- ^
This can be done slowly, serially with a single line
- ^
Not a billion billion times. You need ≈2^100 presses to get any signal, and ≈O(2^200) presses to figure out which way the signal goes. 2^200≈10^60. Planck time’s about 10^-45 seconds. If you try to press the button more than 10^45 times per second the radiation the electrons in the button will emit will be so high frequency an small wavelength that it will collapse into a black hole.
Incentives for NIMBYism is an objection I’ve seldom seen stated. “Of course I don’t want to up-zone my neighborhood to allow more productive buildings—that would triple my taxes!”.
You’re being downvoted and nobody’s telling you why :-(, so I thought I’d give some notes.
You’re not talking to the right audience. Few groups are more emotionally in favor of a glorious transhumanist future than people on LessWrong. This is not technophobes who are afraid of change. It’s technophiles who have realized, in harsh constrat to the conclusion they emotionally want, that making a powerful AI would likely be bad for humanity.
Yes, it’s important to overly anthropomorphize AIs, and you are doing that all over the place in your argument.
These arguments have been rehashed a lot. It’s fine to argue that the LessWrong consensus opinion is wrong, but you should indicate you’re familiar with why the LessWrong consensus opinion is what it is.
(To think about what it might not settle on a cooperative post-enlightenment philosophy, read, I don’t know, correct heaps?)
Conjunction Fallacy. Adding detail make ideas feel more realistic, and strictly less likely to be true.
Virtues for communication and thought can be diametrically opposed.
In a world where AI progress has wildly accelerated chip manufacture
This world?
What distinction are you making between “visualising” and “seeing”?
Good question! By “seeing” I meant having qualia, an apparent subjective experience. By “visualizing” I meant...something like using the geometric intuitions you get by looking at stuff, but perhaps in a philosophical zombie sort of way? You could use non-visual intuitions to count the vertices on a polyhedron, like algebraic intuitions or 3D tactile intuitions (and I bet blind mathematicians do). I’m not using those. I’m thinking about a wireframe image, drawn flat.
I’m visualizing a rhombicosidodecahedron right now. If I ask myself “The pentagon on the right and the one hiding from view on the left—are they the same orientation?”, I’ll think “ahh, let’s see… The pentagon on the right connects through the squares to those three pentagons there, which interlock with those 2⁄4 pentagons there, which connect through squares to the one on the left, which, no, that left one is upside-down compared to the one on the right—the middle interlocking pentagons rotated the left assembly 36° compared to the right”. Or ask “that square between the right pentagon and the pentagon at 10:20 above it <mental point>. Does perspective mean the square’s drawn as a diamond, or a skewed rectangle, weird quadrilateral?” and I think “Nah, not diamond shaped—it’s a pretty rectangular trapezoid. The base is maybe 1.8x height? Though I’m not too good at guessing aspect ratios? Seems like I if I rotate the trapezoid I can fit 2 into the base but go over by a bit?”
I’m putting into words a thought process which is very visual, BUT there is almost no inner cinema going along with those visualizations. At most ghostly, wispy images, if that. A bit like the fleeting oscillating visual feeling you get when your left and right eyes are shown different colors?
...I do not believe this test. I’d be very good at counting vertices on a polyhedron through visualization and very bad at experiencing the sensation of seeing it. I do “visualize” the polyhedra, but I don’t “see” them. (Frankly I suspect people who say they experience “seeing” images are just fooling themselves based on e.g. asking them to visualize a bicycle and having them draw it)
Thanks for crossposting! I’ve highly appreciated your contributions and am glad I’ll continue to be able to see them.
Quick summary of a reason why constituent parts like of super-organisms, like the ant of ant colonies, the cells of multicellular organisms, and endosymbiotic organelles within cells[1] are evolutionarily incentivized to work together as a unit:
Question: why do ants seem to care more about the colony than themselves? Answer: reproduction in an ant colony is funneled through the queen. If the worker ant wants to reproduce its genes, it can’t do that by being selfish. It has to help the queen reproduce. Genes in ant workers have nothing to gain by making their ant more selfish and have much to gain by making their worker protect the queen.
This is similar to why cells in your pancreas cooperate with cells in your ear. Reproduction of genes in the body is funned through gametes. Somatic evolution does pressure the cells in your pancreas to reproduce selfishly at the expense of cells in your ear (this is pancreatic cancer). But that doesn’t help the pancreas genes long term. Pancreas-genes and the ear-genes are forced to cooperate with each other because they can only reproduce when bound together in a gamete.
This sort of bounding together of genes making disperate things cooperate and act like a “super organism” is absent in members of a species. My genes do not reproduce in concert with your genes. If my genes figure out a way to reproduce at your expense, so much the better for them.
- ^
Like mitochondria and chloroplasts, which were separate organisms but evolved to work so close with their hosts that they are now considered part of the same organism.
- ^
EDIT Completely rewritten to be hopefully less condescending.
There are lessons from group selection and the extended phenotype which vaguely reduce to “beware thinking about species as organisms”. It is not clear from this essay whether you’ve encountered those ideas. It would be helpful for me reading this essay to know if you have.
Quick takes are presented inline, posts are not. Perhaps posts could be presented as title + <80 (140?) character summary.