quiet_NaN

Karma: 267

quiet_NaN 21 Nov 2024 12:01 UTC
6 points
0
on: “It’s a 10% chance which I did 10 times, so it should be 100%”
Relatedly, if you perform an experiment n times, and the probability of success is p, and the expected number of total successes kp is much smaller than one, then kp is a reasonable measure of getting at least once success, because the probability of getting more than one success can be neglected.
For example, if Bob plays the lottery for ten days, and each days has a 1:1000,000 chance of winning, then overall he will have a chance of 100,000 of winning once.
This is also why micromorts are roughly additive: if travelling by railway has a mortality of one micomort per 10Mm, then travelling for 50Mm will set you back 5 micomort. Only if you leave what I would call the ‘Newtonian regime of probability’, e.g. by somehow managing to travel 1Tm with the railway, you are required to do proper probability math, because naive addition would tell you that you will surely have a fatal accident (1 mort) in that distance, which is clearly wrong.

quiet_NaN 27 Oct 2024 13:28 UTC
6 points
1
in reply to: Gunnar_Zarncke’s comment on: The Case For Bullying
Getting down-voted to −27 is an achievement. Most things judged ‘bad AI takes’ only go to −11 or so, even that recent P=NP proof only got to −25. Of course, if the author is right, then downvoting further is providing helpful incentives to him.
I think that bullying is quite distinct from status hierarchies. The latter are unavoidable. There will always be some clique of cool kids in the class who will not invite the non-cool kids to their parties. This is ok. Sometimes, status is correlated with behaviors which are pro-social (kids not smoking; donating to EA), sometimes it is correlated with behaviors which are net-negative (kids smoking; serving in the SS). I was not part of the cool kids circle, and I was fine with that. Live and let live and all that.
‘Bullying’ has a distinct negative connotation. The central example is someone who is targeted for sport for being different from the others. The bullies don’t want the victims to change their ways, they just like to make their life miserable for thrills. I am sure that sometimes it unintentionally helps their victim, if you push enough people, at some point you are bound to push someone out of the path of a car or bullet. In the grand scheme of things, the bullies are however net negative for their victims and society overall.

quiet_NaN 27 Oct 2024 12:52 UTC
5 points
3
on: Arithmetic Models: Better Than You Think
I see this as less of an endorsement of linear models and more of a scathing review of expert performance.
- When an arithmetic model is calibrated, it is specifically by including feedback from the real-world effects of its predictions. Experts do not, as a rule, seek out any feedback on their calibration.
This. Basically, if your job is to do predictions, and the accuracy of your predictions is not measured, then (at least the prediction part of) your job is bullshit.
I think that if you compare simple linear models in domains where people actually care about their predictions, the outcome would be different. For example, if simple models predicted stock performance better than experts at investment banks, anyone with a spreadsheet could quickly become rich. There are few if any cases of ‘I started with Excel and 1000$, and now I am a billionaire’. Likewise, I would be highly surprised to see a simple linear model outperform Nate Silver or the weather forecast.
Even predicting chess outcomes from mid-game board configurations is something where I would expect human experts to outperform simple statistical models working on easily quantifiable data (e.g. number of pieces remaining, number of possible moves, being in check, etc).
Neural networks contained in animal brains (which includes human brains) are quite capable of implementing linear models, and such should at least perform equally well when they are properly trained. A wolf pack deciding to chase or not chase some prey has direct evolutionary skin in the game of making their prediction of success as accurate as possible which the average school counselor predicting academic success simply does not have.
--
You touch this a bit in ‘In defense of explainatory modeling’, but I want to emphasize that uncovering causal relationships and pathways is central to world modelling. Often, we don’t want just predictions, we want predictions conditional on interventions. If you don’t have that, you will end up trying to cure chickenpox with makeup, as ‘visible blisters’ is negatively correlated with outcomes.
Likewise, if we know the causal pathway, we have a much better basis to judge if some finding can be applied to out-of-distribution data. No matter how many anvils you have seen falling, without a causal understanding (e.g. Newtonian mechanics), you will not be able to reliably apply your findings to falling apples or pianos.

quiet_NaN 24 Aug 2024 23:01 UTC
38 points
11
on: How I started believing religion might actually matter for rationality and moral philosophy
What I don’t understand is why there should be a link between trapped priors and an moral philosophy.
I mean, if moral realism was correct, i.e. if moral tenets such as “don’t eat pork”, “don’t have sex with your sister”, or “avoid killing sentient beings” had an universal truth value for all beings capable of moral behavior, then one might argue that the reason why people’s ethics differ is that they have trapped priors which prevent them from recognizing these universal truths.
This might be my trapped priors talking, but I am a non-cognitivist. I simply believe that assigning truth values to moral sentences such as “killing is wrong” is pointless, and they are better parsed as prescriptive sentences such as “don’t kill” or “boo on killing”.
In my view, moral codes are intrinsically subjective. There is no factual disagreement between Harry and Professor Quirrell which they could hope to overcome through empiricism, they simply have different utility functions.
--
My second point is that if moral realism was true, and one of the key roles of religion was to free people from trapped priors so they could recognize these universal moral truths, then at least during the founding of religions, we should see some evidence of higher moral standards before they invariably mutate into institutions devoid of moral truths. I would argue that either, our commonly accepted humanitarian moral values are all wrong or this mutation process happened almost instantly:
- Whatever Jesus thought about gender equality when he achieved moral enlightenment, Paul had his own ideas a few decades later.
- Mohammed was clearly not opposed to offensive warfare.
- Martin Luther evidently believed that serfs should not rebel against their lords.
On the other hand, for instances where religions did advocate for tenets compatible with humanitarianism, such as in Christian abolitionism, do not seem to correspond to strong spiritualism. Was Pope Benedict XIV condemning the slave trade because he was more spiritual (and thus in touch with the universal moral truth) than his predecessors who had endorsed it?
--
My last point is that especially with regard to relational conflicts, our map not corresponding to the territory might often not be a bug, but a feature. Per Hanson, we deceive ourselves so that we can better deceive others. Evolution has not shaped our brains to be objective cognitive engines. In some cases, objective cognition is what it advantageous—if you are alone hunting a rabbit, no amount of self-deception will fill your stomach—but in any social situation, expect evolution to put the hand on the scales of your impartial judgement. Arguing that your son should become the new chieftain because he is the best hunter and strongest warrior is much more effective than arguing for that simply because he is your son—and the best way to argue that is to believe it, no matter if it is objectively true.
The adulterer, the slave owner and the wartime rapist all have solid evolutionary reasons to engage in behaviors most of us might find immoral. I think their moral blind spots are likely not caused by trapped priors, like an exaggerated fear of dogs is. Also, I have no reason to believe that I don’t have similar moral blind spots hard-wired into my brain by evolution.
I would bet that most of the serious roadblocks to a true moral theory (if such a thing existed) are of that kind, instead of being maladaptive trapped priors. Thus, even if religion and spirituality are effective at overcoming maladaptive trapped priors, I don’t see how they would us bring closer to moral cognition.

quiet_NaN 29 Jul 2024 4:12 UTC
1 point
0
on: Universal Basic Income and Poverty
Note: there is an AI audio version of this text over here: https://askwhocastsai.substack.com/p/eliezer-yudkowsky-tweet-jul-21-2024
I find the AI narrations offered by askwho generally ok, worse than what a skilled narrator (or team) could do but much better than what I could accomplish.

quiet_NaN 29 Jul 2024 4:08 UTC
5 points
3
on: Universal Basic Income and Poverty
[...] somehow humanity’s 100-fold productivity increase (since the days of agriculture) didn’t eliminate poverty.
That feels to me about as convincing as saying: “Chemical fertilizers have not eliminated hunger, just the other weekend I was stuck on a campus with a broken vending machine.”
I mean, sure, both the broken vending machine and actual starvation can be called hunger, just as both working 60h/week to make ends meet or sending your surviving kids into the mines or prostituting them could be called poverty, but the implication that either scourge of humankind has not lost most of its terror seems clearly false.
Sure, being poor in the US sucks, but I would rather spend a year living the life of someone in the bottom 10% income bracket in the 2024 US than spending a month living the life of a poor person during the English industrial revolution.
I am also not convinced that 60h/week is what it actually takes to survive in the US. I can totally believe that this amount of unskilled labor might be required to rent accommodations in cities, though.

quiet_NaN 18 Jul 2024 12:49 UTC
1 point
0
on: Superbabies: Putting The Pieces Together
Critically, the gene editing of the red blood cells can be done in the lab; trying to devise an injectable or oral substance that would actually transport the gene-editing machinery to an arbitrary part of the body is much harder.
I am totally confused by this. Mature red blood cells don’t contain a nucleus, and hence no DNA. There is nothing to edit. Injecting blood cells produced by gene-edited bone marrow in vitro might work, but would only be a therapy, not a cure: it would have to be repeated regularly. The cure would be to replace the bone marrow.
So I resorted to reading through the linked FDA article. Relevant section:
The modified blood stem cells are transplanted back into the patient where they engraft (attach and multiply) within the bone marrow and increase the production of fetal hemoglobin (HbF), a type of hemoglobin that facilitates oxygen delivery.
Blood stem cells seems to be FDA jargon for ematopoietic stem cells. From the context, I would guess they are harvested from the bone marrow of the patients, then CRISPRed, and then injected back in the blood stream where they will find back to the bone marrow.
I still don’t understand how they would outcompete the non-GMO bone marrow which produces the faulty red blood cells, though.
I would also take the opportunity to point out that the list of FDA-approved gene therapies tells us a lot about the FDA and very little about the state of the art. This is the agency which banned life-saving baby nutrition for two years, after all. Anchoring what is technologically possible to what the FDA approves would be like anchoring what is possible in mobile phone tech to what is accepted by the Amish.
Also, I think that editing of multi-cellular organisms is not required for designer babies at all.
1. Start with a fertilized egg, which is just a single cell. Wait for the cell to split. After it has split, separate the cells into two cells. Repeat until you have a number of single-cell candidates.
2. Apply CRISPR to these cells individually. allow them to split again. Do a genome analysis on one of the two daughter cell. Select a cell line where you did only the desired change in the genome. Go back to step one and apply the next edit.
Crucially, the costs would only scale linearly with the number of edits. I am unsure how easy is that “turn one two-cell embryo into two one-cell embryos”, though.
Of course, it would be neater to synthesize the DNA of the baby from the scratch, but while prices per base pair synthesis have been dropped a lot, they are clearly still to high to pay for building a baby (and there are likely other tech limitations).

quiet_NaN 18 Jul 2024 11:43 UTC
2 points
2
in reply to: Elias711116’s comment on: Superbabies: Putting The Pieces Together
I thought this first too. I checked on Wikipedia:
Adult stem cells are found in a few select locations in the body, known as niches, such as those in the bone marrow or gonads. They exist to replenish rapidly lost cell types and are multipotent or unipotent, meaning they only differentiate into a few cell types or one type of cell. In mammals, they include, among others, hematopoietic stem cells, which replenish blood and immune cells, basal cells, which maintain the skin epithelium [...].
I am pretty sure that the thing a skin cell makes per default when it splits is more skin cells, so you are likely correct.

quiet_NaN 1 Jul 2024 0:56 UTC
3 points
0
in reply to: milanrosko’s comment on: The Incredible Fentanyl-Detecting Machine
See here. Of course, that article is a bit light on information on detection thresholds, false-positive rates and so on as compared to dogs, mass spectrometry or chemical detection methods.
I will also note that humans have 10-20M olfactory receptor neurons, while bees have 1M neurons in total. Probably bees are under more evolutionary pressure to make optimal use of their olcfactory neurons, though.

quiet_NaN 1 Jul 2024 0:34 UTC
1 point
0
in reply to: Review Bot’s comment on: The Incredible Fentanyl-Detecting Machine
Dear Review Bot,
please avoid double-posting.
On the other hand, I don’t think voting you to −6 is fair, so I upvoted you.

quiet_NaN 1 Jul 2024 0:29 UTC
2 points
−2
in reply to: gwern’s comment on: The Incredible Fentanyl-Detecting Machine
My take on sniffer dogs is that frequently, what they are best at is picking up is unconscious tells from their handler. In so far as they do, they are merely science!-washing the (possibly meritful) biases of the police officier.
Packaging something really air-tight without outside contamination is indeed far from trivial. For example, the swipe tests taken at airports are useful because while it is certainly possible to pack a briefcase full of explosives without any residue on the outside is certainly possible, most of the people who could manage to build a bomb would not manage to do that.
Of course, there are also no profit margins in blowing up airplanes, so stopping the amateurs is already 95% of the job.
There are significant profit margins in drug trafficking. After you intercept a few shipments and arrest a few mules, the cleverer drug lords will wisen up.
A multi-method approach might work for a while, glass vials are probably more visible on that CT scan than some organic substance.

quiet_NaN 1 Jul 2024 0:00 UTC
2 points
0
on: The Incredible Fentanyl-Detecting Machine
My first question is about the title picture. I have some priors on how a computer tomography machine for vehicles would look like. Basically, you want to take x-ray images from many different directions. The medical setup where you have a ring which contains the x-ray source on one side and the detectors on the other side, and rotate that ring to take images from multiple directions before moving the patient perpendicular to the ring to record the next slice exists for a reason: high resolution x-ray detectors are expensive. If we scaled this up to a car size, we might have a ring of four meters in diameter. A bridge of some low-Z material (perhaps beams of wood) would go through that ring. Park on the bridge, get out of the car, and watch as the rotating ring moves over the bridge.
The thing in the picture looks does not look like it has big moving parts. I can kind of see it taking a sideways x-ray image of the car but I am puzzled by the big grey boxes visible over the car. Having an x-ray source or detector above the car would only make sense if you had the other device below the car, in the road. Of course, if there is anything in the road, one would imagine that that piece of road was some low-Z material material designed not to block half of your x-rays. Instead, the road material below the car looks exactly like the road outside the detector, concrete probably.
In Europe, we kind of dislike being exposed to ionizing radiation (even though Germany is rather silly about it). I get that the US is a bit more relaxed about it, but a computer tomography device capable of scanning a car with a good resolution would likely emit to a lot of scattered x-rays to anyone nearby. The position in which that boarder guard stands would likely not be a safe position to stand a significant fraction of your work-life.
At the very least, I would expect that black and yellow trefoil sign warning of ionizing radiation, with a visual indicator if the x-ray is on, and a line marking the minimum safe distance. More realistically, you might have something like a garage door before and behind the car.
Just from the vibes, I get that the pictured non-intrusive fentanyl inspection machine might run on the operation principle of the famous ADE 651, which is to say it does nothing whatsoever.
While detecting dry fentanyl through high resolution CT scans seems to be possible in principle at least, once the smugglers go through the additional trouble of dissolving it all bets are off. Wikipedia is light on solubility data, but from the looks of it fentanyl should dissolve well in organic solvents.
You can check the gas tank of every car if you want, but what are you going to do if an eight-wheeler full of one- gallon-bottles of cooking oil crosses the border? If your scan is good, it would perhaps detect if one bottle in the middle of the stack has been filled with powder instead. There is no way in hell it will detect if one such bottle has ten grams (perhaps 10k doses) of fentanyl dissolved in it. Your best bet would be to detect some residue of the oil on product seized off the streets and work backwards from that.
Of course, “as long as the profit margin is that high, we could go full iron curtain on our borders and would not stop the trafficking” is not a politically acceptable answer. Hence some snake oil salesmen are able to get your tax dollars for products which have not been proven in adversarial conditions. (To train an AI, it is not enough to have a ton of samples, you would need known positive and negative samples. Also, while we are playing buzzword bingo, why not mention that the shipping manifests will be securely stored on The Blockchain?)

quiet_NaN 23 May 2024 10:40 UTC
1 point
0
in reply to: Seth Herd’s comment on: The Problem With the Word ‘Alignment’
The deliberately clumsy term “AInotkilleveryoneism” seems good for this, in any context you can get away with it.
Hard disagree. The position “AI might kill all humans in the near future” is still quite some inferential distance away from the mainstream even if presented in a respectable academic veneer.
We do not have weirdness points to spend on deliberately clumsy terms, even on LW. Journalists (when they are not busy doxxing people) can read LW too, and if they read that the worry about AI as an extinction risk is commonly called notkilleveryoneism they are orders of magnitude less likely to take us serious, and being taken serious by the mainstream might be helpful for influencing policy.
We could probably get away with using that term ten pages deep into some glowfic, but anywhere else ‘AI as an extinction risk’ seems much better.

quiet_NaN 21 May 2024 18:46 UTC
1 point
1
in reply to: cata’s comment on: Some perspectives on the discipline of Physics
It is also useful for a lot of practical problems, where you can treat $ℏ$ as being essentially zero and $c_{0}$ as being essentially infinite. If you want to get anywhere with any practical problem (like calculating how long a car will take to come to a stop), half of the job is to know which approximations (“cheats”) are okay to use. If you want to solve the fully generalized problem (for a car near the Planck units or something), you will find that you would need a theory of everything (that is quantum mechanics plus general relativity) to do so and we don’t have that.

quiet_NaN 21 May 2024 18:24 UTC
2 points
0
on: The Problem With the Word ‘Alignment’
I think that “AI Alignment” is a useful label for the somewhat related problems around P1-P6. Having a term for the broader thing seems really useful.
Of course, sometimes you want labels to refer to a fairly narrow thing, like the label “Continuum Hypothesis”. But broad labels are generally useful. Take “ethics”, another broad field label. Nominative ethics, applied ethics, meta-ethics, descriptive ethics, value theory, moral psychology, et cetera. I someone tells me “I study ethics” this narrows down what problems they are likely to work on, but not very much. Perhaps they work out a QALY-based systems for assigning organ donations, or study the moral beliefs of some peoples, or argue if moral imperatives should have a truth value. Still, the label confers a lot of useful information over a broader label like “philosophy”.
By contrast, “AI Alignment” still seems rather narrow. P2 for example seems a mostly instrumental goal: if we have interpretability, we have better chances to avoid a takeover of an unaligned AI. P3 seems helpful but insufficient for good long term outcomes: an AI prone to disobeying users or interpreting their orders in a hostile way would—absent some other mechanism—also fail to follow human values more broadly, but an P3-aligned AI in the hand of a bad human actor could still cause extinction, and I agree that social structures should probably be established to ensure that nobody can unilaterally assign the core task (or utility function) of an ASI.

quiet_NaN 19 May 2024 18:00 UTC
25 points
11
on: “If we go extinct due to misaligned AI, at least nature will continue, right? … right?”
I think an AI is slightly more likely to wipe out or capture humanity than it is to wipe out all life on the planet.
While any true Scottsman ASI is so far above us humans as we are above ants and does not need to worry about any meatbags plotting its downfall, as we don’t generally worry about ants, it is entirely possible that the first AI which has a serious shot at taking over the world is not quite at that level yet. Perhaps it is only as smart as von Neumann and a thousand times faster.
To such an AI, the continued thriving of humans poses all sorts of x-risks. They might find out you are misaligned and coordinate to shut you down. More worrisome, they might summon another unaligned AI which you would have to battle or concede utility to later on, depending on your decision theory.
Even if you still need some humans to dust your fans and manufacture your chips, suffering billions of humans to live in high tech societies you do not fully control seems like the kind of rookie mistake I would not expect a reasonably smart unaligned AI to make.
By contrast, most of life on Earth might get snuffed out when the ASI gets around to building a Dyson sphere around the sun. A few simple life forms might even be spread throughout the light cone by an ASI who does not give a damn about biological contamination.
The other reason I think the fate in store for humans might be worse than that for rodents is that alignment efforts might not only fail, but fail catastrophically. So instead of an AI which cares about paperclips, we get an AI which cares about humans, but in ways we really do not appreciate.
But yeah, most forms of ASI which turn out for out bad for homo sapiens also turn out bad for most other species.

quiet_NaN 10 May 2024 22:38 UTC
1 point
0
on: Dating Roundup #3: Third Time’s the Charm
Cassette AI: “Dude I just matched with a model”
“No way”
“Yeah large language”

This made me laugh out loud.
Otherwise, my idea for a dating system would be that given that the majority of texts written will invariably end up being LLM-generated, it would be better if every participant openly had an AI system as their agent. Then the AI systems of both participants could chat and figure out how their user would rate the other user based on their past ratings of suggestions. If the users end up being rated among each others five most viable candidates,
Of course, if the agents are under the full control of the users, the next step of escalation will be that users will tell their agents to lie on their behalf. (‘I am into whatever she is into. If she is big on horses, make up a cute story about me having had a pony at some point. Just put the relevant points on the cheat sheet for the date’.) This might be solved by having the LLM start by sending out a fixed text document. If horses are mentioned as item 521, after entomology but before figure skating, the user is probably not very interested in them. Of course, nothing would prevent a user from at least generically optimizing their profile to their target audience. “A/B testing has shown that the people you want to date are mostly into manga, social justice and ponies, so this is what you should put on your profile.” Adversarially generated boyfriend?

quiet_NaN 10 May 2024 21:45 UTC
2 points
1
on: AI and Chemical, Biological, Radiological, & Nuclear Hazards: A Regulatory Review
I was fully expecting having to write yet another comment about how human-level AI will not be very useful for a nuclear weapon program. I concede that the dangers mentioned instead (someone putting an AI in charge of a reactor or nuke) seem much more realistic.
Of course, the utility of avoiding sub-extinction negative outcomes with AI in the near future is highly dependent on p(doom). For example, if there is no x-risk, then the first order effects of avoiding locally bad outcomes related to CBRN hazards are clearly beneficial.
On the other hand, if your p(doom) is 90%, then making sure that non-superhuman AI systems work without incident is alike to clothing kids in asbestos gear so they don’t hurt themselves while playing with matches.
Basically, if you think a road leads somewhere useful, you would prefer that the road goes smoothly, while if a road leads off a cliff you would prefer it to be full of potholes so that travelers might think twice about taking it.
Personally, I tend to favor first-order effects (like fewer crazies being able to develop chemical weapons) over hypothetical higher order effects (like chemical attacks by AI-empowered crazies leading to a Butlerian Jihad and preventing an unaligned AI killing all humans). “This looks locally bad, but is actually part of a brilliant 5-dimensional chess move which will lead to better global outcomes” seems like the excuse of every other movie villain.

quiet_NaN 10 May 2024 16:32 UTC
1 point
0
on: Explaining a Math Magic Trick
Edit: looks like was already raised by Dacyn and answered to my satisfaction by Robert_AIZI. Correctly applying the fundamental theorem of calculus will indeed prevent that troublesome zero from appearing in the RHS in the first place, which seems much preferable to dealing with it later.
~~My real analysis might be a bit rusty, but I think defining I as the definite integral breaks the magic trick.~~
~~I mean, in the last line of the ‘proof’,~~ $1 + I + I^{2} + \dots$ ~~gets applied to the zero function.~~
~~Any definitive integral of the zero function is zero, so you end up with f(x)=0, which is much less impressive.~~
More generally, asking the question Op(f)=0 for any invertable linear operator Op is likely to set yourself up for disappointment. Since the trick relies on inverting an operator, we might want to use a non-linear operator.
$I (f (x)) := \int_{0}^{x} f (x) d x + C$ ~~where C is some global constant might be better. (This might affect the radius of convergence of that Taylor series, do not use for production yet!)~~
~~This should result in… uhm…~~ $C + (C x + C) + (C \frac{x^{2}}{2} + C x + C) + \dots$ ?
Which is a lot more work to reorder than the original convention used in the ‘proof’ where all the indefinite integrals of the zero function are conveniently assumed to be the same constant, and all other indefinite integrals conveniently have integration constants of zero.
~~Even if we sed s/C//~~ $ϵ$ ~~ and proclaim that~~ $ϵ$ should be small (e.g. compared to x) and we are only interested in the leading order terms, this would not work. What one would have to motivate is throwing everything but the leading power of x out for every $I^{n}$ ~~evaluation, then later meticulously track these lower order terms in the sum to arrive at the Taylor series of the exponential.~~

quiet_NaN 4 May 2024 20:26 UTC
3 points
−1
in reply to: Nathan Helm-Burger’s comment on: Why I’m doing PauseAI
I think I have two disagreements with your assessment.
First, the probability of a random independent AI researcher or hobbyist discovering a neat hack to make AI training cheaper and taking over. GPT4 took 100M$ to train and is not enough to go FOOM. To train the same thing within the budget of the median hobbyist would require algorithmic advantages of three or four orders of magnitude.
Historically, significant progress has been made by hobbyists and early pioneers, but mostly in areas which were not under intense scrutiny by established academia. Often, the main achievement of a pioneer is discovering a new field, them picking all the low-hanging fruits is more of a bonus. If you had paid a thousand mathematicians to think about signal transmission on a telegraph wire or semaphore tower, they probably would have discovered Shannon entropy. Shannon’s genius was to some degree looking into things nobody else was looking into which later blew up into a big field.
It is common knowledge that machine learning is a booming field. Experts from every field of mathematics have probably thought if there is a way to apply their insights to ML. While there are certainly still discoveries to be made, all the low-hanging fruits have been picked. If a hobbyist manages to build the first ASI, that would likely be because they discover a completely new paradigm—perhaps beyond NNs. The risk that a hobbyist discovers a concept which lets them use their gaming GPU to train an AGI does not seem that much higher than in 2018 -- either would be completely out of the left field.
My second disagreement is the probability of an ASI being roughly aligned with human values, or to be more precise, the difference of that probability conditional on who discovers it. The median independent AI enthusiast is not a total asshole [citation needed], so if alignment is easy and they discover ASI, chances are that they will be satisfied with becoming the eternal god emperor of our light cone and not bother to tell their ASI to turn any any huge number of humans to fine red mist. This outcome will not be so different than if Facebook develops an aligned ASI first. If alignment is hard—which we have some reason to believe it is—then the hobbyist who builds ASI by accident will doom the world, but I am also rather cynical about the odds of big tech having much better odds.
Going full steam ahead is useful if (a) the odds of a hobbyist building ASI if big tech stops capability research are significant and (b) alignment is very likely for big tech and unlikely for the hobbyist. I do not think either one is true.