cfoster0

Karma: 1,453

cfoster0 28 Oct 2024 20:27 UTC
6 points
0
on: Finishing The SB-1047 Documentary In 6 Weeks
Other proponents of the bill (longform, 1-3h)
[...]
Charles Foster
Note: I wouldn’t personally call myself a proponent, but I’m fine with Michaël putting me in that bucket for the sake of this post.

cfoster0 27 Sep 2024 3:59 UTC
4 points
2
in reply to: RobertM’s comment on: What prevents SB-1047 from triggering on deep fake porn/voice cloning fraud?
I’m not sure if you intended the allusion to “the tendentious assumption in the other comment thread that courts are maximally adversarial processes bent on on misreading legislation to achieve their perverted ends”, but if it was aimed at the thread I commented on… what? IMO it is fair game to call out as false the claim that
It only counts if the $500m comes from “cyber attacks on critical infrastructure” or “with limited human oversight, intervention, or supervision....results in death, great bodily injury, property damage, or property loss.”
even if deepfake harms wouldn’t fall under this condition. Local validity matters.

I agree with you that deepfake harms are unlikely to be direct triggers for the bill’s provisions, for similar reasons as you mentioned.

cfoster0 26 Sep 2024 15:04 UTC
13 points
4
in reply to: RamblinDash’s comment on: What prevents SB-1047 from triggering on deep fake porn/voice cloning fraud?
If you read the definition of critical harms, you’ll see the $500m doesn’t have to come in one of those two forms. It can also be “Other grave harms to public safety and security that are of comparable severity”.

cfoster0 6 Sep 2024 19:24 UTC
2 points
0
in reply to: Radford Neal’s comment on: Will we ever run out of new jobs?
I was trying to write a comment to explain my reaction above, but this comment said everything I would have said, in better words.

cfoster0 13 Aug 2024 16:05 UTC
40 points
13
in reply to: Joseph Miller’s comment on: Californians, tell your reps to vote yes on SB 1047!
OK, in case this wasn’t clear: if you are a Californian and think this bill should become law, don’t let my comment excuse you from heeding the above call to action. Contacting your representatives will potentially help move the needle.

cfoster0 12 Aug 2024 23:30 UTC
49 points
17
on: Californians, tell your reps to vote yes on SB 1047!

Unfortunately, due to misinformation and lobbying by big tech companies, SB 1047 is currently stalled in the Assembly Appropriations Committee.

This is extremely misleading. Any bill that would have non-negligible fiscal impact (the threshold is only $150,000 https://apro.assembly.ca.gov/welcome-committee-appropriations/appropriations-committee-rules) must be put in the Appropriation Committee “Suspense File” until after the budget is prepared. That is the status of SB 1047 and many many other bills. It has nothing to do with misinformation or lobbying, it is a part of the standard process. I believe all the bills that make it out of the Suspense File will be announced at the hearing this Thursday.

More on this: https://calmatters.org/newsletter/california-bills-suspense-file/

cfoster0 28 Jul 2024 23:03 UTC
9 points
3
in reply to: Aaron_Scher’s comment on: Re: Anthropic’s suggested SB-1047 amendments
What’s the evidence that this document is real / written by Anthropic?
Axios first reported on the letter, quoting from it but not sharing it directly:
https://www.axios.com/2024/07/25/exclusive-anthropic-weighs-in-on-california-ai-bill
The public link is from the San Francisco Chronicle, which is also visible in the metadata on the page citing the letter as “Contributed by San Francisco Chronicle (Hearst Newspapers)”.
https://www.sfchronicle.com/tech/article/wiener-defends-ai-bill-tech-industry-criticism-19596494.php

cfoster0 2 May 2024 18:09 UTC
9 points
4
on: Q&A on Proposed SB 1047
Left the following comment on the blog:
I appreciate that you’re endorsing these changes in response to the two specific cases I raised on X (unlimited model retraining and composition with unsafe covered models). My gut sense is still that ad-hoc patching in this manner just isn’t a robust way to deal with the underlying issue*, and that there are likely still more cases like those two. In my opinion it would be better for the bill to adopt a different framework with respect to hazardous capabilities from post-training modifications (something closer to “Covered model developers have a duty to ensure that the marginal impact of training/releasing their model would not be to make hazardous capabilities significantly easier to acquire.”). The drafters of SB 1047 shouldn’t have to anticipate every possible contingency in advance, that’s just bad design.
* In the same way that, when someone notices that their supposedly-safe utility function for their AI has edge cases that expose unforseen maxima, introducing ad-hoc patches to deal with those particular noticed edge cases is not a robust strategy to get an AI that is actually safe across the board.

cfoster0 29 Mar 2024 15:59 UTC
4 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
You want to learn an embedding of the opportunities you have in a given state (or for a given state-action), rather than just its potential rewards. Rewards are too sparse of a signal.
More formally, let’s say instead of the Q function, we consider what I would call the Hope function: which given a state-action pair (s, a), gives you a distribution over states it expects to visit, weighted by the rewards it will get. This can still be phrased using the Bellman equation:
Hope(s, a) = rs’ + f Hope(s’, a’)
The “successor representation” is somewhat close to this. It encodes the distribution over future states a partcular policy expects to visit from a particular starting state, and can be learned via the Bellman equation / TD learning.

cfoster0 14 Dec 2023 19:01 UTC
5 points
3
on: AI #42: The Wrong Answer

On reflection these were bad thresholds, should have used maybe 20 years and a risk level of 5%, and likely better defined transformational. The correlation is certainly clear here, the upper right quadrant is clearly the least popular, but I do not think the 4% here is lizardman constant.

Wait, what? Correlation between what and what? 20% of your respondents chose the upper right quadrant (transformational/safe). You meant the lower left quadrant, right?

cfoster0 28 Nov 2023 3:20 UTC
38 points
15
on: Apocalypse insurance, and the hardline libertarian take on AI risk
Very surprised there’s no mention here of Hanson’s “Foom Liability” proposal: https://www.overcomingbias.com/p/foom-liability

cfoster0 3 Nov 2023 16:22 UTC
13 points
9
on: Thoughts on open source AI
I appreciate that you are putting thought into this. Overall I think that “making the world more robust to the technologies we have” is a good direction.
In practice, how does this play out?
Depending on the exact requirements, I think this would most likely amount to an effective ban on future open-sourcing of generalist AI models like Llama2 even when they are far behind the frontier. Three reasons that come to mind:
1. The set of possible avenues for “novel harms” is enormous, especially if the evaluation involves “the ability to finetune [...], external tooling which can be built on top [...], and API calls to other [SOTA models]”. I do not see any way to clearly establish “no novel harms” with such a boundless scope. Heck, I don’t even expect proprietary, closed-source models to be found safe in this way.
2. There are many, many actors in the open-source space, working on many, many AI models (even just fine-tunes of LLaMA/Llama2). That is kind of the point of open sourcing! It seems unlikely that outside evaluators would be able to evaluate all of these, or for all these actors to do high-quality evaluation themselves. In that case, this requirement turns into a ban on open-sourcing for all but the largest & best-resourced actors (like Meta).
3. There aren’t incentives for others to robustify existing systems or to certify “OK you’re allowed to open-source now”, in the way as there are for responsible disclosure. By default, I expect those steps to just not happen, & for that to chill open-sourcing.

cfoster0 2 Nov 2023 21:17 UTC
16 points
21
in reply to: DanielFilan’s comment on: Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk
If we are assessing the impact of open-sourcing LLMs, it seems like the most relevant counterfactual is the “no open-source LLM” one, right?

cfoster0 2 Nov 2023 21:01 UTC
4 points
2
in reply to: DanielFilan’s comment on: Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk
Noted! I think there is substantial consensus within the AIS community on a central claim that the open-sourcing of certain future frontier AI systems might unacceptably increase biorisks. But I think there is not much consensus on a lot of other important claims, like about for which (future or even current) AI systems open-sourcing is acceptable and for which ones open-sourcing unacceptably increases biorisks.

cfoster0 2 Nov 2023 20:00 UTC
19 points
16
in reply to: jacquesthibs’s comment on: Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk
(explaining my disagree reaction)
The open source community seems to consistently assume the case that the concerns are about current AI systems and the current systems are enough to lead to significant biorisk. Nobody serious is claiming this
I see a lot of rhetorical equivocation between risks from existing non-frontier AI systems, and risks from future frontier or even non-frontier AI systems. Just this week, an author of the new “Will releasing the weights of future large language models grant widespread access to pandemic agents?” paper was asserting that everyone on Earth has been harmed by the release of Llama2 (via increased biorisks, it seems). It is very unclear to me which future systems the AIS community would actually permit to be open-sourced, and I think that uncertainty is a substantial part of the worry from open-weight advocates.

cfoster0 1 Nov 2023 18:07 UTC
8 points
2
in reply to: Dagon’s comment on: Snapshot of narratives and frames against regulating AI
Note that the outlook from MIRI folks appears to somewhat agree with this, that there does not exist an authority that can legibly and correctly regulate AI, except by stopping it entirely.

cfoster0 31 Oct 2023 15:38 UTC
6 points
4
in reply to: Nathan Helm-Burger’s comment on: Would it make sense to bring a civil lawsuit against Meta for recklessly open sourcing models?
At face value it seems not-credible that “everyone on Earth” has been actually harmed by the release of Llama2. You could try to make a case that there’s potential for future harms downstream of Llama2′s release, and that those speculative future harms could impact everyone on Earth, but I have no idea how one would claim that they have already been harmed.

cfoster0 26 Oct 2023 17:18 UTC
6 points
1
in reply to: Oliver Sourbut’s comment on: AI as a science, and three obstacles to alignment strategies
I agree that they are related. In the context of this discussion, the critical difference between SGD and evolution is somewhat captured by your Assumption 1:
Fixed ‘fitness function’ or objective function mapping genome to continuous ‘fitness score’
Evolution does not directly select/optimize the content of minds. Evolution selects/optimizes genomes based (in part) on how they distally shape what minds learn and what minds do (to the extent that impacts reproduction), with even more indirection caused by selection’s heavy dependence on the environment. All of that creates a ton of optimization “slack”, such that large-brained human minds with language could steer optimiztion far faster & more decisively than natural selection could. This what 1a3orn was pointing to earlier with
evolution does not grow minds, it grows hyperparameters for minds. When you look at the actual process for how we actually start to like ice-cream—namely, we eat it, and then we get a reward, and that’s why we like it—then the world looks a a lot less hostile, and misalignment a lot less likely.
SGD does not have that slack by default. It acts directly on cognitive content (associations, reflexes, decision-weights), without slack or added indirection. If you control the training dataset/environment, you control what is rewarded and what is penalized, and if you are using SGD, then this lets you directly mold the circuits in the model’s “brain” as desired. That is one of the main alignment-relevant intuitions that gets lost when blurring the evolution/SGD distinction.

cfoster0 26 Oct 2023 0:01 UTC
4 points
in reply to: Buck’s comment on: Buck’s Shortform
An attempt was made last year, as an outgrowth of some assorted shard theory discussion, but I don’t think it got super far:
- Price’s equation for neural networks

cfoster0 25 Oct 2023 23:57 UTC
19 points
14
in reply to: habryka’s comment on: AI as a science, and three obstacles to alignment strategies
Note: I just watched the videos. I personally would not recommend the first video as an explanation to a layperson if I wanted them to come away with accurate intuitions around how today’s neural networks learn / how we optimize them. What it describes is a very different kind of optimizer, one explicitly patterned after natural selection such as a genetic algorithm or population-based training, and the follow-up video more or less admits this. I would personally recommend they opt for videos these instead: