I also think that giving medical advice/being a doctor and so on requires you to have a specific degree and anyone can do math. Also math seems somewhat easier to verify. I would say he is more likely to loss his job sooner.
Kajus
Karma: 89
There is an attitude I see in AI safety from time to time when writing papers or doing projects:
People think more about doing a cool project rather than having a clear theory of change.
They spend a lot of time optimizing for being “publishable.”
I think it’s bad if we want to solve AI safety. On the other hand, having a clear theory of change is hard. Sometimes, it’s just so much easier to focus on an interesting problem instead of constantly asking yourself, “Is this really solving AI safety?”
How to approch this whole thing? Idk about you guys but this is draining for me.
Why would I publish papers in AI safety? Do people even read them? Am I doing it just to gain credibility? Aren’t there already too many papers?
There are apps that can measure when you go to sleep based on your breath or something. Maybe that could be helpful?
I don’t get why people disagree with me and don’t try to comment. I will do it by myself then. There is one thing we do to make use of models that are misaligned with our goals—we jailbreak them—so this is what we can do with scheming models—we can jailbreak them to get useful outputs. Or you might expect that the model is useful but it’s scheming from time to time. Then you can get useful outputs. Validation is still a problem tho.
Isn’t being a real expected value-calculating consequentialist really hard? Like, this week an article about not ignoring bad vibes was trending. I think that it’s very easy to be a naive consequentialist, and it doesn’t pay off, you get punished very easily because you miscalcualte and get ostracized or fuck your emotions up. Why would we get a consequentialist AI?
If a system is scheming then its usefulness is zero. It’s like talking with a devil. It shouldn’t be run at all. Why would it help us with anything? You won’t solve alignment with an AI that is scheming and the trusted one is too stupid otherwise you would have already solved alignment.
Some of the stories assume a lot of AIs, wouldn’t a lot of human-level AIs be very good at creating a better AI? Also it seems implausible to me that we will get a STEM-AGI that doesn’t think about humans much but is powerful enought to get rid of atmosphere. On a different note, evaluating plausability of scenarios is a whole different thing that basically very few people do and write about in AI safety.
What do you mean by
what is the most important thing I could be working on and why aren’t I on track to deal with it?
Like what do you think about when you ask this question? Is this more about “most important today” or “most important in my life”?
I think that AI labs are going to use LoRA to lock cool capabilities in models and offer a premium subscription with those capabilities unlocked.
I recently came up with an idea to improve my red-teaming skills. By red-teaming, I mean identifying obvious flaws in plans, systems, or ideas.
First, find high-quality reviews on open review or somewhere else. Then, create a dataset of papers and their reviews, preferably in a field that is easy to grasp and sufficiently complex. Read papers, compare to the reviews.
Obvious flaw is that you see the reviews before, so you might want to hire someone else to do it. Doing this in a group is also really great.
If you think of Pangolin behaviour and name as control it seems that it is going down slower than Axolotl. Also, I wouldn’t really say that this throws a wrench in the cross context abduction hypothesis. I would say CCAH goes like this:
A LLM will use the knowledge it gained via pre-training to minimize the loss of further training.
In this experiment it does use this knowledge compared to control LLM doesn’t it? At least it has responds differently to control LLM.
I’m trying to say that it surprised me that even though the LLM went through both kinds of finetuning, it didn’t start to self-identify as an axolotl even though it started to use words that start with vowels. (If I understood it correctly).
Cool experiment! The 2b results are surprising to me. I thought that the LLM in 2b should be 1. finetuned on the declarative dataset of 900 questions and answers 2. finetuned on the 7 datasets with increasing proportion of answers containing words starting with vowels and the LLM doesn’t identify as axolotl even though it is trained to answer with vowels and “knows” that answers with vowels are connected to axolotl. Interesting!
It could be really interesting how the employemnt looks before and after the camp.
Great post!
In the past, broad interventions would clearly have been more effective: for instance, there would have been little use in studying empirical alignment prior to deep learning. Even more recently than the advent of deep learning, many approaches to empirical alignment were highly deemphasized when large, pretrained language models arrived on the scene (refer to our discussion of creative destruction in the last post).
As discussed in the last post, a leading motivation for researchers is the interestingness or “coolness” of a problem. Getting more people to research relevant problems is highly dependent on finding interesting and well-defined subproblems for them to work on. This relies on concretizing problems and providing funding for solving them.
This seems be a conflicting advice to me. If you try to follow both you might end up having hard time finding direction for research.
I don’t fully understand the post. Without a clear definition of “winning,” the points you’re trying to make — as well as the distinction between pragmatic and non-pragmatic principles (which also aligns with strategies and knowledge formation) — aren’t totally clear. For instance, “winning,” in some vague sense, probably also includes things like “fitting with evidence,” taking advice from others, and so on. You don’t necessarily need to turn to non-pragmatic principles or those that don’t derive from the principle of winning. “Winning” is a pretty loose term.
I’ve just read “Against the singularity hypothesis” by David Thorstad and there are some things there that seems obviously wrong to me—but I’m not totally sure about it and I want to share it here, hoping that somebody else read it as well. In the paper, Thorstad tries to refute the singularity hypothesis. In the last few chapters, Thorstad discuses the argument for x-risks from AI that’s based on three premises: singularity hypothesis, Orthogonality Thesis and Instrumental Convergence and says that since singularity hypothesis is false (or lacks proper evidence) we shouldn’t worry that much about this specific scenario. Well, it seems to me like we should still worry and we don’t need to have recursively self-improving agents to have agents smart enough so that instrumental convergence and orthogonality hypothesis applies to them.
Interesting! Reading this makes me think that there is some kind of tension between “paperclip maximizer” view on AI. Some interventions or risks you mentioned assume that AI will get its attitude from the training data, while the “paperclip maximizer” is an AI with just a goal and with whatever beliefs it will help it to achieve it. I guess the assumptions is that the AI will be much more human in some way.
The power-seeking, agentic, deceptive AI is only possible if there is a smooth transition from non-agentic AI (what we have right now) to agentic AI. Otherwise, there will be a sign that AI is agentic, and it will be observed for those capabilities. If an AI is mimicking human thinking process, which it might initially do, it will also mimic our biases and things like having pent-up feelings, which might cause it to slip and loose its temper. Therefore, it’s not likely that power-seeking agentic AI is a real threat (initially).
Interesting! I wonder what makes peopel feel like LLMs get them. I for sure don’t feel like Claude gets me. If anything, the opposite.
EDIT: Deepseek totally gets me tho