Zvi’s latest newsletter has a section on this topic! https://thezvi.substack.com/i/151331494/good-advice
TheManxLoiner
Karma: 99
+1 to you continuing with this series.
Pedantic point. You say “Automating AI safety means developing some algorithm which takes in data and outputs safe, highly-capable AI systems.” I do not think semi-automated interpretability fits into this, as the output of interpretability (currently) is not a model but an explanation of existing models.
Unclear why Level (1) does not break down into the ‘empirical’ vs ‘human checking’. In particular, how would this belief obtained: “The humans are confident the details provided by the AI systems don’t compromise the safety of the algorithm.”
Unclear (but good chance I just need to think more carefully through the concepts) why Level (3) does not collapse to Level (1) too, using same reasoning. Might be related to Martin’s alternative framing.
Couple of thoughts:
1. I recently found out about this new-ish social media platform. https://www.heymaven.com/. Good chance they are researching alternative recommendation algorithms.
2. What particular actions do you think rationality/ea community could do that other big efforts have not done enough, e.g. projects by Tristan Harris or Jaron Lanier.
Thanks for the feedback! Have editted the post to include your remarks.
Scattered thoughts on what it means for an LLM to believe
The ‘evolutionary pressures’ being discussed by CGP Grey is not the direct gradient descent used to train an individual model. Instead, he is referring to the whole set of incentives we as a society put on AI models. Similar to memes—there is no gradient descent on memes.
(Apologies if you already understood this, but it seems your post and Steven Byrne’s post focus on training of individual models)
AI as a powerful meme, via CGP Grey
What is the status of this project? Are there any estimates of timelines?
Totally agree! This is my big weakness right now—hopefully as I read more papers I’ll start developing a taste and ability to critique.
Distillation of ‘Do language models plan for future tokens’
How to build a data center, by Construction Physics
AI Safety Institute’s Inspect hello world example for AI evals
Huge thanks for writing this! Particularly liked the SVD intuition and how it can be used to understand properties of . One small correction I think. You wrote:
is simply the projection along the vector
I think is projection along the vector , so is projection on hyperplane perpendicular to
My experience at ML4Good AI Safety Bootcamp
Interesting ideas, and nicely explained! Some questions:
1) First notation: request patching means replacing the vector at activation A for R2 on C2 with vector at same activation A for R1 on C1. Then the question: Did you do any analysis on the set of vectors A as you vary R and C? Based on your results, I expect that the vector at A is similar if you keep R the same and vary C.
2) I found the success on the toy prompt injection surprising! My intuition up to that point was that R and C are independently represented to a large extent, and you could go from computing R2(C2) to R1(C2) by patching R1 from computation of R1(C1). But the success on preventing prompt injection means that corrupting C is somehow corrupting R too, meaning that C and R are actually coupled. What is your intuition here?
3) How robust do you think the results are if you make C and R more complex? E.g. C contains multiple characters who come from various countries but live in same city and R is ‘Where does character Alice come from’?
No need to apologise! I missed your response by even more time...
My instinct is that it is because of the relative size of the numbers, not the absolute size.
It might be an interesting experiment to see how the intuition varies based on the ratio of the total amount to the difference in amounts: “You have two items whose total cost is £1100 and the difference in price is £X. What is the price of the more expensive item?”, where X can be 10p or £1 or £10 or £100 or £500 or £1000.
With X=10p, one possible instinct is ’that means they are basically the same price, so the more expensive item is £550 + 10p = £550.10.
I have the same experience as you, drossbucket: my rapid answer to (1) was the common incorrect answer, but for (2) and (3) my intuition is well-honed.
A possible reason for this is that the intuitive but incorrect answer in (1) is a decent approximation to the correct answer, whereas the common incorrect answers in (2) and (3) are wildly off the correct answer. For (1) I have to explicitly do a calculation to verify the incorrectness of the rapid answer, whereas in (2) and (3) my understanding of the situation immediately rules out the incorrect answers.
Here are questions which might be similar to (I):
(4a) I booked seats J23 to J29 in a cinema. How many seats have I booked?
(4b) There is a 20m fence in which the fence posts are 2m apart. How many fence posts are there?
(4c) How many numbers are there in this list: 200,201,202,203,204,...,300.
(5) In 24 hours, how many times do the hour-hand and minute-hand of a standard clock overlap?
(6) You are in a race and you just overtake second place. What is your new position in the race?
The physical object Eiffel Tower is correlated with itself.
It is highly predictive of the ability of the LLM to book flights to Paris, when I create an LLM-agent out of it and ask it to book a trip to see the Eiffel Tower.
I dont think we disagree here. To clarify, my belief is there are threat models / solutions that are not affected by whether the AI has ‘real’ beliefs, and there are other threats/solutions where it does matter.
I actually do not understand the distinction between Definition 2 and Definition 3. Don’t need to resolve it here. I’ve editted post to include my uncertainty on this.