wassname

Karma: 152

wassname 11 May 2024 9:11 UTC
4 points
0
in reply to: lePAN6517’s comment on: simeon_c’s Shortform
One is the change to the charter to allow the company to work with the military.

https://news.ycombinator.com/item?id=39020778

wassname 9 May 2024 23:37 UTC
1 point
0
in reply to: William_S’s comment on: William_S’s Shortform
Here’s something I’ve been pondering.

hypothesis: If transformers has internal concepts, and they are represented in the residual stream. Then because we have access to 100% of the information then it should be possible for a non-linear probe to get 100% out of distribution accuracy. 100% is important because we care about how a thing like value learning will generalise OOD.

And yet we don’t get 100% (in fact most metrics are much easier than what we care about, being in-distribution, or on careful setups). What is wrong with the assumptions hypothesis, do you think?

wassname 9 May 2024 23:30 UTC
1 point
0
in reply to: JenniferRM’s comment on: William_S’s Shortform

better calibrated than any of these opinions, because most of them don’t seem to focus very much on “hedging” or “thoughtful doubting”

new observations > new thoughts when it comes to calibrating yourself.

The best calibrated people are people who get lots of interaction with the real world, not those who think a lot or have a complicated inner model. Tetlock’s super forecasters were gamblers and weathermen.

wassname 8 May 2024 23:47 UTC
1 point
0
in reply to: faul_sname’s comment on: We are headed into an extreme compute overhang

I think this only holds if fine tunes are composable, which as far as I can tell they aren’t

Anecdotally, a lot of people are using mergekit to combine fine tunes

wassname 5 May 2024 22:25 UTC
1 point
0
AF
in reply to: Neel Nanda’s comment on: Refusal in LLMs is mediated by a single direction

it feels less surgical than a single direction everywher

Agreed, it seems less elegant, But one guy on huggingface did a rough plot the cross correlation, and it seems to show that the directions changes with layer https://huggingface.co/posts/Undi95/318385306588047#663744f79522541bd971c919. Although perhaps we are missing something.

Note that you can just do torch.save(FILE_PATH, model.state_dict()) as with any PyTorch model.

omg, I totally missed that, thanks. Let me know if I missed anything else, I just want to learn.

The older versions of the gist are in transformerlens, if anyone wants those versions. In those the interventions work better since you can target resid_pre, redis_mid, etc.

wassname 5 May 2024 7:57 UTC
LW: 1 AF: -1
0
AF
on: Refusal in LLMs is mediated by a single direction
If anyone wants to try this on llama-3 7b, I converted the collab to baukit, and it’s available here.

wassname 29 Apr 2024 0:48 UTC
1 point
0
in reply to: Andy Arditi’s comment on: Refusal in LLMs is mediated by a single direction
So I ran a quick test (running llama.cpp perplexity command on wiki.test.raw )
- base_model (Meta-Llama-3-8B-Instruct-Q6_K.gguf): PPL = 9.7548 +/- 0.07674
- steered_model (llama-3-8b-instruct_activation_steering_q8.gguf): 9.2166 +/- 0.07023
So perplexity actually lowered, but that might be because the base model I used was more quantized. However, it is moderate evidence that the output quality decrease from activation steering is lower than that from Q8->Q6 quantisation.

I must say, I am a little surprised by what seems to be the low cost of activation editing. For context, many of the Llama-3 finetunes right now come with a measurable hit to output quality. Mainly because they are using worse fine tuning data, than the data llama-3 was originally fine tuned on.

wassname 28 Apr 2024 6:54 UTC
7 points
1
in reply to: Andy Arditi’s comment on: Refusal in LLMs is mediated by a single direction

maintaining model coherence

To determine this, I believe we would need to demonstrate that the score on some evaluations remains the same. A few examples don’t seem sufficient to establish this, as it is too easy to fool ourselves by not being quantitative.

I don’t think DanH’s paper did this either. So I’m curious, in general, whether these models maintain performance, especially on measures of coherence.

In the open-source community, they show that modifications retain, for example, the MMLU and HumanEval score.

wassname 21 Apr 2024 11:08 UTC
1 point
0
on: AI Alignment: A Comprehensive Survey
This is pretty good. It has a lot in it, being a grab bag of things. I particularly enjoyed the scalable oversight sections which succinctly explained debate, recursive reward modelling etc. There were also some gems I hadn’t encountered before, like the concept of training out agentic behavior by punishing side-effects.

If anyone wants the HTML version of the paper, it is here.

wassname 19 Apr 2024 23:45 UTC
4 points
0
on: Evolution did a surprising good job at aligning humans...to social status
Maybe our culture fits our status-seeking surprisingly well because our culture was designed around it.

We design institutions to channel and utilize our status-seeking instincts. We put people in status conscious groups like schools, platoons, or companies. There we have ceremonies and titles that draw our attention to status.

And this works! Ask yourself, is it more effective to educate a child individually or in a group of peers? The latter. Is it easier to lead a solitary soldier or a whole squad? The latter. Do people seek a promotion or a pay rise? Both, probably. The fact is, that people are easier to guide when in large groups, and easier to motivate with status symbols.

From this perspective, our culture and inclination for seeking status have developed in tandem, making it challenging to determine which influences the other more. However, it appears that culture progresses more rapidly than genes, suggesting that culture conforms to our genes, rather than the reverse.

Another perspective: Sometimes our status seeking is nonfunctional and therefore nonaligned. For example we also waste a lot of effort on status, which seems like a nonfunctional drive. People will compete for high status professions like musician, streamer, celebrity and most will fail, which makes it seem like an unwise investment of time. This seems misaligned, as it’s not adaptive.

wassname 12 Apr 2024 6:19 UTC
1 point
0
in reply to: VesaVanDelzig’s comment on: Is CIRL a promising agenda?

would probably eventually stop treating you as a source of new information once it had learned a lot from you, at which point it would stop being deferential.

It seems that 1) when extrapolating to new situations 2) if you add a term to decay the relevance of old information (pretty standard in RL) 3) or you add a minimum bounds to uncertainty then it would remain deferential.

In other words, it doesn’t seem like an unsolvable problem, just an open question. But every other alignment agenda also has numerous open questions. So why the hostility.

Academia and LessWrong are two different groups, which have different cultures and jargon. I think they may be overly skeptical towards each other’s work at times.

It’s worth noting though that many of the nice deferential properties may appear in other value modelling techniques (like recursive reward modelling at OpenAI).

wassname 12 Apr 2024 5:54 UTC
1 point
0
on: The Genie in the Bottle: An Introduction to AI Alignment and Risk
A nice introductory essay, seems valuable for entrants.

There quite a few approaches to alignment beyond CIRL and Value Lear ing. https://www.lesswrong.com/posts/zaaGsFBeDTpCsYHef/shallow-review-of-live-agendas-in-alignment-and-safety

wassname 11 Apr 2024 9:18 UTC
3 points
0
on: Is CIRL a promising agenda?
There’s some recent academic research on CIRL which is overlooked on LessWrong, Here we seem to only discuss Stuart Russell’s work.

Recent work:
- 2024 work by @xuanalogue and team
- 2023 Modeling Boundedly Rational Agents with Latent Inference Budgets Brunke et al., 2022, “Safe learning in robotics: From learning-based control to safe reinforcement learning”
See also this overviews in lecture 3 and 4 of Roger Gross’s CSC2547 Alignment Course.

The most interesting feature is deference: that you can have a pathologically uncertain agent that constantly seeks human input. As part of its uncertainty, It’s also careful how it goes about this seeking of input. For example, if it’s unsure if humans like to be stabbed (we don’t), it wouldn’t stab you to see your reaction, that would be risky! Instead, it would ask or seek out historical evidence.

This is an important safety feature which slows it down, grounds it, and helps avoid risky value extrapolation (and therefore avoids poor convergence).

It’s worth noting that CIRL sometimes goes by other names

Inverse reinforcement learning, inverse planning, and inverse optimal control, are all different names for the same problem: Recover some specification of desired behavior from observed behavior

It’s also highly related to both assistance games and Recursive Reward Modelling (part of OpenAI’s superalignment).

On the other hand, there are some old rebuttals of parts of it
- For example Eleizers article on updated deference talks about the shutdown problem and whether uncertainty really helps

wassname 5 Apr 2024 23:55 UTC
1 point
0
in reply to: charlieoneill’s comment on: Can quantised autoencoders find and interpret circuits in language models?

as long as training and eval error are similar

It’s just that eval and training are so damn similar, and all other problems are so different’t. So while it is technical not overfitting (to this problem), if is certainly overfitting to this specific problems, and it certainly isn’t measuring generalization in any sense of the word. Certainly not in the sense of helping us debug alignment for all problems.

This is an error that, imo, all papers currently make though! So it’s not a criticism so much as an interesting debate, and a nudge to use a harder test or OOD set in your benchmarks next time.

but you can’t say they’re more scalable than SAE, because SAEs don’t have to have 8 times the number of features

Yeah, good point. I just can’t help but think there must be a way of using unsupervised learning to force a compressed human-readable encoding. Going uncompressed just seems wasteful, and like it won’t scale. But I can’t think of a machine learnable, unsupervised learning, human-readable coding. Any ideas?

wassname 2 Apr 2024 23:14 UTC
1 point
0
in reply to: Freyja’s comment on: The Best Tacit Knowledge Videos on Every Subject
Thanks!

wassname 2 Apr 2024 11:41 UTC
1 point
0
in reply to: Freyja’s comment on: The Best Tacit Knowledge Videos on Every Subject
Interesting, got anymore? Especially for toddlers and so on, or would you go through everything those women have uploaded?

wassname 25 Mar 2024 5:54 UTC
3 points
0
in reply to: Kaj_Sotala’s comment on: ChatGPT can learn indirect control

learn an implicit rule like “if I can control it by an act of will, it is me

This was empirically demonstrated to be possible in this paper: “Curiosity-driven Exploration by Self-supervised Prediction”, Pathak et al

We formulate curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model.

It probably could be extended to learn “other” and the “boundary between self and other” in a similar way.

I implemented a version of it myself and it worked. This was years ago. I can only imagine what will happen when someone redoes some of these old RL algo’s, with LLM’s providing the world model.

wassname 25 Mar 2024 4:38 UTC
2 points
1
on: Can quantised autoencoders find and interpret circuits in language models?
What we really want with interpretability is: high accuracy, when out of distribution, scaling to large models. You got very high accuracy… but I have no context to say if this is good or bad. What could a naïve baseline get? And what do SAE’s get? Also it would be nice to see an Out Of Distribution set, because getting 100% on your test suggests that it’s fully within the training distribution (or that your VQ-VAE worked perfectly).

I tried something similar but only got half as far as you. Still my code may be of interest. I wanted to know if it would help with lie detection, out of distribution, but didn’t get great results. I was using a very hard setup where no methods work well.

I think VQ-VAE is a promising approach because it’s more scalable than SAE, which have 8 times the parameters of the model they are interpreting. Also your idea of using a decision tree on the tokenised space make a lot of sense given the discrete latent space!

wassname 24 Mar 2024 0:58 UTC
1 point
0
in reply to: ChristianKl’s comment on: The Worst Form Of Government (Except For Everything Else We’ve Tried)
In that way it’s similar to charter cities. An cheaper intermediate stage could be online organizations like World Of WarCraft gaming clans, or internet forums, or project overviews. They are not quite the same, but they are cheap.

wassname 23 Mar 2024 7:27 UTC
1 point
0
in reply to: ErioirE’s comment on: The Worst Form Of Government (Except For Everything Else We’ve Tried)
Yeah, it does seem tricky. OpenAI recently tried a unique governance structure and that it is tuning out unpredictable and might be costly in terms of legal fees and malfunction.,