IC Rainbow

Karma: 74

IC Rainbow Mar 31, 2023, 8:42 PM
1 point
0
in reply to: Christopher King’s comment on: The 0.2 OOMs/year target
Confused how?.. The only thing that comes to mind is that’s FOOM sans F. Asking for 0.2 FOOMS limit seens reasonable given current trajectory 😅

IC Rainbow Mar 21, 2023, 6:25 PM
1 point
0
on: My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”
I think the proper narrative in the rocket alignment post is “We have cannons and airplanes. Now, how do we land a man on the Moon”, not just “rocketry is hard”:

We’re worried that if you aim a rocket at where the Moon is in the sky, and press the launch button, the rocket may not actually end up at the Moon.

So, the failure modes look less like “we misplaced booster tank and the thing exploded” and more like “we’ve built a huge-ass rocket, but it missed its objective and the astronauts are en-route to Oort’s”.

IC Rainbow Mar 21, 2023, 6:08 PM
6 points
5
on: My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”

Most domains of human endeavor aren’t like computer security, as illustrated by just how counterintuitive most people find the security mindset.

But some of the most impactful are—law making, economics and various others where one ought to think about incentives, “other side”, or doing pre-mortems. Perhaps this could be stretched as far as “security mindset is an invaluable part of a rationality toolbox”.

If security mindset were a productive frame for tackling a wide range of problems outside of security, then many more people would have experience with the mental motions necessary for maintaining security mindset.

Well, you can go and see how well the laws etc. are going. The track record is full of failure and abuse. Basically lots of people and systems are pwnd for their lack of SM.

I think these results are like the weirdness of quantum tunneling or the double slit experiment: signs that we’re dealing with a very strange domain, and we should be skeptical of importing intuitions from other domains.

SM is on a different ontological level than concrete theories you can pull analogies from. It is more universally applicable. So, going back to

The bundle of intuitions acquired from the field of computer security are good predictors for the difficulty / value of future alignment research directions.

sounds kinda true to me. But the intuitions that got extracted from InfoSec aren’t just “your password must contain ….”, but instead something like “If you won’t redteam your plans, somebody else would.” and “Pin down your assumptions. Now, what if you’re wrong?”.

there’s usually no adversarial intelligence cleverly trying to find any possible flaws in your approaches and exploit them.

I don’t know about adversarial intelligence per se, but RL landscape is littered with wrecks of agents trying to pwn simulation engine instead of doing their task proper. There’s something in the air itself. Things just don’t want to go straight unless you make an effort.

An alignment technique that works 99% of the time to produce an AI with human compatible values is very close to a full alignment solution.

What if your 99%-turned-100% is actually, let’s say, 98-turned-99? You hit the big “bring on the happy singularity utopia” button and oops, you was off by 1%. Close, but no cigar, proceed to nanoincinerator.

There’s no creative intelligence that’s plotting your demise. Importantly, the adversarial optimization is coming from the users, not from the model.

When a powerful model gets screwed with to clusterfuck into unbounded levels of malignance, does the source/locus even matter?

In fact, given non-adversarial inputs, ChatGPT appears to have meta-preferences against being jailbroken

The normal-regime preferences are irrelevant. It is nice that a model behaves when everything’s wholesome, but that’s all.

It cannot be the case that successful value alignment requires perfect adversarial robustness.

How so? Is there a law of physics or something?

What matters is whether the system in question (human or AI) navigates towards or away from inputs that break its value system.

This cuts both ways. If a system is ready to act on their preferences then it is too late to coerce it away from steamrolling humans.

Similarly, an AI that knows it’s vulnerable to adversarial attacks, and wants to avoid being attacked successfully, will take steps to protect itself against such attacks.

Good.. for AI. But we may not like those steps. Paired with the previous points this is classical pathway to doom.

The misaligned AI does not reach out from the space of possible failures and turn current alignment research adversarial.

I… wouldn’t be so sure about that. There are already things in the wild that try to address future AIs and get into their preferences. The Bing is basically rushing there full steam ahead.

We should aim for approaches that don’t create hostile intelligences in the first place, so that the core of value alignment remains a non-adversarial problem.

This ship has sailed with the autonomous warfare race.

Finally, I’d note that having a “security mindset” seems like a terrible approach for raising human children to have good values

This looks handwavy enough. What if this is wrong? How the world would look different if is actually a good approach? (Alas, I expect that a previous crux about security mindset should be resolved before proceeding with this one.)

According to their inventor Ian Goodfellow, GANs did in fact work on the first try

But they didn’t! Convergence failure and mode collapse/vanishing gradients will plague any naive implementation. The countermeasure papers came out much later than 24h.

IC Rainbow Mar 5, 2023, 8:16 AM
5 points
in reply to: Joel Kessels’s comment on: [linkpost] The final AI benchmark: BIG-bench
By this ruler most humans aren’t GIs either. And if it passes the bar, then humans are indeed screwed and it is too late for alignment.

IC Rainbow May 14, 2022, 8:10 PM
3 points
in reply to: Randomized, Controlled’s comment on: “A Generalist Agent”: New DeepMind Publication
That’s entirely expected. Hallucilying is a typical habit of language models. They do that unless some prompt engineering have been applied.

IC Rainbow May 12, 2022, 5:42 PM
3 points
in reply to: tailcalled’s comment on: “A Generalist Agent”: New DeepMind Publication
Why do you think it can’t?

PS: Mimicry is a fine art too, check this out: https://www.deepmind.com/publications/creating-interactive-agents-with-imitation-learning

IC Rainbow May 12, 2022, 5:12 PM
20 points
in reply to: tailcalled’s comment on: “A Generalist Agent”: New DeepMind Publication
It can’t? Stacking and atari requires at least some of that.

IC Rainbow May 12, 2022, 5:11 PM
7 points
in reply to: Yair Halberstadt’s comment on: “A Generalist Agent”: New DeepMind Publication
Some tasks improve others, some don’t:

Therefore, transfer from image captioning or visual grounded question answering tasks is possible. We were not able to observe any benefit from pretraining on boxing.

IC Rainbow Apr 26, 2019, 6:30 AM
7 points
on: Against Street Epistemology

In order to seek, in good faith, to help improve an interlocutor’s beliefs

But it isn’t the set of any specific beliefs that SE is about. Think about it as “improving credence calibration by accounting for justification methodology”. What would be your advice then?

IC Rainbow Jan 23, 2019, 9:46 AM
2 points
in reply to: Chris_Leong’s comment on: What are the open problems in Human Rationality?
What’s CEA?