jbash

Karma: 2,432

jbash 23 Dec 2024 19:40 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: When AI 10x’s AI R&D, What Do We Do?
OK, sorry. That’s slightly below “top 3 priorities for the spies”, I think, but I still don’t think it’s reasonable to expect to protect a file that’s in use against it for 2 years.

jbash 23 Dec 2024 18:41 UTC
9 points
2
in reply to: ryan_greenblatt’s comment on: When AI 10x’s AI R&D, What Do We Do?
You mean an effort that’s in the top 3 priorities for the entire Chinese state? Like up there at the same level with “maintain the survival of the state”, and above stuff like “get Taiwan back” or “avoid unrest over the economy”?

You’re not going to get 80 percent two-year protection against that, period. The measures that RAND describes wouldn’t do it, and I don’t think any other measures would either (short of just not creating the target).

I doubt those measures would work even if it were just a top 3 priority for only the spies. In fact, they left out a whole bunch of standard “spy” techniques.

Also, realistically, those measures would never be adopted by anybody. The isolation requirement alone would make them a non-starter, and so would the supply chain requirements. Notice that I’m not saying those things aren’t needed. I’m saying you won’t get them. Before I escaped the formal security world, I had a lot of bitter experience saying “If you want the level of assurance you claim to want, the only way to get it is X”, and being told “we’re not doing X”. Usually followed by them implementing some deeply inadequate substitute and pretending it was equivalent^[1].

By the way, I asked “which SL5”, because n-level models are a dime a dozen, and “SL” is short and generic and likely to get used a lot. I was guessing you meant some extension to ISA/IEC 62443 (which goes up to SL4, but I could imagine people working on an SL5), or maybe the Lloyd’s maturity model. I’m pretty sure I’ve seen other documents that had “SLn” structures, although I can’t name them offhand. Or maybe it was “CL”, or “ML”, or “AL”, or all of the above. People from the more generic security world are going to get confused if you just talk about “SL5” without saying where it comes from.
1. ↩︎
  … and when I start seeing stuff like “Strict limitation of external connections to the completely isolated network”, I tend to think I’m seeing the beginnings of that process…

jbash 23 Dec 2024 18:10 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: When AI 10x’s AI R&D, What Do We Do?

fully state proof security

That’s not a thing. Arguably fully anything-proof security is not a thing, but fully state-proof security definitely isn’t.

Also, formalistic documents that try to define “security levels” tend to equate “security” with “enforcement of policy”. Even if you had perfect enforcement, it’s not obvious you could write a perfect policy to enforce.

(SL5)

Which SL5?

jbash 21 Dec 2024 16:25 UTC
28 points
5
in reply to: Thane Ruthenis’s comment on: o3

Not to say it’s a nothingburger, of course. But I’m not feeling the AGI here.

These math and coding benchmarks are so narrow that I’m not sure how anybody could treat them as saying anything about “AGI”. LLMs haven’t even tried to be actually general.

How close is “the model” to passing the Woz test (go into a strange house, locate the kitchen, and make a cup of coffee, implicitly without damaging or disrupting things)? If you don’t think the kinesthetic parts of robotics count as part of “intelligence” (and why not?), then could it interactively direct a dumb but dextrous robot to do that?

Can it design a nontrivial, useful physical mechanism that does a novel task effectively and can be built efficiently? Produce usable, physically accurate drawings of it? Actually make it, or at least provide a good enough design that it can have it made? Diagnose problems with it? Improve the design based on observing how the actual device works?

Can it look at somebody else’s mechanical design and form a reasonably reliable opinion about whether it’ll work?

Even in the coding domain, can it build and deploy an entire software stack offering a meaningful service on a real server without assistance?

Can it start an actual business and run it profitably over the long term? Or at least take a good shot at it? Or do anything else that involves integrating multiple domains of competence to flexibly pursue possibly-somewhat-fuzzily-defined goals over a long time in an imperfectly known and changing environment?

Can it learn from experience and mistakes in actual use, without the hobbling training-versus-inference distinction? How quickly and flexibly can it do that?

When it schemes, are its schemes realistically feasible? Can it tell when it’s being conned, and how? Can it recognize an obvious setup like “copy this file to another directory to escape containment”?

Can it successfully persuade people to do specific, relatively complicated things (as opposed to making transparently unworkable hypothetical plans to persuade them)?

jbash 20 Dec 2024 18:07 UTC
15 points
0
in reply to: Dagon’s comment on: TheManxLoiner’s Shortform
There is an option for readers to hide names. It’s in the account preferences. The names don’t show up unless you roll over them. I use it, to supplement my long-cultivated habit of always trying to read the content before the author name on every site^[1].

As for anonymous posts, I don’t agree with your blanket dismissal. I’ve seen them work against groupthink on some forums (while often at the same time increasing the number of low-value posts you have to wade through). Admittedly Less Wrong doesn’t seem to have too much of a groupthink problem^[2]. Anyway, there could always be an option for readers to hide anonymous posts.
1. ↩︎
  Actually I’m not sure I had to cultivate it. Back in the days of Usenet, I had to learn to actually ever look at poster’s names to begin with. I do not think that I am normal in this.
2. ↩︎
  … which actually surprises me because at least some people do seem to buy into the “karma” gamification.

jbash 26 Nov 2024 13:41 UTC
4 points
−2
on: Filled Cupcakes
Stretching your mouth wide is part of the fun!

jbash 25 Nov 2024 16:00 UTC
2 points
0
on: Decorated pedestrian tunnels
If you’re going to do something that huge, why not put the cars underground? I suppose it would be more expensive, but adding any extensive tunnel system at all to an existing built up area seems likely to be prohibitively expensive, tremendously disruptive. and, at least until the other two are fixed, politically impossible. So why not go for the more attractive impossibility?

jbash 21 Nov 2024 16:25 UTC
2 points
0
on: AI #91: Deep Thinking
Why so small? If you’re going to offer wall mounts and charge $1000, why not a TV-sized device that is also actually a television, or at least a full computer monitor? What makes this not want to simply be a Macintosh? I don’t fully ‘get it.’
You don’t necessarily have a TV-sized area of wall available to mount your thermostat control, near where you most often find yourself wanting to change your thermostat setting. Nor do you necessarily want giant obtrusive screens all over the place.
And you don’t often want to have to navigate a huge tree of menus on a general-purpose computer to adjust the music that’s playing.

jbash 21 Nov 2024 16:21 UTC
14 points
−1
on: AI #91: Deep Thinking
“Aren’t we going to miss meaning?”
I’ve yet to hear anybody who brings this up explain, comprehensibly, what this “meaning” they’re worried about actually is. Honestly I’m about 95 percent convinced that nobody using the word actually has any real idea what it means to them, and more like 99 percent sure that no two of them agree.

jbash 16 Nov 2024 17:11 UTC
33 points
3
in reply to: jbash’s comment on: OpenAI Email Archives (from Musk v. Altman)
I seem to have gotten a “Why?” on this.

The reason is that checking things yourself is a really, really basic, essential standard of discourse^[1]. Errors propagate, and the only way to avoid them propagating is not to propagate them.

If this was created using some standard LLM UI, it would have come with some boilerplate “don’t use this without checking it” warning^[2]. But it was used without checking it… with another “don’t use without checking” warning. By whatever logic allows that, the next person should be able to use the material, including quoting or summarizing it, without checking either, so long as they include their own warning. The warnings should be able to keep propagating forever.

… but the real consequences of that are a game of telphone:
1. An error can get propagated until somebody forgets the warning, or just plain doesn’t feel like including the warning, and then you have false claims of fact circulating with no warning at all. Or the warning deteriorates into “sources claim that”, or “there are rumors that”, or something equally vague that can’t be checked.
2. Even if the warning doesn’t get lost or removed, tracing back to sources gets harder with each step in the chain.
3. Many readers will end up remembering whatever they took out of the material, including that it came from a “careful” source (because, hey, they were careful to remind you to check up on them)… but forget that they were told it hadn’t been checked, or underestimate the importance of that.
4. If multiple people propagate an error, people start seeing it in more than one “independent” source, which really makes them start to think it must be true. It can become “common knowledge”, at least in some circles, and those circles can be surprisingly large.
That pollution of common knowledge is the big problem.

The pollution tends to be even worse because whatever factoid or quote will often get “simplified”, or “summarized”, or stripped of context, or “punched up” at each step. That mutation is itself exacerbated by people not checking references, because if you check references at least you’ll often end up mutating the version from a step or two back, instead of building even higher on top of the latest round of errors.

All of this is especially likely to happen when “personalities” or politics are involved. And even more likely to happen when people feel a sense of urgency about “getting this out there as soon as possible”. Everybody in the chain is going to feel that same sense of urgency.

I have seen situations like that created very intentionally in certain political debates (on multiple different topics, all unrelated to anything Less Wrong generally cares about). You get deep chains of references that don’t quite support what they’re claimed to support, spawning “widely known facts” that eventually, if you do the work, turn out to be exaggerations of admitted wild guesses from people who really didn’t have any information at all. People will even intentionally add links to the chain to give others plausible deniability. I don’t think there’s anything intentional here, but there’s a reason that some people do it intentionally. It works. And you can get away with it if the local culture isn’t demanding rigorous care and checking up at every step.

You can also see this sort of thing as an attempt to claim social prestige for a minimal contribution. After all, it would have been possible to just post the link, or post the link and suggest that everybody get their AI to summarize it. But the main issue is that spreading unverified rumors causes widespread epistemic harm.
1. ↩︎
  The standard for the reader should still be “don’t be sure the references support this unless you check them”, which actually means that when the reader becomes a writer, that reader/writer should actually not only have checked their own references, but also checked the references of their references, before publishing anything.
2. ↩︎
  Perhaps excusable since nobody actually knows how to make the LLM get it right reliably.

jbash 16 Nov 2024 14:24 UTC
7 points
−34
on: OpenAI Email Archives (from Musk v. Altman)

I used AI assistance to generate this, which might have introduced errors.

Resulting in a strong downvote and, honestly, outright anger on my part.

Check the original source to make sure it’s accurate before you quote it: https://www.courtlistener.com/docket/69013420/musk-v-altman/ [1]

If other people have to check it before they quote it, why is it OK for you not to check it before you post it?

jbash 15 Nov 2024 14:42 UTC
−3 points
0
on: Proposing the Conditional AI Safety Treaty (linkpost TIME)

Fortunately, Nobel Laureate Geoffrey Hinton, Turing Award winner Yoshua Bengio, and many others have provided a piece of the solution. In a policy paper published in Science earlier this year, they recommended “if-then commitments”: commitments to be activated if and when red-line capabilities are found in frontier AI systems.

So race to the brink and hope you can actually stop when you get there?

Once the most powerful nations have signed this treaty, it is in their interest to verify each others’ compliance, and to make sure uncontrollable AI is not built elsewhere, either.

How, exactly?

jbash 14 Nov 2024 20:15 UTC
7 points
1
on: Heresies in the Shadow of the Sequences

Non-causal decision theories are not necessary for A.G.I. design.

I’ll call that and raise you “No decision theory of any kind, causal or otherwise, will either play any important explicit role in, or have any important architectural effect over, the actual design of either the first AGI(s), or any subsequent AGI(s) that aren’t specifically intended to make the point that it’s possible to use decision theory”.

jbash 12 Nov 2024 14:14 UTC
LW: 20 AF: 8
3
AF
in reply to: Buck’s comment on: Buck’s Shortform

Computer security, to prevent powerful third parties from stealing model weights and using them in bad ways.

By far the most important risk isn’t that they’ll steal them. It’s that they will be fully authorized to misuse them. No security measure can prevent that.

jbash 11 Nov 2024 17:12 UTC
LW: 17 AF: 9
9
AF
on: The Evals Gap

Development and interpretation of evals is complicated

Proper elicitation is an unsolved research question

… and yet...

Closing the evals gap is possible

Why are you sure that effective “evals” can exist even in principle?

I think I’m seeing a “we really want this, therefore it must be possible” shift here.

jbash 10 Nov 2024 14:36 UTC
0 points
−11
in reply to: evhub’s comment on: evhub’s Shortform
I don’t have much trouble with you working with the US military. I’m more worried about the ties to Peter Thiel.

jbash 9 Nov 2024 15:45 UTC
2 points
0
on: GPT-4o Can In Some Cases Solve Moderately Complicated Captchas
CAPTCHAs have “adversarial perturbations”? Is that in the sense of “things not visible to humans, but specifically adversarial to deep learning networks”? I thought they just had a bunch of random noise and weird ad hoc patterns thrown over them.

Anyway, CAPTCHAs can’t die soon enough. Although the fact that they persist in the face of multiple commercial services offering to solve 1000 for a dollar doesn’t give me much hope...

jbash 9 Nov 2024 15:32 UTC
4 points
2
on: Force Sequential Output with SCP?
Using scp to stdout looks weird to me no matter what. Why not

ssh -n host cat /path/to/file | weird-aws-stuff

… but do you really want to copy everything twice? Why not run weird-aws-stuff on the remote host itself?

jbash 7 Nov 2024 15:04 UTC
17 points
0
on: Should CA, TX, OK, and LA merge into a giant swing state, just for elections?

To prevent this, there must be a provision that once signed by all 4 states, the compact can’t be repealed by any state until after the next election.

It’s not obvious that state legislatures have the authority, under their own constitutions, to bind themselves that way. Especially not across their own election cycles.

jbash 6 Nov 2024 1:33 UTC
3 points
0
in reply to: tslarm’s comment on: Update on the Mysterious Trump Buyers on Polymarket
Thirty million dollars is a lot of money, but there are plenty of smart rich people who don’t mind taking risks. So, once the identity and (apparent) motives of the Trump whale were revealed, why didn’t a handful of them mop up the free EV?

Well, first I think you’re right to say “a handful”. My (limited but nonzero) experience of “sufficiently rich” people who made their money in “normal” ways, as opposed to by speculating on crypto or whatever, is that they’re too busy to invest a lot of time in playing this kind of market personally, especially if they have to pay enough attention to play it intelligently. They’re not very likely to employ anybody else to play for them either. Many or most of them will see as the whole thing as basically an arcane, maybe somewhat disreputable game. So the available pool is likely smaller than you might think.

That conjecture is at least to some degree supported by the fact that nobody, or not enough people, stepped in when the whole thing started. Nothing prevented the market from moving so far to begin with. It may not have been as certain what was going on then, but things looked weird enough that you’d expect a fair number of people to decide that crazy money was likely at work, and step in to try to take some of it… if enough such people were actually available.

In any case, whether when the whole thing started, after public understanding was reasonably complete, or anywhere along the way, the way I think you’d like to make your profit on the market being miscalibrated would be to buy in, wait for the correction, and then sell out… before the question resolved and before unrelated new information came in to move the price in some other way.

But would be hard to do that. All this is happening potentially very close to resolution time, or at least to functional resolution time. The market is obviously thin enough that single traders can move it, and new information is coming in all the time, and the already-priced-in old information isn’t very strong and therefore can’t be expected to “hold” the price very solidly, and you have to worry about who may be competing with you to take the same value, and you may be questioning how rational traders in general are^[1].

So you can’t be sure you’ll get your correction in time to sell out; you have a really good chance of being stuck holding hold your position through resolution. If “markets can remain irrational longer than you can remain solvent”, then they can also stay irrational for long enough that trading becomes moot.

If you have to hold through resolution, then you do still believe you have positive expected value, but it’s really uncertain expected value. After all, you believe the underlying question is 50-50, even if one of those 50s would pay you more than the other would lose you. And you have at best limited chance to hedge. So you have to have risk tolerance high enough that, for most people, it’d be in the “recreational gambling” range rather than the “uncertain investment” range. The amount of money that any given (sane) person wants to put at risk definitely goes down under that much uncertainty, and probably goes down a lot. So you start to need more than a “handful” of people.

Also, don’t forget the point somebody made the other day about taxes. Unless you’re a regular who plays many different questions in such volume that you expect to offset your winnings with losses, you’re going to stand to “win” a double-digit percentage less than you stand to “lose”, whether on selling off your position or on collecting after resolution. Correcting 60-40 to 50-50 may just plain not be profitable even if you collect.

There are probably other sources of friction, too.
1. ↩︎
  I’d bet at least a recreational amount of money that players in betting markets are sharply more pro-Trump politically than, say, the general voting population, and that would be expected to skew their judgement, and therefore the market, unless almost all of them were superhuman or nearly so. And when you’re seeing the market so easily moved away from expert opinion…