Davidmanheim

Karma: 4,792

Davidmanheim 28 Jan 2025 15:53 UTC
1 point
−2
in reply to: Anthony DiGiovanni’s comment on: Should you go with your best guess?: Against precise Bayesianism and related views
“when, if ever, our credences ought to capture indeterminacy in how we weigh up considerations/evidence”

The obvious answer is only when there is enough indeterminacy to matter; I’m not sure if anyone would disagree. Because the question isn’t whether there is indeterminacy, it’s how much, and whether it’s worth the costs of using a more complex model instead of doing it the Bayesian way.
I’d be surprised if many/most infra-Bayesians would endorse suspending judgment in the motivating example in this post
You also didn’t quite endorse suspending judgement in that case—“If someone forced you to give a best guess one way or the other, you suppose you’d say “decrease”. Yet, this feels so arbitrary that you can’t help but wonder whether you really need to give a best guess at all…” So, yes, if it’s not directly decision relevant, sure, don’t pick, say you’re uncertain. Which is best practice even if you use precise probability—you can have a preference for robust decisions, or a rule for withholding judgement when your confidence is low. But if it is decision relevant, and there is only a binary choice available, your best guess matters. And this is exactly why Eliezer says that when there is a decision, you need to focus your indeterminacy, and why he was dismissive of DS and similar approaches.

Davidmanheim 28 Jan 2025 11:48 UTC
3 points
−1
on: Should you go with your best guess?: Against precise Bayesianism and related views
I’m not merely saying that agents shouldn’t have precise credences when modeling environments more complex than themselves

You seem to be underestimating how pervasive / universal this critique is—essentially every environment is more complex than we are, at the very least when we’re embedded agents, or other humans are involved. So I’m not sure where your criticism (which I agree with) is doing more than the basic argument is in a very strong way—it just seems to be stating it more clearly.

The problem is that Kolmogorov complexity depends on the language in which algorithms are described. Whatever you want to say about invariances with respect to the description language, this has the following unfortunate consequence for agents making decisions on the basis of finite amounts of data: For any finite sequence of observations, we can always find a silly-looking language in which the length of the shortest program outputting those observations is much lower than that in a natural-looking language (but which makes wildly different predictions of future data).
Far less confident here, but I think this isn’t correct as a mater of practice. Conceptually, Solomonoff doesn’t say “pick an arbitrary language once you’ve seen the data and then do the math” it says “pick an arbitrary language before you’ve seen any data and then do the math.” And if we need to implement the silly looking language, there is a complexity penalty to doing that, one that’s going to be similarly large regardless of what baseline we choose, and we can determine how large it is in reducing the language to some other language. (And I may be wrong, but picking a language cleverly should not means that Kolmogorov complexity will change something requiring NP programs to encode into something that P programs can encode, so this criticism seems weak anyways outside of toy examples.)

Davidmanheim 18 Jan 2025 21:59 UTC
2 points
0
in reply to: abramdemski’s comment on: Davidmanheim’s Shortform
Strongly agree. I was making a narrower point, but the metric is clearly different than the goal—if anything it’s more surprising that we see so much correlation as we do, given how much it has been optimized.

Davidmanheim 16 Jan 2025 8:23 UTC
3 points
0
on: Davidmanheim’s Shortform
Toby Ord writes that “the required resources [for LLM training] grow polynomially with the desired level of accuracy [measured by log-loss].” He then concludes that this shows “very poor returns to scale,” and christens it the “Scaling Paradox.” (He continues to point out that this doesn’t imply it can’t create superintelligence, but I agree with him about that.)
But what would it look like if this were untrue? That is, what would be the conceptual alternative, where required resources grow more slowly?I think the answer is that it’s conceptually impossible.
To start, there is a fundamental bound on loss at zero, since the best possible model perfectly predicts everything—it exactly learns the distribution. This can happen when overfitting a model, but it can also happen when there is a learnable ground truth; models that are trained to learn a polynomial function can learn them exactly.
But there is strong reason to expect the bound to be significantly above zero loss. The training data for LLMs contains lots of aleatory randomness, things that are fundamentally conceptually unpredictable. I think it’s likely that things like RAND’s random number book are in the training data, and it’s fundamentally impossible to predict randomness. I think something similar is generally true for many other things—predicting world choice for semantically equivalent words, predicting where typos occur, etc.
Aside from being bound well above zero, there’s a strong reason to expect that scaling is required to reduce loss for some tasks. In fact, it’s mathematically guaranteed to require significant computation to get near that level for many tasks that are in the training data. Eliezer pointed out that GPTs are predictors, and gives the example of a list of numbers followed by their two prime factors. It’s easy to generate such a list by picking pairs of primes and multiplying them, the writing the answer first—but decreasing loss for generating the next token to predict the primes from the product is definitionally going to require exponentially more computation to perform better for larger primes.
And I don’t think this is the exception, I think it’s at least often the rule. The training data for LLMs contains lots of data where the order of the input doesn’t follow the computational order of building that input. When I write an essay, I sometimes arrive at conclusions and then edit the beginning to make sense. When I write code, the functions placed earlier often don’t make sense until you see how they get used later. Mathematical proofs are another example where this would often be true.
An obvious response is that we’ve been using exponentially more compute for better accomplishing tasks that aren’t impossible in this way—but I’m unsure if that is true. Benchmarks keep getting saturated, and there’s no natural scale for intelligence. So I’m left wondering whether there’s any actual content in the “Scaling Paradox.”

(Edit: now also posted to my substack.)

Davidmanheim 15 Jan 2025 9:23 UTC
8 points
6
in reply to: plex’s comment on: AI Safety as a YC Startup
True, and even more, if optimizing for impact or magnitude has Goodhart effects, of various types, then even otherwise good directions are likely to be ruined by pushing on them too hard. (In large part because it seems likely that the space we care about is not going to have linear divisions into good and bad, there will be much more complex regions, and even when pointed in a directino that is locally better, pushing too far is possible, and very hard to predict from local features even if people try, which they mostly don’t.)

Davidmanheim 15 Jan 2025 9:20 UTC
2 points
0
in reply to: Thane Ruthenis’s comment on: AI Safety as a YC Startup
I think the point wasn’t having a unit norm, it was that impact wasn’t defined as directional, so we’d need to remove the dimensionality from a multidimensionally defined direction.

So to continue the nitpicking, I’d argue impact = || Magnitude * Direction ||, or better, ||Impact|| = Magnitude * Direction, so that we can talk about size of impact. And that makes my point in a different comment even clearer—because almost by assumption, the vast majority of those with large impact are pointed in net-negative directions, unless you think either a significant proportion of directions are positive, or that people are selecting for it very strongly, which seems not to be the case.

Davidmanheim 15 Jan 2025 8:23 UTC
14 points
−1
on: We probably won’t just play status games with each other after AGI
I think some of this is on target, but I also think there’s insufficient attention to a couple of factors.
First, in the short and intermediate term, I think you’re overestimating how much most people will actually update their personal feelings around AI systems. I agree that there is a fundamental reason that fairly near-term AI will be able to function as better companion and assistant than humans—but as a useful parallel, we know that nuclear power is fundamentally better than most other power sources that were available in the 1960s, but people’s semi-irrational yuck reaction to “dirty” or “unclean” radiation—far more than the actual risks—made it publicly unacceptable. Similarly, I think the public perception of artificial minds will be generally pretty negative, especially looking at current public views of AI. (Regardless of how appropriate or good this is in relation to loss-of-control and misalignment, it seems pretty clearly maladaptive for generally friendly near-AGI and AGI systems.)
Second, I think there is a paperclip maximizer aspect to status competition, in the sense Eliezer uses the concept. That is, Specifically, given massively increased wealth, abilities, and capacity, even if a implausibly large 99% of humans find great ways to enhance their lives in ways that don’t devolve into status competition, there are few other domains where an indefinite amount of wealth and optimization power can be applied usefully. Obviously, this is at best zero-sum, but I think there aren’t lots of obvious alternative places for positive sum indefinite investments. And even where such positive-sum options exist, they often are harder to arrive at as equilibria. (We see a similar dynamic with education, housing, and healthcare, where increasing wealth leads to competition over often artificially-constrained resources rather than expansion of useful capacity.)
Finally and more specifically, your idea that we’d see intelligence enhancement as a new (instrumental) goal in the intermediate term seems possible and even likely, but not a strong competitor for, nor inhibitor of, status competition. (Even ignoring the fact that intelligence itself is often an instrumental goal for status competition!) Even aside from the instrumental nature of the goal, I will posit that some strongly reduced returns to investment will exist—regardless of the fact that it’s unlikely on priors that these limits are near the current levels. Once those points are reached, the indefinite investment of resources will trade-off between more direct status competition and further intelligence increases, and as the latter shows decreased returns, as noted above, the former becomes the metaphorical paperclip which individuals can invest indefinitely into.

Davidmanheim 8 Jan 2025 16:50 UTC
10 points
0
on: AI Safety as a YC Startup
my uninformed intuition is that the people with the biggest positive impact on the world have prioritized the Magnitude
That’s probably true, but it’s selecting on the outcome variable. And I’ll bet that the people with the biggest negative impact are even more overwhelmingly also those who prioritized magnitude.

Davidmanheim 1 Jan 2025 15:00 UTC
4 points
1
in reply to: Richard121’s comment on: When Is Insurance Worth It?
“If you already know that an adverse event is highly likely for your specific circumstances, then it is likely that the insurer will refuse to pay out for not disclosing “material information”—a breach of contract.”

Having worked in insurance, that’s not what the companies usually do. Denying explicitly for clear but legally hard to defend reasons, especially those which a jury would likely rule against, isn’t a good way to reduce costs and losses. (They usually will just say no and wait to see if you bother following up. Anyone determined enough to push to get a reasonable claim is gonna be cheaper to pay out for than to fight.)

Davidmanheim 30 Dec 2024 0:46 UTC
2 points
0
in reply to: Ben Pace’s comment on: Alexander Gietelink Oldenziel’s Shortform
Yes—the word ‘global’ is a minimum necessary qualification for referring to catastrophes of the type we plausibly care about—and even then, it is not always clear that something like COVID-19 was too small an event to qualify.

Davidmanheim 29 Dec 2024 19:36 UTC
4 points
0
in reply to: martinkunev’s comment on: Refuting Searle’s wall, Putnam’s rock, and Johnson’s popcorn
I definitely appreciate that confusion. I think it’s a good reason to read the sequence and think through the questions clearly; https://www.lesswrong.com/s/p3TndjYbdYaiWwm9x—I think this resolves the vast majority of the confusion people have, even if it doesn’t “answer” the questions.

Davidmanheim 26 Dec 2024 22:22 UTC
6 points
0
in reply to: Ruby’s comment on: When Is Insurance Worth It?
The math is good, the point is useful, the explanations are fine, but the embracing straw vulcan version of rationality and dismissing any notion of people legitimately wanting things other than money seems really quite bad, which leaves me wishing this wasn’t being highlighted for visitors to the site.

Davidmanheim 22 Dec 2024 5:08 UTC
2 points
0
in reply to: ChristianKl’s comment on: Moderately Skeptical of “Risks of Mirror Biology”
I don’t understand. You shouldn’t get any changes from changing encoding if it produces the same proteins—the difference for mirror life is that it would also mirror proteins, etc.

Davidmanheim 21 Dec 2024 16:20 UTC
4 points
2
in reply to: Chris_Leong’s comment on: Announcement: AI for Math Fund
Plausibly, yes. But so does programming capability, which is actually a bigger deal. (And it’s unclear that a traditionally envisioned intelligence explosion is possible with systems built on LLMs, though I’m certainly not convinced by that argument.)

Davidmanheim 20 Dec 2024 13:41 UTC
2 points
0
in reply to: Amalthea’s comment on: Announcement: AI for Math Fund
But “creating safety guarantees into which you can plug in AI capabilities once they arise anyway” is the point, and it requires at least some non-trivial advances in AI capabilities.

You should probably read the current programme thesis.

Davidmanheim 20 Dec 2024 13:04 UTC
2 points
2
in reply to: Amalthea’s comment on: Announcement: AI for Math Fund
It is speculative in the sense that any new technology being developed is speculative—but closely related approaches are already used for assurance in practice, so provable safety isn’t actually just speculative, there are concrete benefits in the near term. And I would challenge you to name a different and less speculative framework that actually deals with any issues of ASI risks that isn’t pure hopium.
Uncharitably, but I think not entirely inaccurately, these include: “maybe AI can’t be that much smarter than humans anyways,” “let’s get everyone to stop forever,” “we’ll use AI to figure it out, even though we have no real ideas,” “we just will trust that no-one makes it agentic,” “the agents will be able to be supervised by other AI which will magically be easier to align,” “maybe multiple AIs will compete in ways that isn’t a disaster,” “maybe we can just rely on prosaic approaches forever and nothing bad happens,” “maybe it will be better than humans at having massive amounts of unchecked power by default.” These all certainly seem to rely far more on speculative claims, with far less concrete ideas about how to validate or ensure them.

Davidmanheim 20 Dec 2024 6:06 UTC
3 points
1
in reply to: Chris_Leong’s comment on: Announcement: AI for Math Fund
It is critical for guaranteed safe AI and many non-prosaic alignment agendas. I agree it has risks, since all AI capabilities and advances pose control risks, but it seems better than most types of general capabilities investments.

Do you have a more specific model of why it might be negative?

Davidmanheim 18 Dec 2024 7:23 UTC
4 points
1
in reply to: deepthoughtlife’s comment on: Trying to translate when people talk past each other
I don’t think it was betrayal, I think it was skipping verbal steps, which left intent unclear.

If A had said “I promised to do X, is it OK now if I do Y instead?” There would presumably have been no confusion. Instead, they announced, before doing Y, their plan, leaving the permission request implicit. The point that “she needed A to acknowledge that he’d unilaterally changed an agreement” was critical to B, but I suspect A thought that stating the new plan did that implicitly.

Davidmanheim 14 Dec 2024 19:56 UTC
2 points
0
in reply to: plex’s comment on: MIRI’s June 2024 Newsletter
Strongly agree that there needs to be an institutional home. My biggest problem is that there is still no such new home!

Davidmanheim 12 Dec 2024 8:44 UTC
2 points
0
in reply to: notfnofn’s comment on: Refuting Searle’s wall, Putnam’s rock, and Johnson’s popcorn
You should also read the relevant sequence about dissolving the problem of free will: https://www.lesswrong.com/s/p3TndjYbdYaiWwm9x