This is a short but decent articulation which strongly agrees with my own observations; good job.
With my ex, it was a common-knowledge-to-the-two-of-us phenomenon that she would always find something to worry about. So when she felt worried or anxious about relatively minor things, I would sometimes jokingly respond “Oh good! If you’re worrying about that then there must not be any serious problems right now.”.
How much money did these cost?
David tries to punt things to LLMs at least once a day on average when we’re working. So far, they continue to work best when they can act as Google Search Plus Plus—i.e. when there’s some already-known fact relevant to what we’re doing, and they surface that fact to us. Occasionally they can complete a conceptually-simple-but-technically-dense proof by combining a few such facts, and very often they can write some useful code by combining a few already-known pieces.
For anything novel, they remain almost always useless in our experience; they string together words which sound relevant but the semantics don’t make any sense.
The evolutionary explanation I thought was standard for sex differences in variability is that males (of most species) face way more potential upside—i.e. they can potentially mate with a whole lot of different females and have far more kids than any single female can. Females typically have a much lower ceiling on kid-count, so they face less potential fitness upside from high-variance strategies.
I liked the part of this post about X-inactivation a lot, even though it turned out wrong. It would be really interesting if most of the variability difference across sex turned out to share one simple mechanistic basis, though that doesn’t seem very likely on my current models of the underlying selection pressures.
Quite the opposite: the subject-we-gesture-at-with-the-word-”alignment” is not particularly provocative or controversial when you think about it deeply, at least not along the axes people generally argue over in the context of morality/ethics, because those axes just aren’t that technically central or relevant.
Personally, my guess is that morality and ethics themselves would not be particularly controversial or provocative if people usually approached them with a goal of deep technical understanding. That’s just not the goal with which approximately-anybody, including nearly all professional philosophers, approaches the subject—as we see e.g. on that Stanford Encyclopedia page. Those are people trying to have the equivalent of fun house party conversations, or in some cases write manifestos, not people seriously trying to achieve deep technical understanding.
Indeed, invoking the words “good” or “right” also tend to make people dumber (though less so than “morality” or “ethics”), and trying to do philosophical analysis of what is “good” or “right” is exactly the thing which seems to insta-brain-kill people; it’s exactly the lever which “morality” and “ethics” pull.
For example, let’s look at two pages in the Stanford Encyclopedia of Philosophy. I picked these by pulling up the table of contents, and then clicking the first one which seemed not-very-morality-loaded and the first one which seemed very-morality-loaded.
First up, abduction. No morality talk here. It’s describing a feature of human reasoning, which seems functionally load-bearing for epistemics in some cases and would probably generalize to other kinds of minds (like aliens or AI). It doesn’t trivially fit a couple common frames of epistemics, which is why it’s interesting. A lot of the discussion is centered around pretty narrow or outdated models of reasoning, but it’s a technically interesting and sensible article, which inspires good questions at least.
In contrast, the ethics of abortion. Before we even get to the actual content, note the topic. Abduction is a topic relevant to understanding minds and reasoning in general, a topic which would likely be relevant even to AIs; it belongs in a generalizable world-model. Abortion, by contrast, would be irrelevant to many other kinds of minds—e.g. human-level-intelligent platypuses would lay eggs, and therefore the whole issue of abortion would not have a clean analogue for them. (And human-level intelligent ants would be in a whole different frame!) Almost certainly, the reason why a Stanford Encyclopedia page exists for abortion at all is that it was a major hot-button political topic for a while, which won the memetic competition for attention in US politics. But in the grand scheme of things, it is just not that important of a question at all even for humans, and entirely irrelevant to many other kinds of minds. The very fact that people pay so much attention to it is itself a strong sign of mindkill.
Looking at the content of the page… the entire thing is a string of analogies and attempts to generalize various heuristics to the case of abortion. Notably sparse or absent is:
Technical engagement with the developmental process, when various things come online for a fetus/baby (like e.g. pain, self-awareness).
Technical engagement with the way humans’ preferences/values actually typically form. Spoiler: it ain’t usually by thought-experiments involving a violinist.
Technical engagement with the first and second-order actual effects of abortion laws/norms (though laws/norms are of course distinct from morality, consequentialism still matters).
More vibe-ishly, compared to the abduction article, the whole thing very much has a bikeshed vibe to it. It’s all the sort of stuff which would make good fodder for conversation at a house party, not the sort of stuff which involves dense technical study and deep understanding.
I personally avoid even using the words “morality” or “ethics” in the context of AI alignment, because both of those words reliably turn the vast majority of otherwise-sensible people into morons the moment they are spoken.
Yeah, that would probably capture the intuition, though this isn’t something which really needs precise operationalization in order to be useful.
That is a cool and relevant domain of research I had not encountered before, thank you!
Good question, we don’t yet know the answer. It is definitely the right kind of question to ask.
My framing has evolved over time; this is a scattered list as opposed to a unified picture. But it’s still basically-right as the motivating list for the bigger picture.
Our own work touches on several of them, but I generally see very few people working on these kinds of problems.
A background piece of my own models (which could be right/wrong independent of the content of this post) is that bounded rationality constraints basically just don’t bind at all in practice once a mind passes a certain threshold, and in particular will not bind for even moderately-superhuman general AI.
The canonical example illustrating how this could happen is inference involving an ideal gas. Roughly speaking, once an agent is smart enough to use the Boltzmann distribution, even a Jupiter brain will not be able to do significantly better in practice, because chaotic dynamics wipe out all the other signal. In that case, bounded rationality constraints cease to bind once the agent is smart enough to use a Boltzmann distribution (again, roughly speaking).
The abstract description of the work sounded interesting, but the post did not explain what you did enough for me to know how to update. You say you “prompted specific modes that were intended to require the model to actually engage that mode internally, and ran my feature extraction on the resulting generations”, then throw around numbers like “The first run hit 76% discrimination overall”, but I’m not sure what that number actually means here.
You prompted “specific modes”? Like, you fed in some kinds of prompts that a human thought would induce some qualitatively different cognition? Or you fed in prompts that have been already confirmed by some kind of mechinterp to push nets into different dynamic regimes internally? Something else? And what exactly were the “modes” in question?
And then it sounds like you trained a discriminator against net internals to distinguish which mode had been prompted? At that point we could maybe drop all the flavor text and say “I put in prompts about X, and prompts about Y, then trained a classifier on <internals> to distinguish X vs Y prompts”. I do not see how that could possibly pick out computational signatures orthogonal to semantics! But maybe you did something else?
Anyway, the general project of detecting internal computational-steering stuff orthogonal to semantics sounds important and interesting. Alas, this post has not really given me enough information to update on that project at all.
Yeah, for both of them, betting all their influence on the button is a “free move” under their own model. It’s a deal which is all upside with zero cost. So they’re definitely going to do that, therefore any additional deals they make have to look like good deals even after that one; we can conceptually reason as though the button-bet happens first.
False. If the two variables are independent (i.e. no information), then correlation is zero no matter what kind of functional relationship we consider.
Long before modern ML, it was not difficult to write a program which would generate programs, test those programs on some task, and make random changes to the programs in an attempt to find programs which perform better on the task.
It was also not difficult to apply such a program to the task of self-improvement, i.e. score how quickly it finds new-and-better programs on a basket of object-level tasks (including the self-improvement task itself).
All that would have been straightforward in the 1970s, if one had access to a machine to run it all. By the 1990s, a bright CS undergrad could do it on their home computer. The loop you’re talking about was easy to code long before modern ML.
So, I take it that Savage’s theorem is a representation theorem under your schema?
Yes. Arguably it is also a coherence theorem, the two are not mutually exclusive, but it’s more unambiguously a representation theorem.
Theoretically or practically? I.e. you can’t derive an exploitability result easily from a parto suboptimality?
Practically. Consider e.g. applying coherence tools to an e coli. That thing is not capable of signing arbitrary contracts or meaningfully choosing between arbitrary trades, and insofar as it’s wasting resources those resources likely end up e.g. as waste heat. For another agent to “eat up” the wasted resources, it would need to e.g. restructure the e coli’s metabolic pathways; it’s not really something which can happen via things-which-look-like-trading-with-an-agent.
I don’t claim that this is canonical, but here’s the way David and I use the terms amongst ourselves.
First, we don’t usually use the term “representation theorem” at all, but if we did, that would naturally refer to a theorem saying that some preferences/behavior/etc can be represented in a particular way, like e.g. expected utility maximization over some particular states/actions/whatever. We would probably classify e.g. VNM as a representation theorem, though we basically-never think about VNM at all so we don’t really need a term for it.
Second, coherence. When we talk about coherence theorems, we usually don’t think about exploitability, but rather about pareto suboptimality. (Of course exploitability is a special case of pareto suboptimality, but the reverse doesn’t always apply easily.) Vibes-wise, we’re usually thinking about a system behaving pareto-optimally across different places—e.g. different parts of the system behaving jointly-pareto-optimally, or choices made across different inputs/worlds being jointly-pareto-optimal, or decisions at different times being jointly-pareto-optimal. The behavior is “coherent” in the sense that the system at all these different parts/places/times all “act in a consistent way”, such that they’re jointly pareto-optimal. That’s the sort of thing which “coherence” gestures at in our usage.
In this case, I’m mildly skeptical, because probability before Laplace bore a lot less resemblance to today’s probability IIUC (though I have not personally read source texts, so don’t update too hard on my understanding). Bayes did discover the theorem, but I don’t know if he conceptually thought of it like we do today or would have used it like we do today; I view that as largely coming from Laplace. On the flip side, that means Laplace’ work on probability theory was maybe highly counterfactual.
Facebookish
In the style of facebook’s feed, including a high proportion of randomly generated ads.