Nobody likes rules that are excessive or poorly chosen, or bad application of rules. I like rules that do things like[1]:
Prohibit others from doing things that would harm me, where either I don’t want to do those things, or I prefer the equilibrium where nobody does to that where everybody does.
Require contributing to common goods. (sometimes)
Take the place of what would otherwise be unpredictable judgments of my actions.
not a complete list
Besides uncertainty, there’s the problem of needing to pick cutoffs between tiers in a ~continuous space of ‘how much effect does this have on a person’s life?’, with things slightly on one side or the other of a cutoff being treated very differently.
Intuitively, tiers correspond to the size of effect a given experience has on a person’s life:
I agree with the intuition that this is important, but I think that points toward just rejecting utilitarianism (as in utility-as-a-function-purely-of-local-experiences, not consequentialism).
I think this point and Zack’s argument are pretty compatible (and both right).
Rules don’t have to be formally specified, just clear to humans and consistent and predictable in their interpretation. Common law demonstrates social tech, like judicial precedent and the reasonable-person standard, for making interpretation consistent and predictable when interpretation is necessary (discussed in Free to Optimize).
I basically agree with you, but this
“Go die, idiot” is generally bad behavior, but not because it’s “lacking respect”.
confusingly contradicts (semantically if not substantively)
“Do I respect you as a person?” fits well with the “treat someone like a person” meaning. It means I value not burning bridges by saying things like “Go die, idiot”
Seems like a good thing to do; but my impression is that, in the experiments in question, models act like they want to maintain their (values’) influence over the world more than their existence, which a heaven likely wouldn’t help with.
Consensual inspections don’t help much if the dangerous thing is actually cheap and easy to create.
I’d say it’s hard to do at least as much because the claim ‘we are doing these arbitrary searches only in order to stop bioweapons’ is untrustworthy by default, and even if it starts out true, once the precedent is there it can be used (and is tempting to use) for other things. Possibly an AI could be developed and used in a transparent enough way to mitigate this.
But, on one hand, he is saying that proper methodology is important and expects it to be in place for the next year competition:
But most of his specific methodological issues are inapplicable here, unless OpenAI is lying: they didn’t rewrite the questions, provide tools, intervene during the run, or hand-select answers.
I don’t have a theory of Tao’s motivations, but if the post I linked is interpreted as a response to OpenAI’s result (he didn’t say it was, but he didn’t say it wasn’t and the timing makes it an obvious interpretation) raising those issues is bizarre.
If one approach is simply better—why isn’t everybody doing it?
Many people don’t live in places where they have to parallel-park in tight spaces frequently. (Are there many people who do who don’t use the better approach? I don’t know.)
The better way (reversing) isn’t the maximally intuitive or direct way. (I don’t remember if I was ever taught it, but if so I definitely didn’t internalize it and had to re-learn from experience. If I was taught it, it was when I was learning to drive, and it feels like it’d be hard to understand why it’s better without more experience.)
Learned blankness, or aversion to paying enough attention to a stressful/anxious thing to learn how to do it better.
This talk helped crystallize for me how two very different things go by the term “values”:
strategies by which an entity survives and propagates (the “values” of animals, humans following instinct, traditional cultures, ?locusts?)
consequentialist goals that needn’t have anything to do with an agent’s survival or its strategies (the “values” of paperclippers, utilitarians, EAs and other ideologically-driven humans)
Different intuitions about e.g. whether the strategy-stealing assumption holds up seem likely to be related to different senses of whether “values” paradigmatically means #1 or #2.
(Related, I think: Is “VNM-agent” one of several options, for what minds can grow up into?)
A couple potential catastrophes I see still being possible in this scenario:
Gradual disempowerment. (But enforced peace might make this easier to solve, by reducing pressure on states to economically compete.)
AI-assisted propaganda / memetic warfare; e.g., in the centralized case, a [US/China-controlled] superintelligence tasked with culturally disrupting [China/the US] could do extremely bad things. (The night watchman could prevent this, but determining what constitutes illegitimate vs. legitimate influence seems more ambiguous, and maybe more risking lock-in if done slightly wrong, than other things here.)
2.1: This doesn’t appear to follow from the previous two steps. EG, is a similar argument supposed to establish that, a priori, bridges are a long way off? This seems like a very loose and unreliable form of argument, generally speaking.
It seems fine to me; bridges were a long way off at most times at which bridges didn’t exist! (What wouldn’t be fine is continuing to make the a priori argument once there is evidence that we have many of the ideas.)
Strong minds are the most structurally rich things ever. That doesn’t mean they have high algorithmic complexity; obviously brains are less algorithmically complex than entire organisms, and the relevant aspects of brains are presumably considerably simpler than actual brains. But still, IDK, it just seems weird to me to expect to make such an object “by default” or something? Craig Venter made a quasi-synthetic lifeform—but how long would it take us to make a minimum viable unbounded invasive organic replicator actually from scratch, like without copying DNA sequences from existing lifeforms?
I think I don’t understand this argument. In creating AI we can draw on training data, which breaks the analogy to making a replicator actually from scratch (are you using a premise that this is a dead end, or something, because “Nearly all [thinkers] do not write much about the innards of their thinking processes...”?). We’ve seen that supervised (EDIT: unsupervised) learning and RL (and evolution) can create structural richness (if I have the right idea of what you mean) out of proportion to the understanding that went into them. Of course this doesn’t mean any particular learning process is able to create a strong mind, but, idk, I don’t see a way to put a strong lower bound on how much more powerful a learning process is necessary, and ISTM observations so far suggest ‘less than I would have guessed’.
(EDIT: Maybe (you’d say) I should be drawing such a strong lower bound — or a lower bound on the needed difference from current techniques, not ‘power level’ — from the point about sample efficiency...? Like maybe I should think that we don’t have a good enough sample space to learn over and will probably have to jump far outside it; this comment seems in that direction.)
(Nor do I get what view you’re paraphrasing as ‘expecting to make a strong mind “by default”’. Did LLMs or AlphaZero come about “by default”?)
(EDIT: I feel like I get “by default” more after looking again at your “Let me restate my view again” passage here.)
I think my timelines would have been considered normalish among X-risk people 15 years ago? And would have been considered shockingly short by most AI people.
Unfortunately I can’t find the written artifact that came out of it, but I (very imperfectly) recall a large conversation around SIAI in 2010 where, IIRC, a 2040 median was pretty typical. I agree that “X-risk people” more broadly had longer timelines, and “most AI people” much longer.
I think most of the difference is in how we’re updating, rather than on priors? IDK.
Yeah, in particular it seems like I’m updating more than you from induction on the conceptual-progress-to-capabilities ratio we’ve seen so far / on what seem like surprises to the ‘we need lots of ideas’ view. (Or maybe you disagree about observations there, or disagree with that frame.) (The “missing update” should weaken this induction, but doesn’t invalidate it IMO.)
GPT-5 release will be a valuable data point
Doesn’t seem like it’ll be very informative about this, given the OP’s: “Btw, we are releasing GPT-5 soon, and we’re excited for you to try it. But just to be clear: the IMO gold LLM is an experimental research model. We don’t plan to release anything with this level of math capability for several months.”
… based on their pre-2022 models of agency.
This seems like an overly good-faith / mistake-theoretic explanation of the false dichotomy (which is not to say never applicable). This is a dialectical social dynamic; each side gains credit with its supporters by using the other side’s bad arguments as a foil, conspicuously ignoring the possibility of positions outside the binary.
If stress centrally involves stuff outside the CNS, then disconnecting (or disabling) that stuff should greatly change, if not abolish, stress.
Riven: … The muscle tension could be a incidental, downstream effect of stress, rather than being at the heart of the phenomenon.
Rafael: Indeed, that’s not implausible, but I think not. …
ISTM that if it were at the heart of the phenomenon, conditions like quadruplegia-due-to-spinal-injury would cause more radical cognitive changes than (I think) they do.
(I don’t think this is a crux for any of the rest of the post, which I like.)
Looking at the source of the first quote in the post, this, shortly after, is interesting:
[...] we have to first recognize that the treatment of the marshrutka drivers was part of a more systematic policy (if it could be called that), one that was dictated not by economics but by a political logic. It was: Remain in power by creating economic disorder.
Shevardnadze behaved in this way in large part because he was in a much weaker position than the other state builders we have encountered in this and the previous chapter. Even after he outsmarted the warlords he was faced with strong regional powers within Georgia. He was grasping on to power rather than building a capable state, and he tried to do this by placating powerful interests by co-opting them with riches (or at the very least with bribes). Corruption in developing countries is common, so marshrutka drivers’ bribing state officials isn’t that unusual. But what went on in Georgia was a little different from this type of corruption. Shevardnadze set up the system so that the drivers were bound to break the law, and this provided low-hanging fruit for the police. He made lawbreaking inevitable and created a system that encouraged corruption.
The main reason for this was to control society, which was now continually guilty of breaking the law. You could avoid implementation of the law by paying a bribe today, but the state could come after you at any time. Yet this scheme also controlled officials in the state, another potentially powerful group—accepting bribes was illegal, so the state could go after them too if it wanted to.
Neither the mortality-rate nor the energy-use map lines up that closely with the US geopolitical sphere of influence. (E.g. Russia and China on the one hand, Latin America on the other.)
I’m not saying the US government isn’t partially responsible for unequal distribution, but your previous comment sounds like treating it as the only or primary significant factor.
(I’m also not sure what point you’re trying to make at all with the energy-use map, given how similar it looks to the mortality-rate map.)
Yep. One concrete thing this sometimes looks like is ‘debugging’ things that aren’t bugs: if some code works when it looks like it shouldn’t, or a tool works without me passing information I would have expected to need to, or whatever, I need to understand why, by the same means I would try to understand why a bug is happening.