Thanks… but wait, this is among the most impressive things you expect to see? (You know more than I do about that distribution of tasks, so you could justifiably find it more impressive than I do.)
TsviBT
What are some of the most impressive things you do expect to see AI do, such that if you didn’t see them within 3 or 5 years, you’d majorly update about time to the type of AGI that might kill everyone?
would you think it wise to have TsviBT¹⁹⁹⁹ align contemporary Tsvi based on his values? How about vice versa?
It would be mostly wise either way, yeah, but that’s relying on both directions being humble / anapartistic.
do you think stable meta-values are to be observed between australopiteci and say contemporary western humans?
on the other hand: do values across primitive tribes or early agricultural empires not look surprisingly similar?
I’m not sure I understand the question, or rather, I don’t know how I could know this. Values are supposed to be things that live in an infinite game / Nomic context. You’d have to have these people get relatively more leisure before you’d see much of their values.
I mean, I don’t know how it works in full, that’s a lofty and complex question. One reason to think it’s possible is that there’s a really big difference between the kind of variation and selection we do in our heads with ideas and the kind evolution does with organisms. (Our ideas die so we don’t have to and so forth.) I do feel like some thoughts change some aspects of some of my values, but these are generally “endorsed by more abstract but more stable meta-values”, and I also feel like I can learn e.g. most new math without changing any values. Where “values” is, if nothing else, cashed out as “what happens to the universe in the long run due to my agency” or something (it’s more confusing when there’s peer agents). Mateusz’s point is still relevant; there’s just lots of different ways the universe can go, and you can choose among them.
I quite dislike earplugs. Partly it’s the discomfort, which maybe those can help with; but partly I just don’t like being closed away from hearing what’s around me. But maybe I’ll try those, thanks (even though the last 5 earplugs were just uncomfortable contra promises).
Yeah, I mean I think the music thing is mainly nondistraction. The quiet of night is great for thinking, which doesn’t help the sleep situation.
Yep! Without cybernetic control (I mean, melatonin), I have a non-24-hour schedule, and I believe this contributes >10% of that.
(1) was my guess. Another guess is that there’s a magazine “GQ”.
Ohhh. Thanks. I wonder why I did that.
No yeah that’s my experience too, to some extent. But I would say that I can do good mathematical thinking there, including correctly truth-testing; just less good at algebra, and as you say less good at picking up an unfamiliar math concept.
(These are 100% unscientific, just uncritical subjective impressions for fun. CQ = cognitive capacity quotient, like generally good at thinky stuff)
Overeat a bit, like 10% more than is satisfying: −4 CQ points for a couple hours.
Overeat a lot, like >80% more than is satisfying: −9 CQ points for 20 hours.
Sleep deprived a little, like stay up really late but without sleep debt: +5 CQ points.
Sleep debt, like a couple days not enough sleep: −11 CQ points.
Major sleep debt, like several days not enough sleep: −20 CQ points.
Oversleep a lot, like 11 hours: +6 CQ points.
Ice cream (without having eaten ice cream in the past week): +5 CQ points.
Being outside: +4 CQ points.
Being in a car: −8 CQ points.
Walking in the hills: +9 CQ points.
Walking specifically up a steep hill: −5 CQ points.
Too much podcasts: −8 CQ points for an hour.
Background music: −6 to −2 CQ points.
Kinda too hot: −3 CQ points.
Kinda too cold: +2 CQ points.
(stimulants not listed because they tend to pull the features of CQ apart; less good at real thinking, more good at relatively rote thinking and doing stuff)
When they’re nearing the hotel, Alice gets the car’s attention. And she’s like, “Listen guys, I have been lying to you. My real name is Mindy. Mindy the Middlechainer.”.
I’m saying that just because we know algorithms that will successfully leverage data and compute to set off an intelligence explosion (...ok I just realized you wrote TAI but IDK what anyone means by anything other than actual AGI), doesn’t mean we know much about how they leverage it and how that influences the explody-guy’s long-term goals.
I assume that at [year(TAI) − 3] we’ll have a decent idea of what’s needed
Why?? What happened to the bitter lesson?
Isn’t this what the “coherent” part is about? (I forget.)
A start of one critique is:
It simply means Darwinian processes have no limits that matter to us.
Not true! Roughly speaking, we can in principle just decide to not do that. A body can in principle have an immune system that doesn’t lose to infection; there could in principle be a world government that picks the lightcone’s destiny. The arguments about novel understanding implying novel values might be partly right, but they don’t really cut against Mateusz’s point.
Reason to care about engaging /acc:
I’ve recently been thinking that it’s a mistake to think of this type of thing—”what to do after the acute risk period is safed”—as being a waste of time / irrelevant; it’s actually pretty important, specifically because you want people trying to advance AGI capabilities to have an alternative, actually-good vision of things. A hypothesis I have is that many of them are in a sense genuinely nihilistic/accelerationist; “we can’t imagine the world after AGI, so we can’t imagine it being good, so it cannot be good, so there is no such thing as a good future, so we cannot be attached to a good future, so we should accelerate because that’s just what is happening”.
I strong upvoted, not because it’s an especially helpful post IMO, but because I think /acc needs better critique, so there should be more communication. I suspect the downvotes are more about the ideological misalignment than the quality.
Given the quality of the post, I think it would not be remotely rude to respond with a comment like “These are are well-tread topics; you should read X and Y and Z if you want to participate in a serious discussion about this.”. But no one wrote that comment, and what would X, Y, Z be?? One could probably correct some misunderstandings in the post this way just by linking to the LW wiki on Orthogonality or whatever, but I personally wouldn’t know what to link to, to actually counter the actual point.
(Interesting. FWIW I’ve recently been thinking that it’s a mistake to think of this type of thing—”what to do after the acute risk period is safed”—as being a waste of time / irrelevant; it’s actually pretty important, specifically because you want people trying to advance AGI capabilities to have an alternative, actually-good vision of things. A hypothesis I have is that many of them are in a sense genuinely nihilistic/accelerationist; “we can’t imagine the world after AGI, so we can’t imagine it being good, so it cannot be good, so there is no such thing as a good future, so we cannot be attached to a good future, so we should accelerate because that’s just what is happening”.)
Ok. So I take it you’re very impressed with the difficulty of the research that is going on in AI R&D.
(FWIW I don’t agree with that; I don’t know what companies are up to, some of them might not be doing much difficult stuff and/or the managers might not be able to or care to tell the difference.)