grantmakers have tried pulling that lever a bunch of times
What do you mean by this? I can think of lots of things that seem in some broad class of pulling some lever that kinda looks like this, but most of the ones I’m aware of fall greatly short of being an appropriate attempt to leverage smart young creative motivated would-be AGI alignment insight-havers. So the update should be much smaller (or there’s a bunch of stuff I’m not aware of).
(FWIW this was my actual best candidate for a movie that would fit, but I remembered so few details that I didn’t want to list it.)
I’m struggling to think of any. Some runners-up:
Threads (1984) because Beyond the Reach of God.
Bird Box because Contrapositive Litany of Hodgell.
Ghostbusters because whatever is real is lawful; it’s up to you to Think Like Reality, and then you can bust ghosts.
The Secret Life of Walter Mitty (1947, NOT 2013) because at some point you have to take the Inside View.
Pi (1998) because The Solomonoff Prior is Malign.
Cf. Moneyball.
Emotions are hardwired stereotyped syndromes of hardwired blunt-force cognitive actions. E.g. fear makes your heart beat faster and puts an expression on your face and makes you consider negative outcomes more and maybe makes you pay attention to your surroundings. So it doesn’t make much sense to value emotions, but emotions are good ways of telling that you value something; e.g. if you feel fear in response to X, probably X causes something you don’t want, or if you feel happy when / after doing Y, probably Y causes / involves something you want.
we’ve checked for various forms of funny business and our tools would notice if it was happening.
I think it’s a high bar due to the nearest unblocked strategy problem and alienness.
I agree that when AGI R&D starts to 2x or 5x due to AI automating much of the process, that’s when we need the slowdown/pause)
If you start stopping proliferation when you’re a year away from some runaway thing, then everyone has the tech that’s one year away from the thing. That makes it more impossible that no one will do the remaining research, compared to if the tech everyone has is 5 or 20 years away from the thing.
10 more years till interpretability? That’s crazy talk. What do you mean by that and why do you think it? (And if it’s a low bar, why do you have such a low bar?)
“Pre-AGI we should be comfortable with proliferation” Huh? Didn’t you just get done saying that pre-AGI AI is going to contribute meaningfully to research (such as AGI research)?
I think you might have been responding to
Susan could try to put focal attention on the scissor origins; but one way that would be difficult is that she’d get pushback from her community.
which I did say in a parenthetical, but I was mainly instead saying
Susan’s community is a key substrate for the scissor origins, maybe more than Susan’s interaction with Robert. Therefore, to put focal attention on the scissor origins, a good first step might be looking at her community—how it plays the role of one half of a scissor statement.
Your reasons for hope make sense.
hope/memory of the previous society that (Susan and Tusan and Vusan) and (Robert and Sobert and Tobert) all shared, which she has some hope of reaccessing here
Anecdata: In my case it would be mostly a hope, not a memory. E.g. I don’t remember a time when “I understand what you’re saying, but...” was a credible statement… Maybe it never was? E.g. I don’t remember a time when I would expect people to be sufficiently committed to computing “what would work for everyone to live together” that they kept doing so in political contexts.
(generic comment that may not apply too much to Mayer’s work in detail, but that I think is useful for someone to hear:) I agree with the basic logic here. But someone trying to follow this path should keep in mind that there’s philosophically thorniness here.
A bit more specifically, the questions one asks about “how intelligence works” will always be at risk of streetlighting. As an example/analogy, think of someone trying to understand how the mind works by analyzing mental activity into “faculties”, as in: “So then the object recognition faculty recognizes the sofa and the doorway, and it extracts their shapes, and sends their shapes to the math faculty, which performs a search for rotations that allow the sofa to pass through the doorway, and when it finds one it sends that to the executive faculty, which then directs the motor-planning faculty to make an execution plan, and that plan is sent to the motor factulty...”. This person may or may not be making genuine progress on something; but either way, if they are trying to answer questions like “which faculties are there and how do they interoperate to perform real-world tasks”, they’re missing a huge swath of key questions. (E.g.: “how does the sofa concept get produced in the first place? how does the desire to not damage the sofa and the door direct the motor planner? where do those desires come from, and how do they express themselves in general, and how do they respond to conflict?”)
Some answers to “how intelligence works” are very relevant, and some are not very relevant, to answering fundamental questions of alignment, such as what determines the ultimate effects of a mind.
Intelligence also has costs and has components that have to be invented, which explains why not all species are already human-level smart. One of the questions here is which selection pressures were so especially and exceptionally strong in the case of humans, that humans fell off the cliff.
IDK, fields don’t have to have names, there’s just lots of work on these topics. You could start here https://en.wikipedia.org/wiki/Evolutionary_anthropology and google / google-scholar around.
See also https://www.youtube.com/watch?v=tz-L2Ll85rM&list=PL1B24EADC01219B23&index=556 (I’m linking to the whole playlist, linking to a random old one because those are the ones I remember being good, IDK about the new ones).
My hope is that this can become more feasible if we can provide accurate patterns for how the scissors-generating-process is trying to trick Susan(/Robert). And that if Susan is trying to figure out how she and Robert were tricked, by modeling the tricking process, this can somehow help undo the trick, without needing to empathize at any point with “what if candidate X is great.”
This is clarifying...
Does it actually have much to do with Robert? Maybe it would be more helpful to talk with Tusan and Vusan, who are also A-blind, B-seeing, candidate Y supporters. They’re the ones who would punish non-punishers of supporting candidate X / talking about A. (Which Susan would become, if she were talking to an A-seer without pushing back, let alone if she could see into her A-blindspot.) You could talk to Robert about how he’s embedded in threats of punishment for non-punishment of supporting candidate Y / talking about B, but that seems more confusing? IDK.
I think I agree, but
It’s hard to get clear enough on your values. In practice (and maybe also in theory) it’s an ongoing process.
Values aren’t the only thing going on. There are stances that aren’t even close to being either a value, a plan, or a belief. An example is a person who thinks/acts in terms of who they trust, and who seems good; if a lot of people that they know who seem good also think some other person seems good, then they’ll adopt that stance.
I don’t care about doing this bet. We can just have a conversation though, feel free to DM me.
(e.g. 1 billon dollars and a few very smart geniuses going into trying to make communication with orcas work well)
That would give more like a 90% chance of superbabies born in <10 years.
Fighting wars with neighboring tribes
Extractive foraging
Persistence hunting (which involves empathy, imagination (cf cave paintings), and tracking)
Niche expansion/travel (i.e. moving between habitat types)
In particular, sometimes entering harsh habitats puts various pressures
Growing up around people with cultural knowledge (advantage to altriciality, language, learning, imitation, intent-sharing)
Altriciality demands parents coordinate
Children’s learning ability incentivizes parents to learn to teach well
etc.
There’s a whole research field on this FYI.
I’m not gonna read the reddit post because
it’s an eyebleed wall of text,
the author spent hours being excited about this stuff without bothering to learn that we have ~20 billion cortical neurons, not 20 trillion,
yeah.
I don’t know whether orcas are supersmart. A couple remarks:
I don’t think it makes that much sense to just look at cortical neuron counts. Big bodies ask for many neurons, including cortical motor neurons. Do cetaceans have really big motor cortices? Visual cortices? Olfactory bulbs? Keyword “allometry”. Yes, brains are plastic, but that doesn’t mean orcas are actually ever doing higher mathematics with their brains.
Scale matters, but I doubt it’s very close to being the only thing! Humans likely had genetic adaptations for neuroanatomical phenotypes selected-for by some of: language; tool-making; persisting transient mental content; intent-inference; intent-sharing; mental simulation; prey prediction; deception; social learning; teaching; niche construction/expansion/migration. Orcas have a few of these. But how many, how much, for how long, in what range of situations and manifestations? Or do you think a cow brain scaled to 40 billion neurons would be superhuman?
Culture matters. The Greeks could be great philosophers… But could a kid living in 8000 BCE, who gets to text message with an advanced alien civilization of kinda dumb people, become a cutting edge philosopher in the alien culture? Even though almost everyone ze interacts with is preagricultural, preliterate? I dunno, maybe? Still seems kinda hard actually?
Regardless of all this, talking to orcas would be super cool, go for it lol.
Superbabies is good. It would actually work. It’s not actually that hard. There’s lots of investment already in component science/tech. Orcas doesn’t scale. No one cares about orcas. There’s not hundreds of scientists and hundreds of millions in orca communications research. Etc. The sense of this plan being weird is a good sense to investigate further. It’s possible for superficial weirdness to be wrong, but don’t dismiss the weirdness out of hand.
I appreciate you being relatively clear about this, but yeah, I think it’s probably better to spend more time learning facts and thinking stuff through, compared to writing breathless LW posts. More like a couple weeks rather than a couple days. But that’s just my stupid opinion. The thing is, there’s probably gonna be like ten other posts in the reference class of this post, and they just… don’t leave much of a dent in things? There’s a lot that needs serious thinking-through, let’s get to work on that! But IDK, maybe someone will be inspired by this post to think through orca stuff more thoroughly.
IIUC, I agree with your vision being desirable. (And, IDK, it’s sort of plausible that you can basically do it with a good toolbox that could be developed straightforwardly-ish.)
But there might be a gnarly, fundamental-ish “levers problem” here:
It’s often hard to do [the sort of empathy whereby you see into your blindspot that they can see]
without also doing [the sort of empathy that leads to you adopting some of their values, or even blindspots].
(A levers problem is analogous to a buckets problem, but with actions instead of beliefs. You have an available action VW which does both V and W, but you don’t have V and W available as separate actions. V seems good to do and W seems bad to do, so you’re conflicted, aahh.)
I would guess that what we call empathy isn’t exactly well-described as “a mental motion whereby one tracks and/or mirrors the emotions and belief-perspective of another”. The primordial thing—the thing that comes first evolutionarily and developmentally, and that is simpler—is more like “a mental motion whereby one adopts whatever aspects of another’s mind are available for adoption”. Think of all the mysterious bonding that happens when people hang out, and copying mannerisms, and getting a shoulder-person, and gaining loyalty. This is also far from exactly right. Obviously you don’t just copy everything, it matters what you pay attention to and care about, and there’s probably more prior structure, e.g. an emphasis on copying aspects that are important for coordinating / synching up values. IDK the real shape of primordial empathy.
But my point is just: Maybe, if you deeply empathize with someone, then by default, you’ll also adopt value-laden mental stances from them. If you’re in a conflict with someone, adopting value-laden mental stances from them feels and/or is dangerous.
To say it another way, you want to entertain propositions from another person. But your brain doesn’t neatly separate propositions from values and plans. So entertaining a proposition is also sort of questioning your plans, which bleeds into changing your values. Empathy good enough to show you blindspots involves entertaining propositions that you care about and that you disagree with.
Or anyway, this was my experience of things, back when I tried stuff like this.
Interesting; I’m less aware of these.
I’ll answer as though I know what’s going on in various private processes, but I don’t, and therefore could easily be wrong. I assume some of these are sort of done somewhere, but not enough and not together enough.
Favor insightful critiques and orientations as much as constructive ideas. If you have a large search space and little traction, a half-plane of rejects is as or more valuable than a guessed point that you knew how to even generate.
Explicitly allow acceptance by trajectory of thinking, assessed by at least a year of low-bandwidth mentorship; deemphasize agenda-ish-ness.
For initial exploration periods, give longer commitments with less required outputs; something like at least 2 years. Explicitly allow continuation of support by trajectory.
Give a path forward for financial support for out of paradigm things. (The Vitalik fellowship, for example, probably does not qualify, as the professors, when I glanced at the list, seem unlikely to support this sort of work; but I could be wrong.)
Generally emphasize judgement of experienced AGI alignment researchers, and deemphasize judgement of grantmakers.
Explicitly asking for out of paradigm things.
Do a better job of connecting people. (This one is vague but important.)
(TBC, from my full perspective this is mostly a waste because AGI alignment is too hard; you want to instead put resources toward delaying AGI, trying to talk AGI-makers down, and strongly amplifying human intelligence + wisdom.)