LessWrong team member / moderator. I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well.
Raemon
Karma: 60,901
Do you think math would have helped in any of the examples listed here? Or, generally, help you elicit your inner intuitive predictor’s models?
(I agree math helps in other ways that I would call “usefully concrete” and that shares structure with the stuff in this post, but it’s not really what this post is about)
In the most extremely optimistic world, you could probably hire 50 extremely talented people by offering them $100M/year each (matching what Meta offered). You could probably also hire ~200 more junior people at $10M/year (the bottleneck on hiring more would be management capacity). So in total you could spend $7B/year.
Over time, I expect this to get more expensive since AI companies’ valuations will increase, and therefore, so will employee compensation.
I don’t know that the idea is fundamentally good but at least is scales somewhat with the equity of the safety-sympathetic people at labs?
Hufflepuff is more of a literary reference, it means “people that are more down to earth, not particularly ambitious, but, are warm and empathetic and operationally competent and reliably show up every day to do the shit that needs doing”. It’s not, like, a particularly natural way of carving up psychometrics, I wrote “Project Hufflepuff” because it was a shared literary reference I could easily build on at the time.
Makes sense, although want to flag one more argument that, the takeaways people tend to remember from posts are ones that are encapsulated in their titles. “Musings on X” style posts tend not to be remembered as much, and I think this is a fairly important post for people to remember.
“I hope Joe Carlsmith works on illegible problems” is, uh, a very fun title but probably bad. :P
Not sure. Let me think about it step by step.
It seems like the claims here are:
Illegible and Legible problems both exist in AI safety research
Decisionmakers are less likely to understand illegible problems
Illegible problems are less likely to cause decisionmakers to slow/stop where appropriate
Legible problems are not the bottleneck (because they’re more likely to get solved by default by the time we reach danger zones)
Working on legible problems shortens timelines without much gain
[From JohnW if you wanna incorporate] If you work on legible problems by making illegible problems worse, you aren’t helping.
I guess you do have a lot of stuff you wanna say, so it’s not like the post naturally has a short handle.
“Working on legible problems shortens timelines without much gain” is IMO the most provocative handle, but, might not be worth it if you think of the other points as comparably important.
“Legible AI problems are not the bottleneck” is slightly more overall-encompassing
I think this post could use a post title that makes the more explicit, provocative takeaway (otherwise I’d have assumed “this is letting you know illegible problems exist” and I already knew the gist)
This is a bit of a random-ass take, but, I think I care more about Joe not taking equity than you not taking equity, because I think Joe is more likely to be a person where it ends up important that he legibly have as little COI as possible (this is maybe making up a bunch of stuff about Joe’s future role in the world, but, it’s where my Joe headcanon is at).
I think this part of Heroic Responsibility isn’t too surprising/novel to people. Obviously the business owner has responsibility for the business. The part that’s novel is more like:
If I’m some guy working in legal, and I notice this hot potato going around, and it’s explicitly not my job to deal with it, I might nonetheless say “ugh, the CEO is too busy to deal with this today and it’s not anyone else’s job. I will deal with it.” Then you go to each department head, even if you’re not even a department head you’re a lowly intern (say), and say “guys, I think we need to decide who’s going to deal with this.”
And if their ego won’t let them take advice from an intern, you might also take it as your responsibility to figure out how to navigate their ego – either by making them feel like it was their own idea, or by threatening to escalate to the CEO if they don’t get to it themselves.
A great example of this, staying with them realm of “random Bureaucracy”, I got from @Elizabeth:
E. D. Morel was a random bureaucrat at a shipping company in 1891. He noticed that his company was shipping guns and manacles into the Congo, and shipping rubber and other resources back out to Britain.
It was not Morel’s job to notice that this was a bit weird.
It was not Morel’s job to notice that that weirdness was a clue, and look into those clues. And then find out that what was happening was, weapons were being sent to the Congo to forcibly steam resources at gunpoint.
It was not his job to make it his mission to raise awareness of the Congo abuses and stop them.
But he did.
...
P.S. A failure mode of rationalists is to try to take Heroic responsibility for everything, esp. in a sort of angsty way that is counterproductive and exhausting. It’s also a failure mode to act as if only you can possibly take Heroic responsibility, rather than trying to model the ecosystem around you and the other actors (some of whom might be Live Players who are also taking Heroic Responsibility, some of whom might be sort of local actors following normal incentives but are still, like, part of the solution)
There is nuance to when and how to do Heroic Responsibility well.
Being “Usefully Concrete”
I’m legit curious if people think of the Partiful as being a value-add if you have the other things here (I didn’t include it because it seemed like most of the point of the Partiful was to make it easy to invite people to the eventBrite and there were a whole lotta other links here)
lol whoops, thanks (also, there’s actually a new URL for that, I edited your comment to point to the new one)
Berkeley Solstice Weekend
In a previous discussion about this, an argument mentioned was “having all your friends and colleagues believe in a thing is probably more epistemically compromising than the equity.”
Which seems maybe true. But, I update in the other direction of “you shouldn’t take equity, and, also, you should have some explicit plan for dealing with the biases of ’the people I spend the most time with think this,
(This also applies to AI pessimists to be clear, but I think it’s reasonable to hold people extra accountable about it when they’re working at a company who’s product has double-digit-odds of destroying the world)
Thanks to both you and Zack Stein-Perlman.
One thing I immediately note is “wow the Logistics Success Curve jargon is particularly impenetrable”, I think this could use a normie-friendly name.
Even people who prefer “natural” food would likely go for e. g. the meat of beasts from carefully designed environments with very fast evolutionary loops set up to make their meat structured for maximum tastiness.[1]
I don’t think there are many people who “prefer natural food” who would consider “you evolved them very carefully” to really satisfy their cruxes. (Which is not to claim their cruxes are coherent)
I’m not sure which logistics curve one you mean?
Yeah I think that’s my main causal story about it atm, I do think some previous pseudo-rants ended up in a more polished form in IABIED.
(A question that should have been on the first list was “have you already actually checked that these sorts of posts have had the sort of outcome you want them to have?”. Cuz, like, maybe they’re totally working and are just kinda annoying for the people who already got the memo)
Okay yeah that explanation is way better.