jacobjacob

Karma: 6,929

jacobjacob 8 Jan 2025 20:20 UTC
9 points
0
on: AI Safety as a YC Startup
Impact = Magnitude * Direction
Surely one should think of this as a vector in a space with more dimensions than 1.
In your equation you can just 1,000,000x magnitude and it will move in the “positive direction”.
In the real world you can become a billionaire from selling toothbrushes and still be “overtaken” by a guy who wrote one blog post that happened to be real dang good
I made a drawing but lw won’t allow adding it on phone I think

jacobjacob 30 Dec 2024 20:47 UTC
11 points
1
on: By default, capital will matter more than ever after AGI
Upstarts will not defeat them, since capital now trivially converts into superhuman labour in any field.

It is false today that big companies with 10x the galaxy brains and 100x the capital reliably outperform upstarts.^[1]
Why would this change? I don’t think you make the case.
1. ^
  My favorite example, though it might still be falsified. Google invented transformers, owns DeepMind, runs their own data centres, builds their own accelerators and have huge amounts of them, have tons of hard to get data (all those books they scanned before that became not okay to do), insane distribution channels with gmail, docs, sheets, and millions of smartphones sold a year, more cash than God, they literally run Google search (and didn’t build it from scratch like Perplexity!).. I almost struggle to mention any on-paper advantage they don’t have in the AGI race… they’re positioned for an earth-shattering vertical integration play… and yet they’re behind OpenAI and Anthropic.
  There are more clear-cut examples than this.

jacobjacob 23 Dec 2024 22:43 UTC
10 points
0
in reply to: Mitchell_Porter’s comment on: jacobjacob’s Shortform Feed
I don’t expect it that soon, but do I expect more likely than not that there’s a covid-esque fire alarm + rapid upheaval moment.

jacobjacob 20 Dec 2024 23:43 UTC
3 points
0
on: o3
For people who don’t expect a strong government response… remember that Elon is First Buddy now. 🎢

jacobjacob 20 Dec 2024 23:19 UTC
21 points
8
on: jacobjacob’s Shortform Feed
It is January 2020.

jacobjacob 20 Dec 2024 22:47 UTC
3 points
0
in reply to: Zach Stein-Perlman’s comment on: Anthropic leadership conversation
Okay, well, I’m not going to post “Anthropic leadership conversation [fewer likes]” 😂

jacobjacob 20 Dec 2024 22:34 UTC
4 points
4
on: Anthropic leadership conversation
(Can you edit out all the “like”s, or give permission for an admin to do edit it out? I think in written text it makes speakers sound, for lack of a better word, unflatteringly moronic)

jacobjacob 10 Dec 2024 20:58 UTC
4 points
2
in reply to: L Rudolf L’s comment on: A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX
McNamara was at Ford, not Toyota. I reckon he modelled manufacturing like an efficient Boeing manager not an efficient SpaceX manager

jacobjacob 19 Sep 2024 17:18 UTC
LW: 3 AF: 2
7
AF
on: Secret Collusion: Will We Know When to Unplug AI?
(Nitpick: I’d find the first paragraphs would be much easier to read if they didn’t have any of the bolding)

jacobjacob 19 Sep 2024 12:30 UTC
4 points
2
in reply to: tdietterich’s comment on: Proveably Safe Self Driving Cars
rename the “provable safety” area as “provable safety modulo assumptions” area and be very explicit about our assumptions.
Very much agree. I gave some feedback along those lines as the term was coined; and am sad it didn’t catch on. But of course “provable safety modulo assumptions” isn’t very short and catchy...
I do like the word “guarantee” as a substitute. We can talk of formal guarantees, but also of a store guaranteeing that an item you buy will meet a certain standard. So it’s connotations are nicely in the direction of proof but without, as it were, “proving too much” :)

jacobjacob 24 Aug 2024 15:56 UTC
12 points
12
in reply to: ESRogs’s comment on: Dario Amodei leaves OpenAI
Interesting thread to return to, 4 years later.

jacobjacob 12 Aug 2024 18:07 UTC
6 points
2
on: Leaving MIRI, Seeking Funding
FYI: I skimmed the post quickly and didn’t realize there was a Patreon!
If you wanted to change that, you might want to put it at the very end of the post, on a new line, saying something like: “If you’d like to fund my work directly, you can do so via Patreon [here](link).”

jacobjacob 23 Jul 2024 6:24 UTC
75 points
20
on: jacobjacob’s Shortform Feed
Someone posted these quotes in a Slack I’m in… what Ellsberg said to Kissinger:
“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret.
“I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you.
[...]
“In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself.
“You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.”
(link)

jacobjacob 18 Jul 2024 4:26 UTC
4 points
2
in reply to: garrison’s comment on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
tbf I never realized “sic” was mostly meant to point out errors, specifically. I thought it was used to mean “this might sound extreme—but I am in fact quoting literally”

jacobjacob 16 Jul 2024 22:12 UTC
8 points
0
in reply to: kave’s comment on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
I mean that in both cases he used literally those words.

jacobjacob 16 Jul 2024 18:57 UTC
10 points
8
in reply to: Joseph Miller’s comment on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
It’s not epistemically poor to say these things if they’re actually true.
Invalid.
Compare:
A: “So I had some questions about your finances, it seems your trading desk and exchange operate sort of closely together? There were some things that confused me...”
B: “our team is 20 insanely smart engineers”
A: “right, but i had a concern that i thought perhaps—”
B: “if you join us and succeed you’ll be a multi millionaire”
A: ”...okay, but what if there’s a sudden downturn—”
B: “bull market is inevitable right now”
Maybe not false. But epistemically poor form.
What links here?
- jacobjacob's comment on Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity by Gideon Futerman (EA Forum; 16 Jul 2024 22:15 UTC; 3 points)

jacobjacob 16 Jul 2024 18:52 UTC
5 points
1
in reply to: Joseph Miller’s comment on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
(crossposted to the EA Forum)
(😭 there has to be a better way of doing this, lol)

jacobjacob 16 Jul 2024 3:20 UTC
39 points
8
on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
(crossposted to EA forum)
I agree with much of Leopold’s empirical claims, timelines, and analysis. I’m acting on it myself in my planning as something like a mainline scenario.
Nonetheless, the piece exhibited some patterns that gave me a pretty strong allergic reaction. It made or implied claims like:
- a small circle of the smartest people believe this
- i will give you a view into this small elite group who are the only who are situationally aware
- the inner circle longed tsmc way before you
- if you believe me; you can get 100x richer—there’s still alpha, you can still be early
- This geopolitical outcome is “inevitable” (sic!)
- in the future the coolest and most elite group will work on The Project. “see you in the desert” (sic)
- Etc.
Combined with a lot of retweets, with praise, on launch day, that were clearly coordinated behind the scenes; it gives me the feeling of being deliberately written to meme a narrative into existence via self-fulfilling prophecy; rather than inferring a forecast via analysis.
As a sidenote, this felt to me like an indication of how different the AI safety adjacent community is now to when I joined it about a decade ago. In the early days of this space, I expect a piece like this would have been something like “epistemically cancelled”: fairly strongly decried as violating important norms around reasoning and cooperation. I actually expect that had someone written this publicly in 2016, they would’ve plausibly been uninvited as a speaker to any EAGs in 2017.
I don’t particularly want to debate whether these epistemic boundaries were correct—I’d just like to claim that, empirically, I think they de facto would have been enforced. Though, if others who have been around have a different impression of how this would’ve played out, I’d be curious to hear.

jacobjacob 6 Jul 2024 23:45 UTC
2 points
0
in reply to: John Steidley’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
[censored_meme.png]
I like review bot and think it’s good

jacobjacob 4 Jul 2024 20:13 UTC
37 points
69
in reply to: Zach Stein-Perlman’s comment on: Habryka’s Shortform Feed
(Sidenote: it seems Sam was kind of explicitly asking to be pressured, so your comment seems legit :)
But I also think that, had Sam not done so, I would still really appreciate him showing up and responding to Oli’s top-level post, and I think it should be fine for folks from companies to show up and engage with the topic at hand (NDAs), without also having to do a general AMA about all kinds of other aspects of their strategy and policies. If Zach’s questions do get very upvoted, though, it might suggest there’s demand for some kind of Anthropic AMA event.)