Amalthea

Karma: 641

Amalthea 20 Dec 2024 13:19 UTC
1 point
0
in reply to: Davidmanheim’s comment on: Announcement: AI for Math Fund
I’m not saying that it’s not worth pursuing as an agenda, but I also am not convinced it is promising enough to justify pursuing math related AI capabilities, compared to e.g. creating safety guarantees into which you can plug in AI capabilities once they arise anyway.

Amalthea 20 Dec 2024 11:02 UTC
1 point
0
in reply to: Davidmanheim’s comment on: Announcement: AI for Math Fund
I think the “guaranteed safe AI” framework is just super speculative. Enough to basically not matter as an argument given any other salient points.

This leaves us with the baseline, which is that this kind of prize re-directs potentially a lot of brainpower from more math-adjacent people towards thinking about AI capabilities. Even worse, I expect it’s mostly going to attract the un-reflective “full-steam-ahead” type of people.

Mostly, I’m not sure it matters at all except maybe slightly accelerating some inevitable development before e.g. deep mind takes another shot at it to finish things off.

Amalthea 6 Dec 2024 13:18 UTC
1 point
9
in reply to: Chris_Leong’s comment on: Announcement: AI for Math Fund
Agreed, I would love to see more careful engagement with this question.

Amalthea 16 Nov 2024 11:36 UTC
1 point
0
in reply to: Garrett Baker’s comment on: D0TheMath’s Shortform
You’re putting quite a lot of weight on what “mathematicians say”. Probably these people just haven’t thought very hard about it?

Amalthea 23 Oct 2024 20:21 UTC
1 point
0
in reply to: eggsyntax’s comment on: The Mask Comes Off: At What Price?
I believe the confusion comes from assuming the current board follows rules rather than doing whatever is most convenient.

The old board was trying to follow the rules, and the people in question were removed (technically were pressured to remove themselves).

Amalthea 21 Oct 2024 11:14 UTC
11 points
4
in reply to: rotatingpaguro’s comment on: OpenAI defected, but we can take honest actions
I’d agree the OpenAI product line is net positive (though not super hung up on that). Sam Altman demonstrating what kind of actions you can get away with in front of everyone’s eyes seems problematic.

Amalthea 14 Oct 2024 20:27 UTC
3 points
2
in reply to: Nathan Helm-Burger’s comment on: leogao’s Shortform
Or simply when scaling becomes too expensive.

Amalthea 13 Oct 2024 18:35 UTC
3 points
2
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
There’s a lot of problems with linking to manifold and calling it “the expert consensus”!
- It’s not the right source. The survey you linked elsewhere would be better.
- Even for the survey, it’s unclear whether these are the “right” experts for the question. This at least needs clarification.
- It’s not a consensus, this is a median or mean of a pretty wide distribution.
I wouldn’t belabor it, but you’re putting quite a lot of weight on this one point.

Amalthea 13 Oct 2024 18:04 UTC
1 point
0
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
I mean it only suggests that they’re highly correlated. I agree that it seems likely they represent the views of the average “AI expert” in this case. (I should take a look to check who was actually sampled)

My main point regarding this is that we probably shouldn’t be paying this particular prediction market too much attention in place of e.g. the survey you mention. I probably also wouldn’t give the survey too much weight compared to opinions of particularly thoughtful people, but I agree that this needs to be argued.

Amalthea 13 Oct 2024 14:24 UTC
1 point
0
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
In general, yes—but see the above (I.e. we don’t have a properly functioning prediction market on the issue).

Amalthea 13 Oct 2024 12:42 UTC
1 point
0
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
Who do you consider thoughtful on this issue?

It’s more like “here are some people who seem to have good opinions”, and that would certainly move the needle for me.

Amalthea 13 Oct 2024 12:27 UTC
1 point
0
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
I mostly try to look around to who’s saying what and why and find that the people I consider most thoughtful tend to be more concerned and take “the weak argument” or variations thereof very seriously (as do I). It seems like the “expert consensus” here (as in the poll) is best seen as some sort of evidence rather than a base rate, and one can argue how much to update on it.

That said, there’s a few people who seem less overall concerned about near-term doom and who I take seriously as thinkers on the topic. Carl Shulman being perhaps the most notable.

Amalthea 13 Oct 2024 11:01 UTC
1 point
−1
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
It eliminates all the aspects of prediction markets that theoretically make them superior to other forms of knowledge aggregation (e.g. surveys). I agree that likely this is just acting as a (weirdly weighted) poll in this case, so the biased resolution likely doesn’t matter so much (but that also means the market itself tells you much less than a “true” prediction market would).

Amalthea 13 Oct 2024 7:18 UTC
1 point
0
in reply to: Logan Zoellner’s comment on: Most arguments for AI Doom are either bad or weak
Fair! I’ve mostly been stating where I think your reasoning looks suspicious to me, but that does end up being points that you already said wouldn’t convince you. (I’m also not really trying to)

Relatedly, this question seems especially bad for prediction markets (which makes me consider the outcome only in an extremely weak sense). First, it is over an extremely long time span so there’s little incentive to correct. Second, and most importantly, it can only ever resolve to one side of the issue, so absent other considerations you should assume that it is heavily skewed to that side.

Amalthea 12 Oct 2024 12:54 UTC
5 points
1
on: Most arguments for AI Doom are either bad or weak
I basically only believe the standard “weak argument” you point at here, and that puts my probability of doom given strong AI at 10-90% (“radical uncertainty” might be more appropriate).

It would indeed seem to me that either I) you are using the wrong base-rate or 2) you are making unreasonably weak updates given the observation that people are currently building AI, and it turns out it’s not that hard.

I’m personally also radically uncertain about correct base rates (given that we’re now building AI) so I don’t have a strong argument for why yours is wrong. But my guess is your argument for why yours is right doesn’t hold up.

Amalthea 11 Oct 2024 18:45 UTC
1 point
0
in reply to: Seth Herd’s comment on: sarahconstantin’s Shortform

If it’s not serving them, it’s pathological by definition, right?

This seems way too strong, otherwise any kind of belief or emotion that is not narrowly in pursuit of your goals is pathological.

I completely agree that it’s important to strike a balance between revisiting the incident and moving on.

but most of the effect is in the repetition of thoughts about the incident and fear of future similar experiences.

This seems partially wrong. The thoughts are usually consequences of the damage that is done, and they can be unhelpful in their own right, but they are not usually the problem. E.g. if you know that X is an abuser and people don’t believe you, I wouldn’t go so far as saying your mental dissonance about it is the problem.

Amalthea 10 Oct 2024 21:55 UTC
3 points
0
in reply to: Noosphere89’s comment on: Joshua Achiam Public Statement Analysis
The point that would justify an airstrike isn’t violation of a treaty, but posing an immediate and grave risk to the international community. The treaty is only the precondition that makes effective and coordinated action possible.

Amalthea 10 Oct 2024 13:41 UTC
1 point
0
in reply to: Nathan Young’s comment on: Nathan Young’s Shortform
Is there a reason to be so specific, or could one equally well formulate this more generally?

Amalthea 9 Oct 2024 7:29 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Bengio and Hinton are the two most influential “old guard” AI researchers turned safety advocates as far as I can tell, with Bengio being more active in research. Your e.g. is super misleading, since my list would have been something like:
1. Bengio
2. Hinton
3. Russell

Amalthea 7 Oct 2024 19:18 UTC
4 points
1
in reply to: sarahconstantin’s comment on: sarahconstantin’s Shortform
One has to be a bit careful with this though. E.g. someone experiencing or having experienced harassment may have a seemingly pathological obsession on the circumstances and people involved in the situation, but it may be completely proportional to the way that it affected them—it only seems pathological to people who didn’t encounter the same issues.