Tristan Wegner

Karma: 96

Tristan Wegner Mar 7, 2025, 9:00 PM
2 points
1
on: On the Rationality of Deterring ASI
The concept of MAIM does imply the US should ALLOW the newest AI chips to be exported to China, because they are the main (rational) state actor that also want to develop AI and would be the one to maim the intentional not hardened US AI efforts, if they cannot keep up. Correct? China needs a similar tech access to keep the balance.

Tristan Wegner May 15, 2024, 3:01 PM
12 points
19
in reply to: mesaoptimizer’s comment on: Ilya Sutskever and Jan Leike resign from OpenAI
Actually a great example of people using the voting system right. It does not contribute anything substantial to the conversation, but just express something most of us feel obviously.
I had to order the 2 votes into the 4 prototypes to makes sure I voted sensibly:
High Karma—Agree: A well expressed opinion I deeply share
High Karma—Disagree: A well argued counterpoint that I would never use myself / It did not convince me.
Low Karma—Agree: Something obvious/trivial/repeated that I agree with, but not worth saying here.
Low Karma—Disagree: low quality rest bucket
Also pure factual statement contribution (helpful links, context etc.) should get Karma votes only, as no opinion to disagree with is expressed.

Tristan Wegner May 15, 2024, 2:51 PM
25 points
2
in reply to: Jelle Donders’s comment on: Ilya Sutskever and Jan Leike resign from OpenAI
On first order, this might have a good effect on safety.
On second order, it might have negative effects, because it increases the risk of and therefor lowers the rate of such companies hiring people openly worrying about AI X-Risk.

Tristan Wegner May 15, 2024, 2:48 PM
14 points
6
in reply to: Lukas_Gloor’s comment on: Ilya Sutskever and Jan Leike resign from OpenAI
Someone serious about alignment seeing dangers better do what is save and not be influenced by a non-disparagement agreement. It might lose them some job prospects and have money and possible lawsuit costs, but if history on earth is on the line? Especially since such a known AI genius would find plenty support from people who supported such open move.
So I hope he assumes talking right NOW it not considered strategically worth it. E.g. He might want to increase his chance to be hired by semi safety serious company (more serious than Open AI, but not enough to hire a proven whistleblower), where he can use his position better.

Tristan Wegner Apr 22, 2024, 10:42 AM
3 points
0
in reply to: Aaron_Scher’s comment on: How to Model the Future of Open-Source LLMs?
I agree with the premise, but not the conclusion of your last point. Any OpenSource development, that will significantly lower the resource requirements can also be used by closed models to just increased their model/training size for the same cost, thus keeping the gap.

Tristan Wegner Apr 21, 2024, 6:57 AM
9 points
5
on: Thoughts on seed oil
The first graph is supposed to show ” BMI at age 50 for white, high-school educated American men born in various years”, but goes up 1986. But People born in 1986 are only 38 right now, so we cannot know their BMI at 50 years old. Something is wrong.

Paper: “The Ethics of Advanced AI Assistants” -Google DeepMind

Tristan WegnerApr 21, 2024, 6:45 AM

20 points

0 comments1 min readLW link

(storage.googleapis.com)

Tristan Wegner Dec 13, 2023, 6:05 AM
1 point
0
in reply to: Bruce W. Lee’s comment on: How LLMs Can Show Self-Serving Bias
No. By off by 100 I meant of by a factor of 100 to small, NOT that they don’t sum up to 100.

Tristan Wegner Nov 25, 2023, 10:31 AM
1 point
0
on: How LLMs Can Show Self-Serving Bias
I think you made a off by 100 error in Unlabeled Evaluation with all win rates <1%

Tristan Wegner Nov 23, 2023, 12:07 PM
1 point
0
in reply to: jbash’s comment on: OpenAI: The Battle of the Board
The former boards only power was to agree/fire to new board members and CEOs.
Pretty sure they only Let Altman back as CEO under the condition of having a strong influence over the new board.

Tristan Wegner Nov 23, 2023, 8:38 AM
5 points
1
on: OpenAI: The Battle of the Board
Just in 5h ago:
Some at OpenAI believe Q* (pronounced Q-Star) could be a breakthrough in the startup’s search for what’s known as artificial general intelligence (AGI), one of the people told Reuters.
https://www.reuters.com/technology/sam-altmans-ouster-openai-was-precipitated-by-letter-board-about-ai-breakthrough-2023-11-22/
Could relate to this Q* for Deep Learning heuristics:
https://x.com/McaleerStephen/status/1727524295377596645?s=20

Tristan Wegner Nov 22, 2023, 7:58 AM
3 points
0
in reply to: Eli Tyre’s comment on: OpenAI: Facts from a Weekend
I have been working all weekend with the OpenAI leadership team to help with this crisis
Jan Leike

Nov 20
I think the OpenAI board should resign
https://x.com/janleike/status/1726600432750125146?s=20

Tristan Wegner Nov 22, 2023, 7:18 AM
5 points
0
in reply to: gwern’s comment on: OpenAI: Facts from a Weekend
Mr. Altman, the chief executive, recently made a move to push out one of the board’s members because he thought a research paper she had co-written was critical of the company.
Here the paper: https://cset.georgetown.edu/publication/decoding-intentions/
Some more recent (Nov/Okt 2023) publications from her here:
https://cset.georgetown.edu/staff/helen-toner/

Tristan Wegner Nov 21, 2023, 8:55 AM
1 point
0
in reply to: Daniel ’s comment on: OpenAI: Facts from a Weekend
From your last link:
Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value.
As the company was doing well recently, with ongoing talks about a investment imply a market cap of $90B, this would mean many employees might have hit their 10x already. The highest payout they would ever get. So all incentive to cash out now (or as soon as the 2-year lock will allow), 0 financial incentive to care about long term value.
This seems worse in aligning employee interest with the long term interest of the company even compare to regular (unlimited allowed growth) equity, where each employee might hope that the valuation could get even higher.
Also:
It’s important to reiterate that the PPUs inherently are not redeemable for value if OpenAI does not turn a profit
So it seems the growth cap actually encourages short term thinking, which seems against their long term mission.
Do you also understand these incentives this way?

Tristan Wegner Nov 18, 2023, 8:08 AM
4 points
1
on: How much to update on recent AI governance moves?
The ousting of Sam Altman by a Board with 3 EA people could be the strongest public move so far.

Tristan Wegner Oct 22, 2023, 2:10 PM
2 points
0
on: LoRA Fine-tuning Efficiently Undoes Safety Training from Llama 2-Chat 70B
Could you compare the average activation of the Lora network with prompts that the original Llama model refused with prompts that it allowed? I would expect more going on when Lora has to modify the output.
Using linear probes in similar manner could also be interesting.

Tristan Wegner Jan 5, 2023, 7:11 AM
2 points
3
in reply to: Vladimir_Nesov’s comment on: 2022 was the year AGI arrived (Just don’t call it that)
Humans/human level can invent technology from distant future if given enough time. Nick Bostrom has a whole chapter about speed intelligence in Superintelligence.

Tristan Wegner Jan 5, 2023, 7:08 AM
1 point
0
in reply to: tailcalled’s comment on: 2022 was the year AGI arrived (Just don’t call it that)
While I agree, there is a positive social feedback mechanism going on right now, where AI/Neural Networks are a very interesting AND well paid field to go into AND the field is young, so a new bright student can be up to speed quite quickly. More motivated smart people leading to more breakthroughs, making AIs more impressive and useful, closing the feedback loop.
I have no doubt that the mainstream popularity of ChatGPT right now decides quite a few career choices.

Tristan Wegner May 13, 2022, 9:35 AM
5 points
in reply to: Aiyen’s comment on: Deepmind’s Gato: Generalist Agent
That is how I have seen the term expert performance used in papers. Yes.

Tristan Wegner

Paper: “The Ethics of Ad­vanced AI As­sis­tants” -Google DeepMind

Paper: “The Ethics of Advanced AI Assistants” -Google DeepMind