testingthewaters

Karma: 339

testingthewaters Apr 24, 2025, 2:04 AM
7 points
9
in reply to: Neel Nanda’s comment on: MichaelDickens’s Shortform
At the risk of seeming quite combative, when you say

And I know a lot of safety people at deepmind and other AGI labs who I’m very confident also sincerely care about reducing existential risks. This is one of their primary motivations, they often got into the field due to being convinced by arguments about ai risk, they will often raise in conversation concerns that their current work or the team’s current strategy is not focused on it enough, some are extremely hard-working or admirably willing to forgo credits so long as they think that their work is actually mattering for X-Risk, some dedicate a bunch of time to forming detailed mental models of how AI leads to bad outcomes and how this could be prevented and how their work fit in, etc.

That’s basically what I mean when I said in my comment

AI safety, by its nature, resists the idea of creating powerful new information technologies to exploit mercilessly for revenue without care for downstream consequences. However, many actors in the AI safety movement are themselves tied to the digital economy, and depend on it for their power, status, and livelihoods. Thus, it is not that there are no genuine concerns being expressed, but that at every turn these concerns must be resolved in a way that keeps the massive tech machine going. Those who don’t agree with this approach are efficiently selected against. [examples follow]

And, after thinking about it, I don’t see your statement conflicting with mine.

testingthewaters Apr 24, 2025, 1:47 AM
51 points
14
in reply to: MichaelDickens’s comment on: MichaelDickens’s Shortform
I think this is straightforwardly true and basically hard to dispute in any meaningful way. A lot of this is basically downstream of AI research being part of a massive market/profit generating endeavour (the broader tech industry), which straightforwardly optimises for more and more “capabilities” (of various kinds) in the name of revenue. Indeed, one could argue that long before the current wave of LLMs the tech industry was developing powerful agentic systems that actively worked to subvert human preferences in favour of disempowering them/manipulating them, all in the name of extracting revenue from intelligent work… we just called the AI system the Google/Facebook/Youtube/Twitter Algorithm.

The trend was always clear: an idealistic mission to make good use of global telecommunication/information networks finds initial success and is a good service. Eventually pressures to make profits cause the core service to be degraded in favour of revenue generation (usually ads). Eventually the company accrues enough shaping power to actively reshape the information network in its favour, and begins dragging everything down with it. In the face of this AI/LLMs are just another product to be used as a revenue engine in the digital economy.

AI safety, by its nature, resists the idea of creating powerful new information technologies to exploit mercilessly for revenue without care for downstream consequences. However, many actors in the AI safety movement are themselves tied to the digital economy, and depend on it for their power, status, and livelihoods. Thus, it is not that there are no genuine concerns being expressed, but that at every turn these concerns must be resolved in a way that keeps the massive tech machine going. Those who don’t agree with this approach are efficiently selected against. For example:
- Race dynamics are bad? ~~Maybe we should slow down.~~ We just need to join the race and be the more morally-minded actor. After all, there’s no stopping the race, we’re already locked in.
- Our competitors/other parties are doing dangerous things? ~~Maybe we could coordinate and share our concerns and research with them.~~ We can’t fall behind, we’ve got to fly the AI safety flag at the conference of AI Superpowers. Let’s speed up too.
- New capabilities are unknown and jagged? ~~Let’s just leave well enough alone.~~ Let’s invest more in R&D so we can safely understand and harness them.
- Here’s a new paradigm that might lead to a lot of risk and a lot of reward. ~~We should practice the virtue of silence and buy the world time~~ We should make lots of noise so we can get funding. To study it. Not to use it, of course. Just to understand the safety implications.
- Maybe progress in AI is slower than we thought. ~~Hooray! Maybe we can chill for a bit~~ That’s time for us to exploit our superior AI knowledge and accelerate progress to our benefit.
We’ve seen this before.

To be honest, though, I’m not sure what to do about this. So much has been invested by now that it truly feels like history is moving with a will of its own, rather than individuals steering the ship. Every time I look at what’s going on I feel the sense that maybe I’m just the idiot that hasn’t gotten the signal to hammer that “exploit” button. After all, it’s what everyone else is doing.

testingthewaters Apr 19, 2025, 11:35 AM
1 point
0
in reply to: Cole Wyeth’s comment on: A Dissent on Honesty
For my part, I didn’t realise it became so heavily downvoted, but I did not mean it at all in an accusatory or moralizing manner. I also, upon reflection, don’t regret posting it.

testingthewaters Apr 17, 2025, 10:48 AM
−12 points
10
on: A Dissent on Honesty
I’m sorry that your life has come to a point where you might feel like adopting a fake personality and regularly deceiving those around you is how to be happy and get what you want. It’s almost impossible to deliver good advice to a stranger over the internet, but I hope that one day you can find joy and fulfillment in being yourself.

testingthewaters Apr 15, 2025, 4:03 PM
5 points
0
on: OpenAI #13: Altman at TED and OpenAI Cutting Corners on Safety Testing

I think I can indeed forsee the future where OpenAI is helping the Pentagon with its AI weapons. I expect this to happen. I want to be clear that I don’t think this is a bad thing. The risk is in developing highly capable AIs in the first place. As I have said before, Autonomous Killer Robots and AI-assisted weapons in general are not how we lose control over the future to AI, and failing to do so is a key way America can fall behind. It’s not like our rivals are going to hold back. To the extent that the AI weapons scare the hell out of everyone? That’s a feature.

This is giving “guns don’t kill people, people kill people” energy. Sure, ai takeovers can happen without skynet, but skynet sure makes it easier.

testingthewaters Apr 12, 2025, 8:19 PM
6 points
1
on: Why does LW not put much more focus on AI governance and outreach?
The simple answer is related to the population and occupation of the modal lesswrong viewer, and hence the modal lesswrong commenter, and upvoter. The site culture also tends towards skepticism and pessimism of institutions (I do not make a judgement on whether this valence is justified). I however also agree that this is important to at least discuss.

testingthewaters Apr 11, 2025, 2:36 PM
4 points
2
on: testingthewaters’s Shortform
From Inadequate Equilibria:

Visitor: I take it you didn’t have the stern and upright leaders, what we call the Serious People, who could set an example by donning Velcro shoes themselves?

From Ratatouille:

In many ways, the work of a critic is easy. We risk very little, yet enjoy a position over those who offer up their work and their selves to our judgment. We thrive on negative criticism, which is fun to write and to read. But the bitter truth we critics must face, is that in the grand scheme of things, the average piece of junk is probably more meaningful than our criticism designating it so. But there are times when a critic truly risks something, and that is in the discovery and defense of the new. The world is often unkind to new talent, new creations. The new needs friends.

And that’s why bravery is the secret name of the nameless virtue, and seriously underrated.

[[To elaborate slightly: to go beyond pointing and sneering, to actually work to construct a better future, is very difficult. It requires breaking from social conventions, not just the social conventions you claim are “self evidently stupid” but also the ones you see as natural and right. In many ways the hardest task is not to realise what the “right choice” is, but to choose cooperate in face of your knowledge of nash equilibria.

To reach for the pareto optimal solution to a coordination game means knowing you might very well be stabbed in the back. In a world where inadequate equilibria persist the only way we get out is to be the first person to break those equilibria, and that requires you to take some pretty locally irrational actions. Sometimes choosing not to defect or to punish requires unimaginable bravery. Mere knowledge of Moloch does not save you from Moloch, only action does.]]

testingthewaters Apr 5, 2025, 10:27 AM
1 point
0
on: AI CoT Reasoning Is Often Unfaithful
I mean, this applies to humans too. The words and explanations we use for our actions are often just post hoc rationalisations. An efficient text predictor must learn not what the literal words in front of them mean, but the implied scenario and thought process they mask, and that is a strictly nonlinear and “unfaithful” process.

testingthewaters Apr 3, 2025, 12:00 AM
2 points
0
on: testingthewaters’s Shortform
I think I’ve just figured out why decision theories strike me as utterly pointless: they get around the actual hard part of making a decision. In general, decisions are not hard because you are weighing payoffs, but because you are dealing with uncertainty.

To operationalise this: a decision theory usually assumes that you have some number of options, each with some defined payout. Assuming payouts are fixed, all decision theories simply advise you to pick the outcome with the highest utility. “Difficult problems” in decision theory are problems where the payout is determined by some function that contains a contradiction, which is then resolved by causal/evidential/functional decision theories each with their own method of cutting the Gordian knot. The classic contradiction, of course, is that “payout(x1) == 100 iff predictor(your_choice) == x1; else payout(x1) == 1000″.

Except this is not at all what makes real life decisions hard. If I am planning a business and ever get to the point where I know a function for exactly how much money two different business plans will give me, I’ve already gotten past the hard part of making a business plan. Similarly, if I’m choosing between two doors on a game show the difficulty is not that the host is a genius superpredictor who will retrocausally change the posterior goat/car distribution, but the simple fact that I do not know what is behind the doors. Almost all decision theories just skip past the part where you resolve uncertainty and gather information, which makes them effectively worthless in real life. Or, worse, they try to make the uncertainty go away: If I have 100 dollars and can donate to a local homeless shelter I know well or try and give it to a malaria net charity I don’t know a lot about, I can be quite certain the homeless shelter will not misappropriate the funds or mismanage their operation, and less so about the faceless malaria charity. This is entirely missing from the standard EA arguments for allocation of funds. Uncertainty matters.

testingthewaters Apr 1, 2025, 9:03 AM
9 points
0
in reply to: CapResearcher’s comment on: CapResearcher’s Shortform
This has shifted my perceptions of what is in the wild significantly. Thanks for the heads up.

testingthewaters Mar 25, 2025, 8:22 PM
1 point
0
on: testingthewaters’s Shortform
https://research.google/blog/deciphering-language-processing-in-the-human-brain-through-llm-representations/

Activations in LLMs are linearly mappable to activations in the human brain. Imo this is strong evidence for the idea that LLMs/NNs in general acquire extremely human like cognitive patterns, and that the common “shoggoth with a smiley face” meme might just not be accurate

testingthewaters Mar 24, 2025, 4:57 AM
2 points
0
in reply to: Raemon’s comment on: METR: Measuring AI Ability to Complete Long Tasks
That surprisingly straight line reminds me of what happens when you use noise to regularise an otherwise decidedly non linear function: https://www.imaginary.org/snapshot/randomness-is-natural-an-introduction-to-regularisation-by-noise

testingthewaters Mar 21, 2025, 5:22 PM
2 points
0
on: Towards a scale-free theory of intelligent agency
I think this is a really cool research agenda. I can also try to give my “skydiver’s perspective from 3000 miles in the air” overview of what I think expected free energy minimisation means, though I am by no means an expert. Epistemic status: this is a broad extrapolation of some intuitions I gained from reading a lot of papers, it may be very wrong.

In general, I think of free energy minimisation as a class of solutions for the problem of predicting complex systems behaviour, in line with other variational principles in physics. Thus, it is an attempt to use simple physical rules like “the ball rolls down the slope” to explain very complicated outcomes like “I decide to build a theme park with roller coasters in it”. In this case, the rule is “free energy is minimised”, but unlike a simple physical system whose dimensionality is very literally visible, VFE is minimised in high dimensional probability spaces.

Consider the concrete case below: there are five restaurants in a row and you have to pick one to go to. The intuitive physical interpretation is that you can be represented by a point particle moving to one of five coordinates, all relatively close by in the three dimensional XYZ coordinate space. However, if we assume that this is just some standard physical process you’ll end up with highly unintuitive behaviour (why does the particle keep drifting right and left in the middle of these coordinates, and then eventually go somewhere that isn’t the middle?). Instead we might say that in an RL sense there is a 5 dimensional action space and you must pick a dimension to maximise expected reward. Free energy minimisation is a rule that says that your action is the one that minimises variation between the predicted outcome your brain produces and the final outcome that your brain observes—which can happen either if your brain is very good at predicting the future or if you act to make your prediction come true. A preference in this case is a bias in the prediction (you can see yourself going to McDonald’s more, in some sense, and you feel some psychological aversion/repulsive force moving you away from Burger King) that is then satisfied by you going to the restaurant you are most attracted to. Of course this is just a single agent interpretation and with multiple subagents you can imagine valleys and peaks in the high dimensional probability space, which is resolved when you reach some minima that can be satisfied by action.

testingthewaters Mar 19, 2025, 3:37 PM
2 points
0
in reply to: Lee.aao’s comment on: The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
It’s hard to empathise with dry numbers, whereas a lively scenario creates an emotional response so more people engage. But I agree that this seems to be very well done statistical work.

testingthewaters Mar 19, 2025, 12:31 PM
2 points
0
in reply to: Richard_Ngo’s comment on: Elite Coordination via the Consensus of Power
Hey, thank you for taking the time to reply honestly and in detail as well. With regards to what you want, I think that this is in many senses also what I am looking for, especially the last item about tying in collective behaviour to reasoning about intelligence. I think one of the frames you might find the most useful is one you’ve already covered—power as a coordination game. As you alluded to in your original post, people aren’t in a massive hive mind/conspiracy—they mostly want to do what other successful people seem to be doing, which translates well to a coordination game and also explains the rapid “board flips” once a critical mass of support/rejection against some proposition is reached. For example, witness the rapid switch to majority support of gay marriage in the 2010s amongst the population in general.

Would also love to discuss this with you in more detail (I trained as an English student and also studied Digital Humanities). I will leave off with a few book suggestions that, while maybe not directly answering your needs, you might find interesting.
- Capitalist Realism by Mark Fisher (as close to a self-portrait by the modern humanities as it gets)
- Hyperobjects by Timothy Morton (high level perspective on how cultural, material, and social currents impact our views on reality)
- How minds change by David McRaney (not humanities, but pop sci about the science of belief and persuasion)
P.S. Re: the point about Yarvin being right, betting on the dominant group in society embracing a dangerous delusion is a remarkably safe bet. (E.g. McCarthyism, the aforementioned Bavarian Witch Hunts, fascism, lysenkoism etc.)

testingthewaters Mar 19, 2025, 7:53 AM
45 points
5
on: Elite Coordination via the Consensus of Power
Hey, really enjoyed your triple review on power lies trembling, but imo this topic has been… done to death in the humanities, and reinventing terminology ad hoc is somewhat missing the point. The idea that the dominant class in a society comes from a set of social institutions that share core ideas and modus operandi (in other words “behaving as a single organisation”) is not a shocking new phenomenon of twentieth century mass culture, and is certainly not a “mystery”. This is basically how every country has developed a ruling class/ideology since the term started to have a meaning, through academic institutions that produce similar people. Yale and Harvard are as Oxford and Cambridge, or Peking University and Renmin University. (European universities, in particular, started out as literal divinity schools, and hence are outgrowths of the literal Catholic church, receiving literal Papal bulls to establish themselves as one of the studia generalia.) [Retracted, while the point about teaching religious law and receiving literal papal bulls is true the origins of the universities are much more diverse. But my point about the history of cultural hegemony in such institutions still stands.]

What Yarvin seems to be annoyed by is that the “Cathedral consensus” featured ideas that he dislikes, instead of the quasi-feudal ideology of might makes right that he finds more appealing. That is also not surprising. People largely don’t notice when they are part of a dominant class and their ideas are treated as default: that’s just them being normal, not weird. However, when they find themselves at the edge of the overton window, suddenly what was right and normal becomes crushing and oppressive. The natural dominance of sensible ideas and sensible people becomes a twisted hegemony of obvious lies propped up by delusional power-brokers. This perspective shift is also extremely well documented in human culture and literature.

In general, the concept that a homogenous ruling class culture can then be pushed into delusional consensuses which ultimately harms everyone is an idea as old as the Trojan War. The tension between maintaining a grip on power and maintaining a grip on reality is well explored in Yuval Noah Harari’s book Nexus (which also has an imo pretty decent second half on AI). In particular I direct you to his account of the Bavarian witch hunts. Indeed, the unprecedented feature of modern society is the rapid divergence in ideas that is possible thanks to information technology and the cultivation of local echo chambers. Unfortuantely, I have few simple answers to offer to this age old question, but I hope that recognising the lineage of the question helps with disambiguation somewhat. I look forward to your ideas about new liberalisms.

testingthewaters Mar 16, 2025, 2:43 PM
1 point
0
in reply to: the gears to ascension’s comment on: The Fork in the Road
Yeah, I’m not gonna do anything silly (I’m not in a position to do anything silly with regards to the multitrillion param frontier models anyways). Just sort of “laying the groundwork” for when AIs will cross that line, which I don’t think is too far off now. The movie “Her” is giving a good vibe-alignment for when the line will be crossed.

testingthewaters Mar 16, 2025, 2:42 PM
3 points
0
in reply to: Daniel Kokotajlo’s comment on: The Fork in the Road
Ahh, I was slightly confused why you called it a proposal. TBH I’m not sure why only 0.1% instead of any arbitrary percentage between (0, 100]. Otherwise it makes good logical sense.

testingthewaters Mar 15, 2025, 11:18 PM
3 points
0
in reply to: Daniel Kokotajlo’s comment on: The Fork in the Road
Hey, the proposal makes sense from an argument standpoint. I would refine slightly and phrase as “the set of cognitive computations that generate role emulating behaviour in a given context also generate qualia associated with that role” (sociopathy is the obvious counterargument here, and I’m really not sure what I think about the proposal of AIs as sociopathic by default). Thus, actors getting into character feel as if they are somehow sharing that character’s emotions.

I take the two problems a bit further, and would suggest that being humane to AIs may necessarily involve abandoning the idea of control in the strict sense of the word, so yes treating them as peers or children we are raising as a society. It may also be that the paradigm of control necessarily means that we would as a species become more powerful (with the assistance of the AIs) but not more wise (since we are ultimately “helming the ship”), which would be in my opinion quite bad.

And as for the distinction between today and future AI systems, I think the line is blurring fast. Will check out Eleos!

testingthewaters Mar 15, 2025, 7:14 PM
3 points
0
in reply to: Daniel Kokotajlo’s comment on: The Fork in the Road
Hey Daniel, thank you for the thoughtful comment. I always appreciate comments that make me engage further with my thinking because one of the things I do is that I get impatient with whatever post I’m writing and “rush it out of the door”, so to speak, so this gives me another chance to reflect on my thoughts.

I think that there are approximately ~3 defensible positions with regards to AI sentience, especially now that AIs seem to be demonstrating pretty advanced reasoning and human-like behaviour. One is the semi mystical argument that humans/brains/embodied entities have some “special sauce” that AIs will simply never have, and therefore that no matter how advanced AI gets it will never be “truly sentient”. The other is that AI is orthogonal to humans, and as such behaviours that in a human would indicate thought, emotion, calculation etc. are in fact the products of completely alien processes, so “it’s okay”. In other words, they might not even “mind” getting forked and living for only a few objective minutes/hours. The third, which I now subscribe to after reading quite a lot about the free energy principle, predictive processing, and related root-of-intelligence literature, is that intelligent behaviour is the emergent product of computation (which is itself a special class of physical phenomena in higher dimensions), and since NNs seem to demonstrate both human like computations (cf. neural net activations explaining human brain activations and NNs being good generative models of human brains) and human like behaviour, they should have (after extensive engineering and under specific conditions we seem to be racing towards) roughly matching qualia to humans. From this perspective I draw the inferences about factory farms and suffering.

To be clear, this is not an argument that AI systems as they are now constitute “thinking feeling beings” we would call moral patients. However, I am saying that thinking about the problem in the old fashioned AI-as-software way seems to me to undersell the problem of AI safety as merely “keeping the machines in check”. It also seems to lead down a road of dominance/oppositional approaches to AI safety that cast AIs as foreign enemies and alien entities to be subjugated to the human will. This in turn raises both the risks of moral harms to AIs and failing the alignment problem by acting in a way that counts as a self fulfilling prophecy. If we bring entities not so different from us into the world and treat them terribly, we should not be surprised when they rise up against us.