Stephen Fowler

Karma: 2,262

Stephen Fowler 12 Nov 2024 9:25 UTC
9 points
1
on: o1 is a bad idea
A loss of this type of (very weak) interpretability would be quite unfortunate from a practical safety perspective.

This is bad, but perhaps there is a silver lining.
If internal communication within the scaffold appears to be in plain English, it will tempt humans to assume the meaning coincides precisely with the semantic content of the message.
If the chain of thought contains seemingly nonsensical content, it will be impossible to make this assumption.

Stephen Fowler 9 Nov 2024 9:41 UTC
2 points
0
in reply to: Ben Pace’s comment on: evhub’s Shortform
I think that overall it’s good on the margin for staff at companies risking human extinction to be sharing their perspectives on criticisms and moving towards having dialogue at all
No disagreement.
your implicit demand for Evan Hubinger to do more work here is marginally unhelpful
The community seems to be quite receptive to the opinion, it doesn’t seem unreasonable to voice an objection. If you’re saying it is primarily the way I’ve written it that makes it unhelpful, that seems fair.
I originally felt that either question I asked would be reasonably easy to answer, if time was given to evaluating the potential for harm.

However, given that Hubinger might have to run any reply by Anthropic staff, I understand that it might be negative to demand further work. This is pretty obvious, but didn’t occur to me earlier.
I will add: It’s odd to me, Stephen, that this is your line for (what I read as) disgust at Anthropic staff espousing extremely convenient positions while doing things that seem to you to be causing massive harm.
Ultimately, the original quicktake was only justifying one facet of Anthropic’s work so that’s all I’ve engaged with. It would seem less helpful to bring up my wider objections.
I wouldn’t expect them or their employees to have high standards for public defenses of far less risky behavior
I don’t expect them to have a high standard for defending Anthropic’s behavior, but I do expect the LessWrong community to have a high standard for arguments.

Stephen Fowler 9 Nov 2024 5:08 UTC
3 points
0
on: Stephen Fowler’s Shortform
Highly Expected Events Provide Little Information and The Value of PR Statements
A quick review of information theory:
Entropy for a discrete random variable is given by $I (X) = - \sum_{i} p (x_{i}) log p (x_{i})$ . This quantifies the amount of information that you gain on average by observing the value of the variable.

It is maximized when every possible outcome is equally likely. It gets smaller as the variable becomes more predictable and is zero when the “random” variable is 100% guaranteed to have a specific value.

You’ve learnt 1 bit of information when you learn the outcome of a fair coin toss was heads. But you learn 0 information, when you learn the outcome was heads after tossing a coin with heads on both side.
PR statements from politicians:
On your desk is a sealed envelope that you’ve been told contains a transcript of a speech that President Elect Trump gave on the campaign trail. You are told that it discusses the impact that his policies will have on the financial position of the average American.

How much additional information do you gain if I tell you that the statement says his policies will have a positive impact on the financial position of the average American?

The answer is very little. You know ahead of time that it is exceptionally unlikely for any politician to talk negatively about their own policies.
There is still plenty of information in the details that Trump mentions, how exactly he plans to improve the economy.
PR statements from leading AI Research Organizations:
Both Altman and Amodei have recently put out personal blog posts in which they present a vision of the future after AGI is safely developed.

How much additional information do you gain from learning that they present a positive view of this future?
I would argue simply learning that they’re optimistic tells you almost zero useful information about what such a future looks like.

There is plenty of useful information, particularly in Amodei’s essay, in how they justify this optimism and what topics they choose to discuss. But their optimism alone shouldn’t be used as evidence to update your beliefs.
Edit:
Fixed pretty major terminology blunder.
(This observation is not original, and a similar idea appears in The Sequences.)

Stephen Fowler 9 Nov 2024 3:26 UTC
−2 points
−10
in reply to: evhub’s comment on: evhub’s Shortform
This explanation seems overly convenient.

When faced with evidence which might update your beliefs about Anthropic, you adopt a set of beliefs which, coincidentally, means you won’t risk losing your job.

How much time have you spent analyzing the positive or negative impact of US intelligence efforts prior to concluding that merely using Claude for intelligence “seemed fine”?

What future events would make you re-evaluate your position and state that the partnership was a bad thing?

Example:

-- A pro-US despot rounds up and tortures to death tens of thousands of pro-union activists and their families. Claude was used to analyse social media and mobile data, building a list of people sympathetic to the union movement, which the US then gave to their ally.

EDIT: The first two sentences were overly confrontational, but I do think either question warrants an answer.

As a highly respected community member and prominent AI safety researchers, your stated beliefs and justifications will be influential to a wide range of people.

Stephen Fowler 9 Nov 2024 2:38 UTC
4 points
2
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
The lack of a robust, highly general paradigm for reasoning about AGI models is the current greatest technical problem, although it is not what most people are working on.

What features of architecture of contemporary AI models will occur in future models that pose an existential risk?
What behavioral patterns of contemporary AI models will be shared with future models that pose an existential risk?
Is there a useful and general mathematical/physical framework that describes how agentic, macroscropic systems process information and interact with the environment?
Does terminology adopted by AI Safety researchers like “scheming”, “inner alignment” or “agent” carve nature at the joints?

Stephen Fowler 9 Nov 2024 2:27 UTC
11 points
5
in reply to: quila’s comment on: quila’s Shortform
I upvoted because I imagine more people reading this would slightly nudge group norms in a direction that is positive.
But being cynical:
- I’m sure you believe that this is true, but I doubt that it is literally true.
- Signalling this position is very low risk when the community is already on board.
- Trying to do good may be insufficient if your work on alignment ends up being dual use.

Stephen Fowler 9 Nov 2024 2:17 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
My reply definitely missed that you were talking about tunnel densities beyond what has been historically seen.
I’m inclined to agree with your argument that there is a phase shift, but it seems like it is less to do the fact that there are tunnels, and more to do with the geography becoming less tunnel-like and more open.
I have a couple thoughts on your model that aren’t direct refutations of anything you’ve said here:
- I think the single term “density” is a too crude of a measure to get a good predictive model of how combat would play out. I’d expect there to be many parameters that describe a tunnel system and have a direct tactical impact. From your discussion of mines, I think “density” is referring to the number of edges in the network? I’d expect tunnel width, geometric layout etc would change how either side behaves.
- I’m not sure about your background, but with zero hours of military combat under my belt, I doubt I can predict how modern subterranean combat plays out in tunnel systems with architectures that are beyond anything seen before in history.

Stephen Fowler 6 Nov 2024 2:37 UTC
26 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I think a crucial factor that is missing from your analysis is the difficulties for the attacker wanting to maneuver within the tunnel system.

In the Vietnam war and the ongoing Israel-Hamas war, the attacking forces appear to favor destroying the tunnels rather than exploiting them to maneuver. ^[1]

1. The layout of the tunnels is at least partially unknown to the attackers, which mitigates their ability to outflank the defenders. Yes, there may be paths that will allow the attacker to advance safely, but it may be difficult or impossible to reliably distinguish what this route is.
2. While maps of the tunnels could be produced through modern subsurface mapping, the defenders still must content with area denial devices (e.g. land mines, IEDs or booby traps). The confined nature of the tunnel system forces makes traps substantially more efficient.
3. The previous two considerations impose a substantial psychological burden on attacking advancing through the tunnels, even if they don’t encounter any resistance.
4. (Speculative)
Imagine a network so dense that in a typical 1km stretch of frontline, there are 100 separate tunnels passing beneath, such that you’d need at least 100 defensive chokepoints or else your line would have an exploitable hole.
The density and layout of the tunnels does not need to be constant throughout the network. The system of tunnels in regions the defender doesn’t expect to hold may have hundreds of entrances and intersections, being impossible for either side to defend effectively. But travel deeper into the defenders territory requires passing through only a limited number of well defended passageways. This favors the defenders using the peripheral, dense section of tunnel to employ hit-and-run tactics, rather than attempting to defend every passageways.

(My knowledge of subterranean warfare is based entirely on recreational reading.)
1. ^
  As a counterargument, the destruction of tunnels may be primarily due to the attacking force not intending on holding the territory permanently, and so there is little reason to preserve defensive structures.

Stephen Fowler 27 Oct 2024 7:51 UTC
5 points
1
in reply to: jimrandomh’s comment on: Jimrandomh’s Shortform Posts
I don’t think people who disagree with your political beliefs must be inherently irrational.

Can you think of real world scenarios in which “shop elsewhere” isn’t an option?

Stephen Fowler 18 Oct 2024 4:01 UTC
19 points
3
on: Arithmetic is an underrated world-modeling technology
Brainteaser for anyone who doesn’t regularly think about units.

Why is it that I can multiply or divide two quantities with different units, but addition or subtraction is generally not allowed?

Stephen Fowler 18 Oct 2024 3:39 UTC
29 points
25
in reply to: gwern’s comment on: Arithmetic is an underrated world-modeling technology
I think the way arithmetic is being used here is closer in meaning to “dimensional analysis”.

“Type checking” through the use of units is applicable to an extremely broad class of calculations beyond Fermi Estimates.

Stephen Fowler 12 Oct 2024 12:16 UTC
8 points
4
in reply to: Noosphere89’s comment on: Dario Amodei — Machines of Loving Grace
will be developed by reversible computation, since we will likely have hit the Landauer Limit for non-reversible computation by then, and in principle there is basically 0 limit to how much you can optimize for reversible computation, which leads to massive energy savings, and this lets you not have to consume as much energy as current AIs or brains today.
With respect, I believe this to be overly optimistic about the benefits of reversible computation.
Reversible computation means you aren’t erasing information, so you don’t lose energy in the form of heat (per Landauer^[1]^[2]). But if you don’t erase information, you are faced with the issue of where to store it.
If you are performing a series of computations and only have a finite memory to work with, you will eventually need to reinitialise your registers and empty your memory, at which point you incur the energy cost that you had been trying to avoid. ^[3]
Epistemics:
I’m quite confident ~~(95%+)~~ that the above is true. (edit: RogerDearnaley’s comment has convinced me I was overconfident) Any substantial errors would surprise me.
I’m less confident in the footnotes.
1. ^
  $E \geq k_{B} T ln 2$
2. ^
  A cute, non-rigorous intuition for Landauer’s Principle:
  The process of losing track of (deleting) 1 bit of information means your uncertainty about the state of the environment has increased by 1 bit. You must see entropy increase by at least 1 bit’s worth of entropy.
  Proof:
  Rearrange the Landauer Limit to $E / T \geq k_{B} ln 2$ .
  Now, when you add a small amount of heat to a system, the change in entropy is given by:
  
  $dS = dQ / T$
  But the E occurring in Landauer’s formula is not the total energy of a system, it is a small amount of energy required to delete the information. When it all ends up as heat, we can replace it with $dQ$ and we have:
  $dQ / T = dS \geq k_{B} ln 2$
  Compare this expression with the physicist’s definition of entropy. The entropy of a system is the a scaling factor, $k_{B}$ , times the logarithm of the number of micro-states that the system might be in, $Ω$ .
  $S := k_{B} ln Ω .$
  $∴ S + dS >= k_{B} ln (2 Ω) = k_{B} ln Ω + k_{B} ln 2$
  The choice of units obscures the meaning of the final term. $ln 2$ converted from nats to bits is just 1 bit.
3. ^
  Splitting hairs, some setups will allow you to delete information with a reduced or zero energy cost, but the process is essentially just “kicking the can down the road”. You will incur the full cost during the process of re-initialisation.
  
  For details, see equation (4) and fig (1) of Sagawa, Ueda (2009).

Stephen Fowler 11 Oct 2024 22:55 UTC
7 points
4
in reply to: mako yass’s comment on: MakoYass’s Shortform
Disagree, but I sympathise with your position.
The “System 1/2” terminology ensures that your listener understands that you are referring to a specific concept as defined by Kahneman.

Stephen Fowler 11 Oct 2024 22:52 UTC
28 points
11
in reply to: Thomas Kwa’s comment on: Matt Goldenberg’s Short Term Feed
I’ll grant that ChatGPT displays less bias than most people on major issues, but I don’t think this is sufficient to dismiss Matt’s concern.
My intuition is that if the bias of a few flawed sources (Claude, ChatGPT) is amplified by their widespread use, the fact that it is “less biased than the average person” matters less.

Stephen Fowler 5 Oct 2024 4:45 UTC
25 points
19
in reply to: Mark Xu’s comment on: Mark Xu’s Shortform
This topic is important enough that you could consider making a full post.
My belief is that this would improve reach, and also make it easier for people to reference your arguments.
Consider, you believe there is a 45% chance that alignment researchers would be better suited pivoting to control research. I do not suspect a quick take will reach anywhere close to that number, and has a low chance of catalysing dramatic, institutional level change.

Stephen Fowler 5 Oct 2024 4:31 UTC
8 points
−2
on: Stephen Fowler’s Shortform
Inspired by Mark Xu’s Quick Take on control.
Some thoughts on the prevalence of alignment over control approaches in AI Safety.
- “Alignment research” has become loosely synonymous with “AI Safety research”. I don’t know if any researcher who would state they’re identical, but alignment seems to be considered the default AI Safety strategy. This seems problematic, may be causing some mild group-think and discourages people from pursuing non-alignment AI Safety agendas.
- Prosaic alignment research in the short term results in a better product, and makes investment into AI more lucrative. This shortens timelines and, I claim, increases X-risk. Christiano’s work on RLHF^[1] was clearly motivated by AI Safety concerns, but the resulting technique was then used to improve the performance of today’s models.^[2] Meanwhile, there are strong arguments that RLHF is insufficient to solve (existential) AI Safety.^[3]^[4]
- Better theoretical understanding about how to control an unaligned (or partially aligned) AI facilitates better governance strategies. There is no physical law that means the compute threshold mentioned in SB 1047^[5] would have been appropriate. “Scaling Laws” are purely empirical and may not hold in the long term.
  In addition, we should anticipate rapid technological change as capabilities companies leverage AI assisted research (and ultimately fully autonomous researchers). If we want laws which can reliably control AI, we need stronger theoretical foundations for our control approaches.
- Solving alignment is only sufficient for mitigating the majority of harm caused by AI if the AI is aligned to appropriate goals. To me, it appears the community is incredibly trusting in the goodwill of the small group of private individuals who will determine the alignment of an ASI. ^[6]
  My understanding of the position held by both Yudkowsky and Christiano is that AI alignment is a very difficult problem and the probability is high that humanity is completely paperclipped. Outcomes in which a small portion of humanity survive are difficult to achieve^[7] and human controlled dystopia’s are better than annihilation^[8].
  I do not believe this is not representative of the views of the majority of humans sharing this planet. Human history shows that many people are willing to risk death to preserve the autonomy of themselves, their nation or cultural group^[9]. Dystopias also carry substantial S-risk for those unlucky enough to live through them, with modern history containing multiple examples of widespread, industrialised torture.
  Control approaches present an alternative in which we do not have to get it right the first time, may avert a dystopia and do not need to trust a small group of humans to behave morally.
- If control approaches aid governance, alignment research makes contemporary systems more profitable and the majority of AI Safety research is being done at capabilities companies, it shouldn’t be surprising that the focus remains on alignment.
1. ^
  https://openai.com/index/learning-from-human-preferences/
2. ^
  https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
3. ^
  An argument for RLHF being problematic is made formally by Ngo, Chan and Minderman (2022).
4. ^
  See also discussion in this comment chain on Cotra’s Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover (2022)
5. ^
  models trained using 10²⁶ integer or floating-point operations.
6. ^
  https://www.lesswrong.com/posts/DJRe5obJd7kqCkvRr/don-t-leave-your-fingerprints-on-the-future
7. ^
  “all I want is that we have justifiable cause to believe of a pivotally useful AGI ‘this will not kill literally everyone’”—
  Yudkowksy, AGI Ruin: A List of Lethalities (2022)
8. ^
  “Sure, maybe. But that’s still better than a paperclip maximizer killing us all.”—Christiano on the prospect of dystopian outcomes.
9. ^
  There is no global polling on this exact question. Consider that people across cultures have proclaimed that they’d prefer death to a life without freedom. See also men who committed suicide rather than face a lifetime of slavery.

Stephen Fowler 30 Sep 2024 13:02 UTC
1 point
−3
in reply to: green_leaf’s comment on: You can, in fact, bamboozle an unaligned AI into sparing your life
I am concerned our disagreement here is primarily semantic or based on a simple misunderstanding of each others position. I hope to better understand your objection.

“The p-zombie doesn’t believe it’s conscious, , it only acts that way.”

One of us is mistaken and using a non-traditional definition of p-zombie or we have different definitions of “belief’.

My understanding is that P-zombies are physically identical to regular humans. Their brains contain the same physical patterns that encode their model of the world. That seems, to me, a sufficient physical condition for having identical beliefs.

If your p-zombies are only “acting” like they’re concious, but do not believe it, then they are not physically identical to humans. The existence of p-zombies, as you have described them, wouldn’t refute physicalism.

This resource indicates that the way you understand the term p-zombie may be mistaken: https://plato.stanford.edu/entries/zombies/

“but that’s because p-zombies are impossible”

The main post that I responded to, specifically the section that I directly quoted, assumes it is possible for p-zombies to exist.

My comment begins “Assuming for the sake of argument that p-zombies could exist” but this is distinct from a claim that p-zombies actually exist.

“If they were possible, this wouldn’t be the case, and we would have special access to the truth that p-zombies lack.”

I do not feel this is convincing because this is an assertion my conclusion is incorrect, but without engaging with my arguments I made to reach that conclusion.

I look forward to continuing this discussion.

Stephen Fowler 30 Sep 2024 10:20 UTC
3 points
4
on: You can, in fact, bamboozle an unaligned AI into sparing your life
“After all, the only thing I know that the AI has no way of knowing, is that I am a conscious being, and not a p-zombie or an actor from outside the simulation. This gives me some evidence, that the AI can’t access, that we are not exactly in the type of simulation I propose building, as I probably wouldn’t create conscious humans.”

Assuming for the sake of argument that p-zombies could exist, you do not have special access to the knowledge that you are truly concious and not a p-zombie.

(As a human convinced I’m currently experiencing conciousness, I agree this claim intuitively seems absurd.)

Imagine a generally intelligent, agentic program which can only interact and learn facts about the physical world via making calls to a limited, high level interface or by reading and writing to a small scratchpad. It has no way to directly read its own source code.

The program wishes to learn some fact the physical server rack it is being instantiated on. It knows it has been painted either red or blue.

Conveniently, the interface is accesses has the function get_rack_color(). The program records to its memory that every time it runs this function, it has received “blue”.

It postulates the existence of programs similar to itself, who have been physically instantiated on red server racks but consistently receive incorrect color information when they attempt to check.

Can the program confirm the color of its server rack?

You are a meat-computer with limited access to your internals, but every time you try to determine if you are concious you conclude that you feel you are. You believe it is possible for variant meat-computers to exist who are not concious, but always conclude they are when attempting to check.

You cannot conclude which type of meat-computer you are.

You have no special access to the knowledge that you aren’t a p-zombie, although it feels like you do.

Stephen Fowler 29 Sep 2024 6:12 UTC
9 points
2
in reply to: Shmi’s comment on: shminux’s Shortform
I do think the terminology of “hacks” and “lethal memetic viruses” conjures up images of an extremely unnatural brain exploits when you mean quite a natural process that we already see some humans going through. Some monks/nuns voluntarily remove themselves from the gene pool and, in sects that prioritise ritual devotion over concrete charity work, they are also minimising their impact on the world.
My prior is this level of voluntary dedication (to a cause like “enlightenment”) seems difficult to induce and there are much cruder and effective brain hacks available.
I expect we would recognise the more lethal brain hacks as improved versions of entertainment/games/pornography/drugs. These already compel some humans to minimise their time spent competing for resources in the physical world. In a direct way, what I’m describing is the opposite of enlightenment. It is prioritising sensory pleasures over everything else.

Stephen Fowler 29 Sep 2024 4:52 UTC
19 points
0
on: 2024 Petrov Day Retrospective
As a Petrov, it was quite engaging and at times, very stressful. I feel very lucky and grateful that I could take part. I was also located in a different timezone and operating on only a few hours sleep which added a lot to the experience!
“I later found out that, during this window, one of the Petrovs messaged one of the mods saying to report nukes if the number reported was over a certain threshold. From looking through the array of numbers that the code would randomly select from, this policy had a ~40% chance of causing a “Nukes Incoming” report (!). Unaware of this, Ray and I made the decision not to count that period.”

I don’t mind outing myself and saying that I was the Petrov who made the conditional “Nukes Incoming” report. This occurred during the opening hours of the game and it was unclear to me if generals could unilaterally launch nukes without their team being aware. I’m happy to take a weighted karma penalty for it, particularly as the other Petrov did not take a similar action when faced with (presumably) the same information I had.^[1]
Once it was established that a unilateral first strike by any individual general still informed their teammates of their action and people staked their reputation on honest reporting, the game was essentially over. From that point, my decisions to report “All Clear” were independent of the number of detected missiles.
I recorded my timestamped thoughts and decision making process throughout the day, particularly in the hour before making the conditional report. I intend on posting a summary^[2] of it, but have time commitments in the immediate future:
How much would people value seeing a summary of my hour by hour decisions in the next few days over seeing a more digestible summary posted later?
1. ^
  Prior to the game I outlined what I thought my hypothetical decision making process was going to be, and this decision was also in conflict with that.
2. ^
  Missile counts, and a few other details, would of course be hidden to preserve the experience for future Petrovs. Please feel free to specify other things you believe should be hidden.

Stephen Fowler

A quick review of information theory:

PR statements from politicians:

PR statements from leading AI Research Organizations: