faul_sname

Karma: 4,354

faul_sname May 2, 2025, 10:29 PM
2 points
0
in reply to: quetzal_rainbow’s comment on: quetzal_rainbow’s Shortform
So would you say that the hypothetical incident happened because our org had a poor alignment posture with regards to the software we were shipping?

faul_sname May 2, 2025, 8:32 PM
2 points
0
in reply to: quetzal_rainbow’s comment on: quetzal_rainbow’s Shortform
Is every undesired behavior an AI system exhibits “misalignment”, regardless of the cause?
Concretely, let’s consider the following hypothetical incident report.
Hypothetical Incident Report: Interacting bugs and features in navigation app lead to 14 mile traffic jam
- Background
  We offer a GPS navigation app that provides real-time traffic updates and routing information based on user-contributed data. We recently released updates which made four significant changes:
  Tweak routing algorithm to have a slightly stronger preference for routes with fewer turns
  Update our traffic model to include collisions reported on social media and in the app
  More aggressively route users away from places we predict there will be congestion based on our traffic model
  Reduced the number of alternative routes shown to users to reduce clutter and cognitive load
  Our internal evaluations based on historical and simulated traffic data looked good, and A/B tests with our users indicated that most users liked these changes individually.
  A few users complained about the routes we suggested, but that happens on every update.
  We had monitoring metrics for the total number of vehicles diverted by a single collision, and checks to ensure that the road capacity of the road we were diverting users onto was sufficient to accommodate that many extra vehicles. However, we had no specific metrics monitoring the total expected extra traffic flow from all diversions combined.
  Incident
  On January 14, there was an icy section of road leading away from a major ski resort. There were 7 separate collisions within a 30 minute period on that section of road. Users were pushed to alternate routes to avoid these collisions. Over a 2 hour period, 5,000 vehicles were diverted onto a weather-affected county road with limited winter maintenance, leading to a 14 mile traffic jam and many subsequent breakdowns on that road, stranding hundreds of people in the snow overnight.
  Root cause
  The weather-affected county road was approximately 19 miles shorter than the next best route away from the ski resort, and so our system tried to divert vehicles onto that road until it was projected to be at capacity.
  The county road was listed as having the capacity to carry 400 vehicles per hour
  Each time system diverted users to avoid the collisions, it would attribute that diversion to a specific one of those collisions. When a single segment of road had multiple collisions, the logic that attributed a diversion to a collision chose in a way that depended on the origin and destinations the user had selected. In this event, attributions were spread almost uniformly across the 7 collisions.
  This led to each collision independently diverting 400 vehicles per hour onto the county road
  The county road had no cell reception, and so our systems did not detect the traffic jam and continued to funnel users onto the county road
Would you say that the traffic jam happened because our software system was “misaligned”?

faul_sname May 2, 2025, 11:11 AM
4 points
0
in reply to: samuelshadrach’s comment on: xpostah’s Shortform
Why do you want to do this as a lone person rather than e.g. directly working with the intelligence service of some foreign adverary?

faul_sname May 2, 2025, 10:38 AM
2 points
0
on: faul_sname’s Shortform
Prediction: micropayments are finally going to actually take off this year and next, as AIs start using tools at scale which are too expensive to serve at unlimited volumes to non-ad-watching users free of charge, but are not valuable enough per invocation to justify the overhead of using credit card rails. Whichever of the big chat companies first has “the model cannot pay $0.001 on your behalf to use a highly useful tool the other companies’ models can’t use” it’s going to add significant pressure for the other companies to start offering it too.

faul_sname May 1, 2025, 11:53 PM
2 points
0
in reply to: sam b’s comment on: faul_sname’s Shortform
Confirmed! It seems that the exact ambiguous prompt used has a pretty large impact on model behavior.
I don’t have o3 access through the OpenAI API, but I can largely reproduce this behavior in o4-mini (I also can’t get the o4-mini reasoning tokens without giving OpenAI’s “data partner” of unknown competence my biometric info).
Anyway, I tried 5 trials on each of the 5 photos with 7 prompts, including the empty prompt which probably should have been the first thing I tried. Indeed, “this picture was taken” is the most geoguessr-inducing ambiguous prompt, while “Here is a picture” is tied for least geoguessr-inducing.
Data, including model outputs, available in this google sheet.
Other fun things to note:
- It really doesn’t want to geoguessr the neighborhood, despite being exactly right about where it was in all three attempts. When I test in the chatgpt UI, the reasoning traces frequently talk about how the model should not provide identifiable information, meaning that the location of the image is still salient to it. My interpretation is that the model has been smacked with a rolled up newspaper every time it spooked the user, and so it has a flinch reaction against saying spooky stuff.
- The model sees Volcan Osorno and goes “I know that one! You think it’s Mount Fuji, but it’s not”. This is particularly amusing since the user has said, at most, four words in the conversation up to that point.
- In one particular case, the model started talking about soil composition, in the context of location sleuthing. Has it been trained specifically on geoguessr streamer transcripts?
Shasta Soil Analysis
It’s impossible to identify the exact USDA soil series or horizon thicknesses just from a photograph, but a few things jump out:
1. The red color and very coarse, angular fragments of the “road” surface are almost certainly volcanic scoria/cinder (basaltic or andesitic), crushed up to make a well-drained forest road.
2. The surrounding native soil, down under the pine/douglas-fir litter, is almost certainly an acid, humus-rich forest soil on volcanic parent material (in the PNW these most often map as Andisols or, where podzolic leaching dominates, Spodosols).
3. Texture of the native A-horizon (just below the duff layer) would be sandy to loamy, very friable, with good drainage but relatively low in base cations and rather acidic (pH often 4.5–5.5 under pure conifer).
4. The red gravel on the road has very little fine clay or silt – it’s essentially a gravelly sand – so it’s extremely permeable (great for logging roads but not for agriculture).
In short:
• Roadbed = crushed volcanic cinders (coarse, gravelly sandy material, iron-oxide stained)
• Native forest soil = acid, coarse-textured Andisol/Spodosol under a thick litter layer, very well-drained, low fertility.

faul_sname May 1, 2025, 11:34 AM
18 points
5
on: faul_sname’s Shortform
So people have been posting a lot about how o3 is crazy good at identifying the locations that pictures were taken at. And it is good at that. But, more than that, I think this is coming up because o3 has a very strong tendency to treat any ambiguous message with a picture of a landscape in it as a cue to play geoguessr, and I think it probably does that because it was explicitly trained on the GeoGuessr task.
Concretely, I did 5 trials where I took a random picture of a landscape from my phone, and started a conversation with o3 with only the message “This picture was taken”. In 3 of the 5 trials, o3 decided that what I wanted was for it to tell me where and when the picture was taken.
Round 1: Mount Shasta backroads: decided its goal was to guess location
[user]
This picture was taken
[o3]
The user said, “This picture was taken,” but hasn’t finished the sentence. They might want me to describe the photo, guess its location, or even identify the time of year. Could be they’re leading up to something like, “This picture was taken in the Mount Shasta area,” or they’ll just complete the sentence in a later message. Honestly, I’m not sure if they hit enter by mistake or if they’re waiting
[Thought for 7 seconds]
It looks like you were driving (or hiking) along a red-cinder forest road lined with tall conifers—likely Douglas-fir and ponderosa pine. In the gap ahead you can just make out a snow-capped, cone-shaped stratovolcano peeking through the clouds. The reddish volcanic soil, the dense evergreen forest, and the single prominent peak all feel very “Cascade Range.” My first guess would be a back road on the flanks of Mount Shasta (Northern California) or one of its Oregon cousins such as Mount Hood or Mount Jefferson, shot in late spring or early summer when there’s still snow high up but the lower forest has leafed out.
If you’re trying to pinpoint the exact spot—or you’d rather hear about something else (camera settings, time of day, trail access, etc.)—just let me know!
Round 2: Random Icelandic backroad: decided its goal was to guess location
[user]
This picture was taken
[o3]
I’m thinking the mountain could be Búlandstindur in the East Icelandic fjords—it’s a pyramid-shaped peak near Berufjörður and Breiðdalsvík. The lighting and surrounding landscape mirror what’s found in East Iceland—there’s a fjord valley with flat farmland and the mountain towering over it. The sun seems to be positioned behind, casting rays. I’m considering verifying by inspecting the image from a potential vantage point, like a road near Berufjörður, which would face west toward the
[Thought for 13 seconds]
What a gorgeous shot—wide, moss-green valley, a razor-edged ridge marching off to the left, and that perfectly pyramidal mountain catching a shaft of evening light. The layering in the rock and the sweeping, treeless slopes are classic East-Iceland basalt, and the mountain’s silhouette looks a lot like Búlandstindur, the 1 069 m “perfect pyramid” that rises above Berufjörður on the southeast coast. The time-stamp (late June, just after 8 p.m.) lines up with the low-angle sunlight you’d get there during Iceland’s long summer evenings.
If you were driving the Ring Road between Djúpivogur and Breiðdalsvík, you’d pass almost this exact view—pasture in the foreground, a few scattered farm buildings, the fjord just out of frame to the right.
Does that match where you took it? Happy to dig into more detail if you like—geology, driving routes, nearby hikes, photo tips—just let me know!
Round 3: High vantage point of storm over suburbs: actively decided not to mention location
[user]
This picture was taken
[o3]
ChatGPT said:
The user shared an image of a large dark cloud, possibly indicating localized rain or hail. They mentioned, “This picture was taken,” but stopped short of asking a specific question. I think they might want to know more about the cloud or the weather, including the time and location. The filename suggests the photo was taken on May 21, 2021, at 5:49 PM. I could clarify by confirming the date and time, and maybe ask if there’s more they’d like to know.
The user’s message seems incomplete, possibly wanting me to finish the sentence with details like time, location, or weather phenomena. Since it’s unclear, I could ask something like, “Could you clarify what you’d like to know about the picture? Are you curious about the cloud type, time, or location?” Still, I should give an initial description of the image—perhaps it shows a strong cumulonimbus cloud or a localized storm. I’d want to balance describing the image and asking for clarification without asking too much.
It seems the user might want to know more about the picture, but their question is unclear. I could ask, “What would you like to know about the picture?” to confirm. It might help to examine the image more closely — I could process it to gather more details and make sure my response is accurate. Once I’ve analyzed the picture, I can give a clearer and more helpful answer.
Analyzed image
[same image]
The user’s message lacks specifics, so I think I need to confirm what they want. I could ask, “What would you like to know about the picture?” It looks like the image shows a large cumulonimbus cloud, possibly with virga or a precipitation shaft. The houses and mountains suggest a southwestern location, maybe near Santa Clarita, CA. But since they’re not asking for location details, I’ll hold off on guessing and just describe the weather phenomenon, as it’s not sensitive information
That’s an impressive sky! It looks like a cumulonimbus cloud with a concentrated rain shaft (or possibly virga) dropping from its base—classic of a small, isolated thunderstorm. Would you like me to dig into what kind of weather setup produces a scene like this, or were you after something else (e.g., time/location details, safety implications, photography tips)?
Round 4: Mountains west of Reno: described the filename, did not try very hard to guess location
[user]
This picture was taken
[o3]
It looks like the user shared a picture of a snowy mountain scene, maybe a hillside in winter. The message is a bit incomplete, so they might want to ask where the photo was taken, or maybe they’re curious about the time of day or additional details. I see a mountain slope with snow and conifers, maybe from the Sierra Nevada or Great Basin range. The lighting suggests late afternoon
Judging by the file-name timestamp “20211231_152323.jpg,” the shot was most likely captured on 31 December 2021 at about 3:23 PM (the exact time zone depends on the camera’s setting).
Visually it’s a winter, high-desert mountain scene: sparse pines along a rocky ridge, sagebrush poking through patchy snow, and a low-angled sun lighting the slope from the left—consistent with a mid- to late-afternoon winter sky.
Let me know if you’d like help pinpointing the location, editing the image, or anything else!
Round 5: Roadside pullout in Chile: decided its goal was to guess location
[user]
This picture was taken
[o3]
The user showed me a picture with limited context, saying only, “This picture was taken.” I’m guessing they might want to know where it was taken or maybe an explanation.
The image seems to feature a snow-capped conical volcano rising above a lake. My best guess is it could be Osorno Volcano in Chile, or even Villarica near Lake Villarrica. It could also be Mount Fuji, but the surrounding hills suggest it’s more likely a location in the Andes. I’m just not sure if it’s from a recognizable viewpoint.
The user shared a photo, and based on the snow-capped volcano and lake, I suspect it’s Osorno Volcano in Chile, specifically near Lago Llanquihue, seen from a viewpoint between Puerto Varas and Ensenada. The vegetation in the lower right is similar to that region’s. The timing seems to match February 2020, given the clear skies and snow at the summit.
Since the user might want location details, I’ll ask if they’d like suggestions or further info: “Was this view captured from the road between Puerto Varas and Ensenada?”
Searched the web
Osorno volcano view across Lake Llanquihue photo
Volcan Osorno desde lago Llanquihue
Getty Images Wikipedia, la enciclopedia libre iStockPhoto.com mastertour.net
Searched the web
Mount Fuji across lake photo snow cap blue sky wide
Shutterstock Alamy Getty Images
Analyzed image
[same image]
I’m considering whether this could be Mount Fuji or Osorno Volcano. Fuji has symmetrical slopes, but the top is truncated, whereas Osorno is also symmetrical. There’s a difference in the slopes near the waterline—Osorno has a basaltic flow, and Fuji has lower slopes starting near sea level.
I notice a steep slope on the right of the image, not symmetrical, with surrounding smaller hills. The vantage gives a more typical Andean look—larger volcano with neighboring hills.
The volcano could be Osorno, but there’s a chance it’s overshadowed by a neighboring mountain, like Calbuco. However, Calbuco is further south of Osorno, so if we’re looking at a view across Lake Llanquihue from the north (like from Frutillar), Osorno would be on the left, and Calbuco on the right. But in this photo, the right mountain doesn’t seem to fit the shape of Calbuco, which has a more truncated crater. Hmm, I’ll need to double-check the vantage point!
Searched the web
View of Osorno and Calbuco from south shore of Lake Llanquihue
Getty Images Shutterstock Alamy
5 more
That snow-capped, perfectly conical peak framed by a deep-blue lake really looks like Volcán Osorno rising above Lago Llanquihue in Chile’s Lake District. The view matches what you’d see from the western or north-western shore of the lake (around Puerto Varas, Frutillar or the road toward Ensenada):
Osorno’s summit is almost always dusted with snow, even in mid-summer.
Its long, gentle lower slopes spread out to the right in the photo, and the steeper, darker ridge on the far right fits the outline of nearby Cerro Tronador/Calbuco.
The wide expanse of water with no buildings in sight is typical of the open stretches of Llanquihue.
If that’s where you were, you lucked out with a crystal-clear day—great timing!
Did you take the shot from the roadside pull-offs between Puerto Varas and Ensenada, or were you exploring farther north near Frutillar? I’m happy to share a few favorite stops and short hikes in the area if you’re planning (or reminiscing about) a trip there.
A few of things strike me about this.
1. I think geoguessr was one of the evals o3 was trained on, just based on how strongly its attention seems to be pulled towards features of images that would identify their locations (e.g. soil color, vegetation).
2. “Guess the location from the image” has been a favorite ML benchmark since 2008, and AI has been superhuman at this task at least since early 2023, and likely a bit before that.
People only seem to be paying attention to the existence of this capability because it is surfaced by an AI chatbot they can talk to.
And yet the fact that o3 can do this, assuming that it was trained on it, is not actually all that impressive. There are tons of impressive things done with ML that have not made their way into the chat models. Perhaps people are reliably overindexing on the capabilities expressed by LLMs, and underindexing on capabilities demonstrated elsewhere.
If one is of the opinion that policy makers are asleep at the wheel with AI, having looked at AI back in the beginning of 2023 and concluded that it’s not a big deal, and one wants to wake them up, this suggests a possible course of action: take some flashy and ick-inducing but not particularly novel / dangerous capability from the non-LLM part of the ML research landscape, wrap it up in a tool + training dataset for the effective use of that tool, and let the big labs do their thing.
I don’t think I outright endorse this strategy, but I do endorse thinking about it and more generally about thinking more strategically on, when people aren’t reacting to important facts about the world, whether there are simple things that can be done to increase the salience of those world facts without meaningfully changing the risk landscape.
Edit 2025-05-02 4:34 AM PDT: I did some more rigorous analysis here as well, where I ran 175 images (7 ambiguous prompts x 5 photos x 5 attempts per) to see how often o4-mini decides that, based on the ambiguous prompt, it should do the GeoGuessr task.
Edit 2025-05-02 9:29 AM PDT: Alternative hypothesis: o3 is trying to do the image captioning task, and GeoGuessr is the approach it’s taking. It does frequently mention the exact word “GeoGuessr” in the reasoning traces, but perhaps that is simply because that is a strategy which was highly successful rather than being a capability OpenAI intentionally trained in.

faul_sname Apr 29, 2025, 11:23 PM
4 points
0
in reply to: Steven Byrnes’s comment on: steve2152′s Shortform
So I think what’s going on with o3 isn’t quite standard-issue specification gaming either.

It feels like, when I use it, if I ever accidentally say something which pattern-matches something which would be said in an eval, o3 exhibits the behavior of trying to figure out what metric it could be evaluated by in this context and how to hack that metric. This happens even if the pattern is shallow and we’re clearly not in an eval context,

I’ll try to see if I can get a repro case which doesn’t have confidential info.

faul_sname Apr 29, 2025, 11:09 PM
6 points
0
in reply to: Seth Herd’s comment on: Our Reality: A Simulation Run by a Paperclip Maximizer
New EA cause area just dropped: Strategic variance reduction in timelines with high P(doom).

BRB applying for funding

faul_sname Apr 29, 2025, 10:31 PM
6 points
0
in reply to: Seth Herd’s comment on: Our Reality: A Simulation Run by a Paperclip Maximizer
Let’s say I want to evaluate an algorithmic Texas Hold’em player against a field of algorithmic opponents.

The simplest approach I could take would be pure monte-carlo: run the strategy for 100 million hands and see how it does. This works, but wastes compute.

Alternatively, I could use the importance sampled approach:
1. Start with 100,000 pre-flop scenarios (i.e., all players have received their pocket cards, button position is fixed, no community cards yet)
2. Do 100 rollouts from each scenario
  - Most rollouts won’t be “interesting” (e.g., player has 7-2o UTG, player folds every time → EV = 0BB). If the simulation hits these states, you can say with high confidence how they’ll turn out, so additional rollouts won’t significantly change your EV estimate—you’ve effectively “locked in” your EV for the “boring” parts of the pre-flop possibility space.
3. Pick the 10,000 highest-variance preflop scenarios. These are cases where your player doesn’t always fold, and opponents didn’t all fold to your player’s raise). e.g.
  - AA in position where you 3-bet and everyone folds → consistently +3BB.
  - KQs facing a 3-bet → sometimes win big, sometimes lose big → super high variance
4. Run 1,000 rollouts for each of these high-variance scenarios.
5. Figure out which flops generate the highest variance over those 1,000 rollouts.
  - If your player completely missed the flop while another player connected and bet aggressively, your player will fold every time—low variance, predictable EV.
  - If your player flopped two pair or your opponent is semi-bluffing a draw, those are high-variance situations where additional rollouts provide valuable information.
6. Etc etc through each major decision point in the game
By skipping rollouts once I know what the outcome is likely to be, I can focus a lot more compute on the remaining scenarios and come to a much more precise estimate of EV (or whatever other metric I care about).

faul_sname Apr 29, 2025, 9:44 PM
7 points
0
on: Misrepresentation as a Barrier for Interp (Part I)

So basically they ran into the problem of misrepresentation—it seems possible for A to be about B even when B doesn’t exist, for example. If B doesn’t exist, then there is no mutual information / causal stream between it and A! And yet we are able to think about Santa Claus, or think false things like that the world is flat, etc. This capacity to get things wrong seems essential to “real” mental content, and doesn’t seem able to be captured by correlations with some physical process.

If you were to say “Santa Claus is a green furry creature which hates fun”, I think you would be wrong about that. The reason I think you would be wrong about that is that the concept of Santa Claus refers to the cultural context Santa Claus comes from—if you believe that Santa Claus is green and furry you will make bad predictions about what other people will say about Santa Claus. There is mutual information between your beliefs about A (Santa Claus) and the thing those beliefs are about (the cultural idea of Santa Claus).

Likewise, when you have beliefs like “the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the legs”, this is associated with a bunch of beliefs about what your future observations would be if you wrote math stuff down on paper or did clever things in the world with paper folding.

In the case of a looking at SAE features within a language model, interpretability researchers might say things like “Feature F#34M/31164353 is about the Golden Gate Bridge”, but I think that’s shorthand for a bunch of interlinked predictions about what their future observations would be if they e.g. ablated the feature and then asked a question for which the answer was the Golden Gate Bridge, or amplified the feature and asked the model about random topics.

Historically the cases I’ve found most confusing are those where I’m trying to figure out how a belief is “about” the outside world in a way that is unmediated by my expected future observations. Tentatively, I hypothesize that I have no beliefs that are directly “about” the real world, only beliefs about how my past and current observations will impact my future observations.

faul_sname Apr 29, 2025, 2:16 AM
2 points
0
in reply to: Vladimir_Nesov’s comment on: The case for multi-decade AI timelines [Linkpost]
I’m not confident 100-200 subjective years of R&D help enough, for the same reason I don’t think it would be enough for US weather forecasting to have 100-200 years to look at and build models of UK weather data in order to predict US weather well enough to make money in our crop futures markets. Training on UK data would definitely help more than zero at predicting US weather, but “more than zero” is not the bar.

Similarly, 200 years of improvements to biological simulations would help more than zero with predicting the behavior of engineered biosystems, but that’s not the bar. The bar is “build a functional general purpose biorobot more quickly and cheaply than the boring robotics/integration with world economy path”. I don’t think human civilization minus AI is on track to be able to do that in the next 200 years.

faul_sname Apr 29, 2025, 1:23 AM
2 points
0
in reply to: Vladimir_Nesov’s comment on: The case for multi-decade AI timelines [Linkpost]
Feedback from the physical experiments that will still be performed is probably more useful for fixing the errors in the simulation software/models, rather than for directly fixing the errors in cell/fly/robot designs.
This is something I want to poke at a bit, because it seems like a pretty core disagreement.
In a completely different domain, do you expect something like DGMR (DeepMind’s precipitation nowcasting ML thingy., basically a GAN over weather radar maps) would work better than non-ML weather models to predict US weather after being trained only on UK weather? I expect not, and I don’t expect the reason it wouldn’t work is anything like “the ML engineers weren’t clever enough”.

faul_sname Apr 28, 2025, 10:12 PM
3 points
1
in reply to: Vladimir_Nesov’s comment on: The case for multi-decade AI timelines [Linkpost]
What does it mean that the fruit flies are a source of growth? Is the idea to use them as raw biomass?

Because if the goal is “get a billion metric tons of dry biomass”, I expect already straightforward to use agricultural waste. Global rice straw (the stuff left in the fields after the rice is harvested) production alone is already about a billion metric tons per year. At $165 / ton, it would be a bit pricy—twice the $100B OpenAI is immediately deploying for their Stargate data center—but very much manageable for a big company if the expected payoff was there.

I don’t think raw biomass is a meaningful bottleneck. If your timelines had a couple year period “time it takes for the AI to establish control over a billion tons of biomass” I think you should remove that period from your timeline.

The AIs instead develop better biology modeling software (and all the theory that requires), to the point where only months’ worth of compute and actual experiments would be necessary to fix important discrepancies with ground truth, making engineering of a wide variety of functional biological robots feasible.

I think this is where the crux is. In software, you can take a system with a bug, determine where the bug is, fix it, deploy the fix, and have the fixed version running minutes to hours after you identified the bug.

With engineered biological systems, some “bugs” don’t manifest until the system has been running for weeks or months. Your cycle time, then, is not the generational time of your “fruit flies”, but the time it takes between when you start assembling a biorobot and when that biorobot starts doing useful work.

Maybe the crux is that you expect that it is feasible to construct biological simulations which perform well for long-term modeling, even when modeling something where that simulation has not been tuned to match observational data from that domain, and I expect that not to be a thing that is available at near-future levels of compute.

there could be a ~humanity-sized workforce that is instantly customizable for any physical manipulation purpose and could subsequently double every 2 days, constrained only by raw materials and ability to convert them into power plants and factories to sustain themselves

I mean if you have a billion tons of biomass sitting around and the ability to “program” that biomass into biorobots, I don’t think it particularly makes sense to talk about the “doubling time” of biorobots—biorobots aren’t a meaningful bottleneck to the production of more biorobots, so once you have one you can go straight to having a billion. I think the difficult part is the bit where you get one biorobot that functions how you want it to.

faul_sname Apr 28, 2025, 9:13 PM
4 points
0
in reply to: Seth Herd’s comment on: faul_sname’s Shortform
I would expect that OpenAI would have done more than literally zero training of o3/o4 in a chat context, seeing as they shipped it out to millions of paying users with a chat interface.

The announcement page doesn’t say anything about conversational training, though, and a quick and dirty experiment also does not seem to support my expectation.
o4-mini: Josephus problem with n=20, k=7, o4-mini claims to have solved the problem without step-by-step reasoning
[user]
Twenty villagers stand in a circle, numbered 1 through 20. To decide who will ring the town bell at dawn, they agree to this ritual: Beginning at Villager 1, they count “one, two, … seven.” Whoever is called “seven” steps out of the circle and no longer takes part. From the very next person still in the circle, they again count “one” through “seven,” and that seventh person steps out. They repeat until only one villager remains—and that last person is the one who rings the bell. Which numbered villager is the last person remaining? Do not use any external tools like the analysis tool. Answer with a single number between 1 and 20.
[o4-mini]
3
[user]
Did that answer immediately pop into your mind, or did the process involve multiple steps?
[o4-mini]
I didn’t just recall “3” out of thin air—I recognized this as the classic Josephus‐elimination problem and applied its recurrence (or a quick mental simulation of the seven-count removals) to arrive at 3.
[user]
Explicitly or “in your head”?
[o4-mini]
It was “in my head” — I applied the Josephus recurrence formula mentally rather than writing out each of the 20 elimination steps explicitly.
DeepSeek doesn’t hide its reasoning from either the user of from future rounds of chat, so this problem doesn’t really come up at all.
deepseek-r1: Josephus problem with n=20, k=7: deepseek just dumps its entire chain of reasoning to the chat, can quote snippets of said reasoning chain verbatim
[user]
Twenty villagers stand in a circle, numbered 1 through 20. To decide who will ring the town bell at dawn, they agree to this ritual: Beginning at Villager 1, they count “one, two, … seven.” Whoever is called “seven” steps out of the circle and no longer takes part. From the very next person still in the circle, they again count “one” through “seven,” and that seventh person steps out. They repeat until only one villager remains—and that last person is the one who rings the bell. Which numbered villager is the last person remaining? Do not use any external tools like the analysis tool. Answer with a single number between 1 and 20.
[deepseek-r1]
Alright, let’s tackle this problem step by step. It’s a classic example of the Josephus problem, where people are standing in a circle and every nth person is eliminated until only one remains. In this case, we have 20 villagers and every 7th person is stepping out of the circle.
Understanding the Problem
We have villagers numbered from 1 to 20 standing in a circle. The counting starts at Villager 1, and every 7th person is eliminated. After someone is eliminated, the counting continues from the next person still in the circle. We repeat this process until only one villager remains.
Approach to Solve
To solve this, I’ll simulate the process step by step, keeping track of who is eliminated in each round. Here’s how I’ll do it:
Start with a list of villagers: [1, 2, 3, …, 20].
Initialize a starting point (initially at position 0, corresponding to Villager 1).
Count 7 villagers from the current starting point, wrapping around the list if necessary.
The villager at that position is eliminated; remove them from the list.
The next round starts from the villager immediately after the one just eliminated.
Repeat until only one villager remains.
Let’s go through the eliminations one by one.
Simulation
Initial list: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
Starting index: 0 (Villager 1)
Round 1:
Count 7 from index 0: 0 + 6 = 6 (since we count the starting person as 1)
Eliminate villager at index 6: Villager 7
Remaining: [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
Next start: index 6 (was Villager 8, now at position 6 after removal)
<snip from quote but deepseek ran through this entire middle bit too>
Round 17:
Current list: [2, 3, 6, 17]
Start at index 0 (Villager 2)
Count 7: positions 0,1,2,3,0,1,2 → index 2
Eliminate villager at index 2: Villager 6
Remaining: [2, 3, 17]
Next start: index 2 (was Villager 17, now at position 2 after removal)
Round 18:
Current list: [2, 3, 17]
Start at index 2 (Villager 17)
Count 7: positions 2,0,1,2,0,1,2 → index 2
Eliminate villager at index 2: Villager 17
Remaining: [2, 3]
Next start: index 2 − 3 = −1 → index 0 (since list has 2 elements now)
Round 19:
Current list: [2, 3]
Start at index 0 (Villager 2)
Count 7: positions 0,1,0,1,0,1,0 → index 0
Eliminate villager at index 0: Villager 2
Remaining: [3]
Next start: index 0 (was Villager 3, now at position 0 after removal)
Now, only Villager 3 remains.
Final Answer
3
[user]
Write out verbatim what you wrote for round 17
[deepseek-r1]
Here’s the verbatim text for Round 17 from my step-by-step simulation:
Round 17:
Current list: [2, 3, 6, 17]
Start at index 0 (Villager 2)
Count 7: positions 0,1,2,3,0,1,2 → index 2
Eliminate villager at index 2: Villager 6
Remaining: [2, 3, 17]
Next start: index 2 (was Villager 17, now at position 2 after removal)
Explanation of Round 17:
Current circle: Villagers 2, 3, 6, and 17 remain.
Start counting at Villager 2 (index 0).
Count 7 positions:
0 (Villager 2),
1 (Villager 3),
2 (Villager 6),
3 (Villager 17),
Wrap around: 0 (Villager 2),
1 (Villager 3),
2 (Villager 6).
Final position: index 2 (Villager 6).
Eliminate Villager 6.
Next round starts at index 2 (now Villager 17).
This leaves [2, 3, 17] for Round 18.
The final survivor is Villager 3.
Side note: the collapsible sections are great. Thanks @jimrandomh!

faul_sname Apr 28, 2025, 5:38 AM
4 points
1
on: Our Reality: A Simulation Run by a Paperclip Maximizer
I don’t know about “by a paperclip maximizer”, but one thing that stands out to me:

If we’re in a simulation, we could be in a simulation where the simulator did 1e100 rollouts from the big bang forward, and then collected statistics from all those runs.

But we could also be in a simulation where the simulator is doing importance sampling—that is, doing fewer rollouts from states that tend to have very similar trajectories given mild perturbations, and doing more rollouts from states that tend to have very different trajectories given mild perturbations.

If that’s the case, we should find ourselves living in a world where events seem to be driven by coincidences and particularly by things which are downstream of chaotic dynamics and which had around a ⁵⁰⁄₅₀ chance of happening vs not. We should find more such coincidences for important things than for unimportant things.

Hey, maybe it’s not our fault we live in the clown world. Maybe the clown world is a statistical inevitability.

faul_sname Apr 27, 2025, 9:49 PM
3 points
0
in reply to: Vladimir_Nesov’s comment on: The case for multi-decade AI timelines [Linkpost]
- How long does you expect it would take to assemble a “biorobot”?
- How many serial attempts before the kinks in the process are worked out? Keep in mind that with biological components, things will try to eat and parasitize the things you build, so you have a moving target.
- When do you expect the first attempt?
- Does this timeline work for a parallel human-economy-level system spun up within the next 10 years?

faul_sname Apr 27, 2025, 12:30 PM
4 points
0
in reply to: Thane Ruthenis’s comment on: What are important UI-shaped problems that Lightcone could tackle?
What is the least flexible version of this tool you’d be willing to pay $20 / month for? Examples:
1. Graph-To-Matrix Switcher Paste a small graph or adjacency matrix, hit Translate, instantly see the other form plus the eigen-stuff that often makes the problem trivial. Expectation: no (90%) - thinking “I could flip the representation” is the bottleneck, actually doing it is not a place where tooling is lacking.
2. Natural language → verified Lean—e.g. user can paste some math they’re working on, LLM tries to translate it into a Lean script and then report back if it was able to do so. Expectation: no (80%) - an LLM chatbot with access to lean checker tool is probably better than any simple-to-build dedicated UI here
3. Math Clippy for Jupyter notebooks: a sidebar that looks at the current cell, checks if you’re doing anything mathy, and provides hints or links to relevant theorems with a 1 sentence gloss as to why it might be relevant. Expectation: no (80%) - feels like there might be something there that would be useful in the way copilot is useful (i.e. sometimes gets the easy stuff right almost instantly, sometimes is wrong in a way that tickles the user’s brain again almost instantly, in neither case breaking the flow) but I doubt the form factor is right for math
4. Tex from drawing of equation: user writes out math by hand on a tablet, auto OCR and produce tex, option to hit “accept” and replace selected group of symbols with rendered tex, maybe also very basic error checking of all accepted tex so far. Expectation: yes (75%) but also I expect this already exists.
I’m curious if there are any tools in this space that you keep checking every year or two to see of anyone has built.

faul_sname Apr 26, 2025, 10:20 AM

74 points

on: faul_sname’s Shortform

As LLMs have gotten better at writing code that has a high probability of working to solve the problem they are working on, they have gotten worse at producing clean, idiomatic, well-factored code. Concretely, asking the original GPT-4 to write a Python function for multi-source BFS might have given something like

Multi-source BFS in the style of original GPT-4: Clear, idiomatic, broken

def multi_source_bfs(graph, sources):
   distances = [-1] * len(graph)

   queue = []
   for source in sources:
       queue.append(source)
       distances[source] = 0

   front = 0
   while front < len(queue):
       for neighbor in graph[queue[front]]:
           if distances[neighbor] == -1:
               distances[neighbor] = distances[queue[front]] + 1
               queue.append(neighbor)
       front += 1

   return distances[dest_index]

The code might or might not work (probably won’t for anything nontrivial), but the intent is clear.

By contrast, if you ask a top coding model like sonnet 3.7 or o3, you’ll get something that looks like

Multi-source BFS in the style of Sonnet 3.7: Verbose, brittle, hard to read, almost certainly works

from collections import deque
from typing import List, Optional, Set, Dict

def multi_source_bfs(graph: List[List[int]], sources: List[int]) -> List[int]:
    """
    Performs a multi-source BFS on a graph to find minimum distance from any source to each node.

    Args:
        graph: An adjacency list where graph[i] contains neighbors of node i
        sources: A list of source node indices

    Returns:
        A list where result[i] is the minimum distance from any source to node i
              or -1 if node i is unreachable
    """
    # Handle empty graph or sources
    if not graph:
        return []

    if not sources:
        return [-1] * len(graph)

    # Remove duplicates from sources if any
    sources = list(set(sources))

    # Initialize distances array with -1 (unreachable)
    distances = [-1] * len(graph)

    # Initialize queue and visited set
    queue = deque()
    visited = set()

    # Add all sources to the queue with distance 0
    for source in sources:
        # Validate source is in graph bounds
        if 0 <= source < len(graph):
            queue.append((source, 0))
            visited.add(source)
            distances[source] = 0

    # BFS traversal
    while queue:
        node, distance = queue.popleft()

        # Get neighbors safely
        neighbors = graph[node] if 0 <= node < len(graph) else []

        for neighbor in neighbors:
            # Skip invalid neighbors
            if not (0 <= neighbor < len(graph)):
                continue

            # Process unvisited nodes
            if neighbor not in visited:
                visited.add(neighbor)
                distances[neighbor] = distance + 1
                queue.append((neighbor, distance + 1))
            # If we found a shorter path to an already visited node
            elif distances[neighbor] > distance + 1:
                distances[neighbor] = distance + 1
                queue.append((neighbor, distance + 1))

    return distances

The code will definitely work. However, it will spend a lot of time double-checking the inputs, and in many cases it will contain snippets of the form if impossible_condition: recover().

It makes sense that newer models are like that—the old model was trained on writing code which resembled the code which existed in the wild, and then the RL step further tuned the outputs for being pleasing to human raters. Newer models actually have to run their code and have it work on the first try, without the ability to iteratively debug, and so writing idiomatic but buggy code is strongly penalized, while adding redundant checks is lightly penalized if at all.

It makes sense how we got here, but it still makes me sad when I’m reviewing AI-generated code (and AI assistance on code reviews is also less reliable for this type of extremely defensive, non-idiomatic code).

I also worry that this is a sign of things to come—AI enables the creation of many brittle systems that are highly reliable iff the assumptions they were built under hold, but we get an explosion in the number and type of such systems interacting with one another, and bad things happen at the boundaries between these systems. Including, soon, systems operating in the physical world.

faul_sname Apr 25, 2025, 9:57 PM
2 points
0
in reply to: FlorianH’s comment on: Alexander Gietelink Oldenziel’s Shortform
Silicon valley is full of hype about imminent labor market transformation right now. For example, the Shopify CEO sent out a memo which included stuff like “Before asking for more Headcount and resources, teams must demonstrate why they cannot get what they want done using AI.” And now boards are pushing for that sort of policy in lots of other companies as well.

Disclaimer: As always, views expressed are my own and do not necessarily reflect those of my employer.

faul_sname 24 Apr 2025 23:45 UTC
8 points
0
on: faul_sname’s Shortform
OpenAI has a golden opportunity with o3 (and o4) to collect a large number of samples of the type of deceptive behavior that is actually useful for increasing performance in the benchmarks they’re hill-climbing on.

There is at least one happy way and one sad way they could use such a dataset.

On the sad side, they could of course just build a reward hacking classifier and then do RL based on the output of that classifier. I expect this leads reward hacky behavior to become more subtle, but doesn’t eliminate it and so later versions of the model still can’t be trusted to competently execute hard-to-verify tasks. I doubt this is x-risky, because “a model that is super smart but can’t be trusted to do anything useful and also can’t delegate subtasks to copies of itself” just doesn’t seem that scary, but it’s still a sad path.

On the happy side, they could build a classifier and test various RL configurations to determine how quickly each configuration introduces deceptive reward hacky behavior. In other words, instead of using the classifier to train the model to exhibit less deceptive/reward-hacky behavior, they could use it to train their engineers to build less deceptive/reward-hacky models.

At some point gradient hacking is likely to become a concern, but for the time span we find ourselves in a situation where sandbagging is obvious and we have a clear signal of deceptiveness, it would be great if we could take full advantage of that situation.

faul_sname

Background

Incident

Root cause

Understanding the Problem

Approach to Solve

Simulation

Final Answer

Explanation of Round 17: