But is this comparable to G? Is it what we want to measure?
Matt Goldenberg
Karma: 5,515
Brain surgeon is the prototypical “goes last”example:
a “human touch” is considered a key part of the health care
doctors have strong regulatory protections limiting competition
Literal lives at at stake and medical malpractice is one of the most legally perilous areas imaginable
Is neuralink the exception that proves the rule here? I imagine that IF we come up with live saving or miracle treatments that can only be done with robotic surgeons, we may find a way through the red tape?
This exists and is getting more popular, especially with coding, but also in other verticals
This is great, matches my experience a lot
I think they often map onto three layers of training—First, the base layer trained by next token prediction, then the rlhf/dpo etc, finally, the rules put into the prompt
I don’t think it’s perfectly like this, for instance, I imagine they try to put in some of the reflexive first layer via dpo, but it does seem like a pretty decent mapping
When you start trying to make an agent, you realize how much your feedback, rerolls, etc are making chat based llms useful
the error correction mechanism is you in a chat based llms, and in the absence of that, it’s quite easy for agents to get off track
you can of course add error correction mechanism like multiple llms checking each other, multiple chains of thought, etc, but the cost can quickly get out of hand
It’s been pretty clear to me as someone who regularly creates side projects with ai that the models are actually getting better at coding.
Also, it’s clearly not pure memorization, you can deliberately give them tasks that have never been done before and they do well.
However, even with agentic workflows, rag, etc all existing models seem to fail at some moderate level of complexity—they can create functions and prototypes but have trouble keeping track of a large project
My uninformed guess is that o3 actually pushes the complexity by some non-trivial amount, but not enough to now take on complex projects.
Do you like transcripts? We got one of those at the link as well. It’s an mid AI-generated transcript, but the alternative is none. :)
At least when the link opens the substack app on my phone, I see no such transcript.
Is this true?
I’m still a bit confused about this point of the Kelly criterion. I thought that actually this is the way to maximize expected returns if you value money linearly, and the log term comes from compounding gains.
That the log utility assumption is actually a separate justification for the Kelly criterion that doesn’t take into account expected compounding returns
I was figuring that the SWE-bench tasks don’t seem particularly hard, intuitively. E.g. 90% of SWE-bench verified problems are “estimated to take less than an hour for an experienced software engineer to complete”.
I mean, fair but when did a benchmark designed to test REAL software engineering issues that take less than an hour suddenly stop seeming “particularly hard” for a computer.
Feels like we’re being frogboiled.
I don’t think you can explain away SWE-bench performance with any of these explanations
We haven’t yet seen what happens when they turn to the verifiable property of o3 to self-play on a variety of strategy games. I suspect that it will unlock a lot of general reasoning and strategy
can you say the types of problems they are?
can you say more about your reasoning for this?
Excellent work! Thanks for what you do
fwiw while it’s fair to call this “heavy nudging”, this mirrors exactly what my prompts for agentic workflows look like. I have to repeat things like “Don’t DO ANYTHING YOU WEREN’T ASKED” multiple times to get them to work consistently.
I found this post to be incredibly useful to get a deeper sense of Logan’s work on naturalism.
I think his work on Naturalism is a great and unusual example of original research happening in the rationality community and what actually investigating rationality looks like.
Emailed you.
In my role as Head of Operations at Monastic Academy, every person in the organization is on a personal improvement plan that addresses the personal responsibility level, and each team in the organization is responsible for process improvements that address the systemic level.
In the performance improvement weekly meetings, my goal is to constantly bring them back to the level of personal responsibility. Any time they start saying the reason they couldn’t meet their improvement goal was because of X event or Y person, I bring it back. What could THEY have done differently, what internal psychological patterns prevented them from doing that, and what can they do to shift those patterns this week.
Meanwhile, each team also chooses process improvements weekly. In those meetings, my role is to do the exact opposite, and bring it back to the level of process. Any time they’re examining a team failure and come to the conclusion “we just need to prioritize it more, or try harder, or the manager needs to hold us to something”, I bring it back to the level of process. How can we change the order or way we do things, or the incentives involved, such that it’s not dependent on any given person’s ability to work hard or remember or be good at a certain thing.
Personal responsibility and systemic failure are different levels of abstraction.
If you’re within the system and doing horrible things while saying, “🤷 It’s just my incentives, bro,” you’re essentially allowing the egregore to control you, letting it shove its hand up your ass and pilot you like a puppet.
At the same time, if you ignore systemic problems, you’re giving the egregore power by pretending it doesn’t exist—even though it’s puppeting everyone. By doing so, you’re failing to claim your own power, which lies in recognizing your ability to work towards systemic change.
Both truths coexist:
There are those perpetuating evil by surrendering their personal responsibility to an evil egregore.
There are those perpetuating evil by letting the egregore run rampant and denying its existence.
The solution requires addressing both levels of abstraction.
I remember reading this and getting quite excited about the possibilities of using activation steering and downstream techniques. The post is well written with clear examples.
I think that this directly or indirectly influenced a lot of later work in steering llms.