The question of context might be important, see here. I wouldn’t find 15 minutes that surprising for ~50% success rate, but I’ve seen numbers more like 1.5 hours. I thought this was likely to be an overestimate so I went down to 1 hour, but more like 15-30 minutes is also plausible.
Keep in mind that I’m talking about agent scaffolds here.
Keep in mind that I’m talking about agent scaffolds here.
Yeah, I have failed to get any value out of agent scaffolds, and I don’t think I know anyone else who has so far. If anyone has gotten more value out of them than just the Cursor chat, I would love to see how they do it!
All things like Cursor composer and codebuff and other scaffolds have been worse than useless for me (though I haven’t tried it again after o3-mini, which maybe made a difference, it’s been on my to-do list to give it another try).
FYI I do find that aider using a mixed routing between r1 and o3-mini-high as the architect model with sonnet as the editor model to be slightly better than cursor/windsurf etc.
Or for minimal setup, this is what is ranking the highest on aider-polyglot test: aider --architect --model openrouter/deepseek/deepseek-r1 --editor-model sonnet
(I don’t expect o3-mini is a much better agent than 3.5 sonnet new out of the box, but probably a hybrid scaffold with o3 + 3.5 sonnet will be substantially better than 3.5 sonnet. Just o3 might also be very good. Putting aside cost, I think o1 is usually better than o3-mini on open ended programing agency tasks I think.)
The question of context might be important, see here. I wouldn’t find 15 minutes that surprising for ~50% success rate, but I’ve seen numbers more like 1.5 hours. I thought this was likely to be an overestimate so I went down to 1 hour, but more like 15-30 minutes is also plausible.
Keep in mind that I’m talking about agent scaffolds here.
Yeah, I have failed to get any value out of agent scaffolds, and I don’t think I know anyone else who has so far. If anyone has gotten more value out of them than just the Cursor chat, I would love to see how they do it!
All things like Cursor composer and codebuff and other scaffolds have been worse than useless for me (though I haven’t tried it again after o3-mini, which maybe made a difference, it’s been on my to-do list to give it another try).
FYI I do find that aider using a mixed routing between r1 and o3-mini-high as the architect model with sonnet as the editor model to be slightly better than cursor/windsurf etc.
Or for minimal setup, this is what is ranking the highest on aider-polyglot test:
aider --architect --model openrouter/deepseek/deepseek-r1 --editor-model sonnet(I don’t expect o3-mini is a much better agent than 3.5 sonnet new out of the box, but probably a hybrid scaffold with o3 + 3.5 sonnet will be substantially better than 3.5 sonnet. Just o3 might also be very good. Putting aside cost, I think o1 is usually better than o3-mini on open ended programing agency tasks I think.)