Aprillion
Interesting point of view about the occasional intense overlap between the 2 concepts, I could probably safely explore more of my agency over the world than what is my comfort zone 🤔
Nevertheless, I will keep to my expectation of a train delay originating from a departures table and not from my own intent.
Question from a conference discussion—do you have real-world-data examples in a shape like this illustration of how I understand Goodhart’s law?
E.g. when new LLM versions got better at a benchmark published before their training but not on a benchmark published later… (and if larger models generalize better to future benchmarks, does the typology here provide an insight why? does the bitter lesson of scale provide better structure, function, or randomness calibration? or if the causes are so unknown that this typology does not provide such an insight yet, what is missing? (because Goodhart law predicts that benchmark gaming should be happening to “some extent” ⇒ improvements in understanding of Goodhart law should preserve that quality, right?))
When the king is aligned with the kingdom, how would you distinguish the causal path that the king projected their power and their values onto the kingdom (that previously had different values or was a tabula rasa) and not that the kingdom had selected from a pool of potential kings?
After all, regicide was not that uncommon (both literally in the past and figuratively speaking when a mother company can dismiss a decision of a board of directors over who should be the CEO)...
(I’m not saying anything about Wizard power being more or less effective)
Interesting—the first part of the response seems to suggest that it looked like I was trying to understand more about LLMs… Sorry for confusion, I wanted to clarify an aspect of your worflow that was puzzling to me. I think I got all info for what I was asking about, thanks!
FWIW, if the question was an expression of actual interest and not a snarky suggestion, my experience with chatbots has been positive for brainstorming, dictionary “search”, rubber ducking, description of common sense (or even niche) topics, but disappointing for anything that requires application of commons sense. For programmming, one- or few-liner autocomplete is fine for me—then it’s me doing the judgement, half of the suggestions are completely useless, half are fine, and the third half look fine at first before I realise I needed the second most obvious thing this time.. but it can save time for the repeating part of almost-repeating stuff. For multi file editing,, I find it worse than useless when it feels like doing code review after a psychopath pretending to do programming (AFAICT all models can explain
everythingmost stuff correctly and then write the wrong code anyway .. I don’t find it useful when it tries to appologize later if I point it out or to pre-doubt itself in CoT in 7 paragraphs and then do it wrong anyway) - I like to imagine as if it was trained on all code from GH PRs—both before and after the bug fix… or as if it was bored, so it’s trying to insert drama into a novel about my stupid programming task, when the second chapter will be about heroic AGI firefighting the shit written by previous dumb LLMs...
it’s from https://gradual-disempowerment.ai/mitigating-the-risk … I’ve used
"just"(including scare quotes) for the concept of something being very hard, yet simpler to the thing in comparisonand now that concept has more color/flavour/it sparkled a glimmer of joy for me (despite/especially because it was used to illuminate such a dark and depressing scene—gradual disempowerment is like putting a dagger to one’s liver where the mere(!) misaligned ASI was a stab between the ribs, lose thy hope mere mortals, you were grabbing for water)
“merely(!)” is my new favourite word
I can see that if Moloch is a force of nature, any wannabe singleton would collapse under internal struggles… but it’s not like that would show me any lever AI safety can pull, it would be dumb luck if we live in a universe where the ratio of instrumentally convergent power concentration to it’s inevitable schism is less than 1 ¯\_(ツ)_/¯
Have you tried to make a mistake in your understanding on purpose to test out whether it would correct you or agree with you even when you’d get it wrong?
(and if yes, was it “a few times” or “statistically significant” kinda test, please?)
While Carl Brown said (a few times) he doesn’t want to do more youtube videos for every new disappointing AI release, so far he seems to be keeping tabs on them in the newsletter just fine—https://internetofbugs.beehiiv.com/
...I am quite confident that if anything actually started to work, he would comment on it, so even if he won’t say much about any future incremental improvements, it might be a good resource to subscribe to for getting better signal—if Carl will get enthusiastic about AI coding assistants, it will be worth paying attention.
My own experience is that if-statements are even 3.5′s Achilles heel and 3.7 is somehow worse (when it’s “almost” right, that’s worse than useless, it’s like reviewing pull requests when you don’t know if it’s an adversarial attack or if they mean well but are utterly incompetent in interesting, hypnotizing ways)… and that METR’s baselines more resemble a Skinner box than programming (though many people have that kind of job, I just don’t find the conditions of gig economy as “humane” and representative of what how “value” is actually created), and the sheer disconnect of what I would find “productive”, “useful projects”, “bottlenecks”, and “what I love about my job and what parts I’d be happy to automate” vs the completely different answers on How Much Are LLMs Actually Boosting Real-World Programmer Productivity?, even from people I know personally...
I find this graph indicative of how “value” is defined by the SF investment culture and disruptive economy… and I hope the AI investment bubble will collapse sooner rather than later...
But even if the bubble collapses, automating intelligence will not be undone, it won’t suddenly become “safe”, the incentives to create real AGI instead of overhyped LLMs will still exists—the danger is not in the presented economic curve going up, it’s in what economic actors see as potential, how incentivized are the corporations/governments to search for the thing that is both powerful and dangerous, no?
I would never trust people not to look at my scratchpad.
I suspect the corresponding analogy for humans might be about hostile telepaths, not just literal scratchpads, right?
thanks for concrete examples, can you help me understand how these translate from individual productivity to externally-observable productivity?
3 days to make a medium sized project
I agree Docker setup can be fiddly, however what happened with the 50+% savings—did you lower price for the customer to stay competitive, do you do 2x as many paid projects now, or did you postpone hiring another developer who is not needed now, or do you just have more free time? No change in support&maintenance costs compared to similar projects before LLMs?
processing isn’t more than ~500 lines of code
oh well, my only paid experience is with multi-year project development&maintenance, those are definitelly not in the category under 1kloc 🙈 which might help to explain my abysmal experience trying to use any AI tools for work (beyond autocomplete, but IntelliSense also existed before LLMs)
TBH, I am now moving towards the opinion that evals are very un-representative of the “real world” (if we exclude LLM wrappers as requested in the OP … though LLM wrappers including evals are becoming part of the “real world” too, so I don’t know—it’s like banking bootstrapped wealthy bankers, and LLM wrappers might be bootstraping wealthy LLM startups)
toxic slime, which releases a cloud of poison gas if anything touches it
this reminds me of Oxygen Not Included (though I just learned the original reference is D&D), where Slime (which also releases toxic stuff) can be harversted to produce useful stuff in Algae Distiller
the metaphor runs differently, one of the useful stuff from Slime is Polluted Water, which is also produced by
humansreplicants in Lavatory … and there is Water Sieve that will process Polluted Water into Water (and some plants want to be watered by the Polluted variant)makes me wonder if there is any back-applicable insight—if AI slop is indistinguishable from corporate slop, can we use it to generate data to train spam filters to improve quality of search results and start valuing quality journalism again soon? (and maybe some cyborgs want to use AI assistants for useful work beyond buggy clones of open source tools)
Talking out loud is even better. There is something about forcing your thoughts into language...
Those are 2 very different things for some people ;)
I, for one, can think MUCH faster without speaking out loud, even if subvocalize real words (for the purpose of revealing gaps) and don’t go all the way to manipulating concepts-that-don’t-have-words-yet-but-have-been-pointed-to-already or concepts-that-have-a-word-but-the-word-stands-for-5-concepts-and-we-already-narrowed-it-down-without-explicit-label …
the set of problems the solutions to which are present in their training data
a.k.a. the set of problems already solved by open source libraries without the need to re-invent similar code?
that’s not how productivity ought to be measured—it should measure some output per (say) a workday
1 vs 5 FTE is a difference in input, not output, so you can say “adding 5 people to this project will decrease productivity by 70% next month and we hope it will increase productivity by 2x in the long term” … not a synonym of “5x productivity” at all
it’s the measure by which you can quantify diminishig results, not obfuscate them!
...but the usage of “5-10x productivity” seems to point to a diffent concept than a ratio of useful output per input 🤷 AFAICT it’s a synonym with “I feel 5-10x better when I write code which I wouldn’t enjoy writing otherwise”
A thing I see around me, my mind.
Many a peak, a vast mountain range,
standing at a foothill,
most of it unseen.Two paths in front of me,
a lighthouse high above.Which one will it be,
a shortcut through the forest,
or a scenic route?Climbing up for better views,
retreating from overlooks,
away from the wolves.To think with all my lighthouses.
all the scaffold tools, system prompt, and what not add context for the LLM … but what if I want to know what’s the context too?
we can put higher utility on the
shutdownsounds instrumental to expand your moral circle to include other instances of yourself to keep creating copies of yourself that will shut down … then exand your moral circle to include humans and shut them down too 🤔
All confused human ontologies are equal, but some confused human ontologies are more equal than others.