Have you felt this from your own experience trying to get funding, or from others, or both? Also, I’m curious what you think is their specific kind of bullshit, and if there’s things you think are real but others thought to be bullshit.
evalu
Karma: 26
I disagree because to me this just looks like LLMs are one algorithmic improvement away from having executive function, similar to how they couldn’t do system 2 style reasoning until this year when RL on math problems started working.
For example, being unable to change its goals on the fly: If a kid kept trying to go forward when his pokemon were too weak. He would keep losing, get upset, and hopefully in a moment of mental clarity, learn the general principle that he should step back and reconsider his goals every so often. I think most children learn some form of this from playing around as a toddler, and reconsidering goals is still something we improve at as adults.
Unlike us, I don’t think Claude has training data for executive functions like these, but I wouldn’t be surprised if some smart ML researchers solved this in a year.
evalu’s Shortform
There’s a lot of discussion about evolution as an example of inner and outer alignment.
However, we could instead view the universe as the outer optimizer that maximizes entropy, or power, or intelligence. From this view, both evolution and humans are inner optimizers, and the difference between evolution’s and our optimization targets is more of an alignment success than a failure.
Before evolution, the universe increased entropy by having rocks in space crash into each other. When life and evolution finally came around, it was way more effective than rock collisions at increasing entropy even though entropy isn’t in its optimization target. If there were an SGD loop around the universe to maximize entropy, it would choose the evolution mesa-optimizer instead of the crashing rocks.
Compared to evolution, humans optimizing to win wars, make money, and be famous was once again way better at increasing entropy. Entropy is still not in the optimization target, but an entropy-maximizing SGD around the universe would choose the human mesa-optimizer over evolution.
Importantly, humans not caring about genetic fitness is no longer an alignment failure from this view. The mesa-optimizer for our values is more aligned than evolution was, so it’s good that we rather spread our ideas and influence than our genes.
I’ve had caps lock remapped to escape for a few years now, and I also remapped a bunch of symbol keys like parentheses to be easier to type when coding. On other people’s computers it is slower for me type text with symbols or use vim, but I don’t mind since all of my deeply focused work (when the mini-distraction of reaching for a difficult key is most costly) happens on my own computers.
I’m skeptical of the claim that the only things that matter are the ones that have to be done before AGI.
Ways it could be true:
The rate of productivity growth has a massive step increase after AI can improve its capabilities without the overhead of collaborating with humans. Generally the faster the rate of productivity growth, the less valuable it is to do long-horizon work. For example, people shouldn’t work on climate change because AGI will instantly invent better renewables.
If we expect short timelines and also smooth takeoff, then that might mean our current rate of productivity growth is much higher or a different shape (e.g. doubly exponential instead of just exponential) than it was a few years ago. This much higher rate of productivity growth means any work with 3+ year horizons has negligible value.
Ways it could be false:
Moloch still rules the world after AGI (maybe there are multiple competing AGIs). For example, a scheme that allows an aligned AGI to propagate it’s alignment to the next generation would be valuable to work on today because it might be difficult for our first generation aligned AGI to invent this before someone else (another AGI) creates the second generation, smarter AGI.
DALYs saved today are still valuable.
Q: Why save lives now when it will be so much cheaper after we build aligned AGI?
A: Why do computer scientists learn to write fast algorithms when they could just wait for compute speed to double?Basic research might always be valuable because it’s often not possible to see the applications of a research field until it’s quite mature. A post-AGI world might dedicate some constant fraction of resources towards basic research with no obvious applications, and in that world it’s still valuable to pull ahead the curve of basic research accomplishments.
I lean towards disagreeing because I give credence to smooth takeoffs, mundane rates of productivity growth, and many-AGI worlds. I’m curious if those are the big cruxes or if my model could be improved.
Do you mean 99% of circuits that don’t satisfy P? Because there probably are distributions of random reversible circuits that satisfy P exactly 1% of the time, and that would make V’s job as hard as NP = coNP.