Dan Elton blog: https://moreisdifferent.substack.com/ website: http://www.moreisdifferent.com twitter: https://twitter.com/moreisdifferent
delton137
Boston ACX meetup with special guest Mingyuan + Harvard Art Museums at Night
Boston ACX Spring Schelling Point Meetup
Thanks for sharing! That’s a pretty sophisticated modeling function but it makes sense. I personally think Moore’s law (the FLOPS/$ version) will continue, but I know there’s a lot of skepticism about that.
Could you make another graph like Fig 4 but showing projected cost, using Moore’s law to estimate cost? The cost is going to be a lot, right?
Networks with loops are much harder to train.. that was one of the motivations for going to transformers instead of RNNs. But yeah, sure, I agree. My objection is more that posts like this are so high level I have trouble following the argument, if that makes sense. The argument seems roughly plausible but not making contact with any real object level stuff makes it a lot weaker, at least to me. The argument seems to rely on “emergence of self-awareness / discovery of malevolence/deception during SGD” being likely which is unjustified in my view. I’m not saying the argument is wrong, more that I personally don’t find it very convincing.
Has GPT-3 / large transformers actually led to anything with economic value? Not from what I can tell although anecdotal reports on Twitter are that many SWEs are finding Github Copilot extremely useful (it’s still in private beta though). I think transformers are going to start providing actual value soon, but the fact they haven’t so far despite almost two years of breathless hype is interesting to contemplate. I’ve learned to ignore hype, demos, cool cherry-picked sample outputs, and benchmark chasing and actually look at what is being deployed “in the real world” and bringing value to people. So many systems that looked amazing in academic papers have flopped when deployed—even from top firms—for instance Microsoft’s Tay and Google Health’s system for detecting diabetic retinopathy. Another example is Google’s Duplex. And for how long have we heard about burger flipping robots taking people’s jobs?
There are reasons to be skeptical about about a scaled up GPT leading to AGI. I touched on some of those points here. There’s also an argument that the hardware costs are going to balloon so quickly to make the entire project economically unfeasible, but I’m pretty skeptical about that.
I’m more worried about someone reverse engineering the wiring of cortical columns in the neocortex in the next few years and then replicating it in silicon.
Long story short, is existentially dangerous AI eminent? Not as far as we can see right now knowing what we know right now (we can’t see that far in the future, since it depends on discoveries and scientific knowledge we don’t have). Could that change quickly anytime? Yes. There is Knightian uncertainty here, I think (to use a concept that LessWrongers generally hate lol).
I’m interested!
Walk on esplanade followed by food/drinks in Kendall Square
This is a shot in the dark, but I recall there was a blog post that made basically the same point visually, I believe using Gaussian distributions. I think the number they argued you should aim for was 3-4 instead of 6. Anyone know what I’m talking about?
Hi, I just wanted to say thanks for the comment / feedback. Yeah, I probably should have separated out the analysis of Grokking from the analysis of emergent behaviour during scaling. They are potentially related—at least for many tasks it seems Grokking becomes more likely as the model gets bigger. I’m guilty of actually conflating the two phenomena in some of my thinking, admittedly.
Your point about “fragile metrics” being more likely to show Grokking great. I had a similar thought, too.
I think a bit too much mindshare is being spent on these sci-fi scenario discussions, although they are fun.
Honestly I have trouble following these arguments about deception evolving in RL. In particular I can’t quite wrap my head around how the agent ends up optimizing for something else (not a proxy objective, but a possibly totally orthogonal objective like “please my human masters so I can later do X”). In any case, it seems self awareness is required for the type of deception that you’re envisioning. Which brings up an interesting question—can a purely feed-forward network develop self-awareness during training? I don’t know about you, but I have trouble picturing it happening unless there is some sort of loop involved.
Zac says “Yes, over the course of training AlphaZero learns many concepts (and develops behaviours) which have clear correspondence with human concepts.”
What’s the evidence for this? If AlphaZero worked by learning concepts in a sort of step-wise manner, then we should expect jumps in performance when it comes to certain types of puzzles, right? I would guess that a beginning human would exhibit jumps from learning concepts like “control the center” or “castle early, not later”.. for instance the principle “control the center”, once followed, has implications on how to place knights etc which greatly effect win probability. Is the claim they found such jumps? (eyeing the results nothing really stands out in the plots).
Or is the claim that the NMF somehow proves that AlphaZero works off concepts? To me that seems suspicious as NMF is looking at weight matrices at a very crude level, it seems.
I ask this partially because I went to a meetup talk (not recorded sadly) where a researcher from MIT showed a go problem that alphaGo can’t solve but which even beginner go players can solve, which shows that alphaGo actually doesn’t understand things the same way as humans. Hopefully they will publish their work soon so I can show you.
Huh, that’s pretty cool, thanks for sharing.
This is pretty interesting. There is a lot to quibble about here, but overall I think the information about bees here is quite valuable for people thinking about where AI is at right now and trying to extrapolate forward.
A different approach, perhaps more illuminating would be to ask how much of a bee’s behavior could we plausibly emulate today by globing together a bunch of different ML algorithms into some sort of virtual bee cognitive architecture—if say we wanted to make a drone that behaved like a bee ala Black Mirror. Obviously that’s a much more complicated question, though.
I feel compelled to mention my friend Logan Thrasher Collins’ paper, The case for emulating insect brains using anatomical “wiring diagrams” equipped with biophysical models of neuronal activity. He thinks we may be able to emulate the fruit fly brain in about 20 years at near-full accuracy, and this estimate seems quite plausible.
There were a few sections I skipped, if I have time I’ll come back and do a more thorough reading and give some more comments.
The compute comparison seems pretty sketchy to me. A bee’s visual cortex can classify many different things, and the part responsible for doing the classification task in the few shot learning study is probably just a small subset. [I think below Rohin made a similar point below.] Deep learning models can be pruned somewhat without loosing much accuracy, but generally all the parameters are used. Another wrinkle is the rate of firing activity in the visual cortex depends on the input, although there is a baseline rate too. The point I’m getting at is it’s sort of an apples-to-oranges comparison. If the bee only had to do the one task in the study to survive, evolution probably would have found a much more economical way of doing it, with far fewer neurons.
My other big quibble I have is I would have made transparent that Cotra’s biological anchors method for forecasting TAI assumes that we will know the right algorithm before the hardware becomes available. That is a big questionable assumption and thus should be stated clearly. Arguably algorithmic advancement in AI at the level of core algorithms (not ML-ops / dev ops / GPU coding) is actually quite slow. In any case, it just seems very hard to predict algorithmic advancement. Plausibly a team at DeepMind might discover the key cortical learning algorithm underlying human intelligence tomorrow, but there’s other reasons to think it could take decades.
Another point is that when you optimize relentlessly for one thing, you have might have trouble exploring the space adequately (get stuck at local maxima). That’s why RL agents/algorithms often take random actions when they are training (they call this “exploration” instead of “exploitation”). Maybe random actions can be thought of as a form of slack? Micro-slacks?
Look at Kenneth Stanley’s arguments about why objective functions are bad (video talk on it here). Basically he’s saying we need a lot more random exploration. Humans are similar—we have an open-ended drive to explore in addition to drives to optimize a utility function. Of course maybe you can argue the open-ended drive to explore is ultimately in the service of utility optimization, but you can argue the same about slack, too.
Bostrom talks about this in his book “Superintelligence” when he discusses the dangers of Oracle AI. It’s a valid concern, we’re just a long way from that with GPT-like models, I think.
I used to think a system trained on text only could never learn vision. So if it escaped onto the internet, it would be pretty limited in how it could interface with the outside world since it couldn’t interpret streams from cameras. But then I realized that probably in it’s training data is text on how to program a CNN. So in theory a system trained on only text could build a CNN algorithm inside itself and use that to learn how to interpret vision streams. Theoretically. A lot of stuff is theoretically possible with future AI, but how easy it is to realize in practice is a different story.
I just did some tests… it works if you go to settings and click “Activate Markdown Editor”. Then convert to Markdown and re-save (note, you may want to back up before this, there’s a chance footnotes and stuff could get messed up).
$stuff$ for inline math and double dollar signs for single line math work when in Markdown mode. When using the normal editor, inline math doesn’t work, but $$ works (but puts the equation on a new line).
I have mixed feelings on this. I have mentored ~5 undergraduates in the past 4 years and observed many others, and their research productivity varies enormously. How much of that is due to IQ vs other factors I really have no idea. My personal feeling was most of the variability was due to life factors like the social environment (family/friends) they were ensconced in and how much time that permitted them to focus on research.
My impression from TAing physics for life scientists for two years was that a large number felt they were intrinsically bad at math. That’s really bad! We need to be spreading more growth mindset ideas, not the idea that you’re limited by your IQ. Or at the very least, the idea that math doesn’t have to come naturally or be easy for you to be a scientist or engineer. I struggled with math my entire way through undergrad and my PhD. If the drive I developed as a child to become a scientist wasn’t so strong, I’m sure I would have dropped out.
My feeling is we are more bottlenecked on great engineers than sciences. [Also, the linear model (science → invention → engineering/innovation) is wrong!] Also, we should bring back inventors—that should be a thing again.
I think it would be awesome if some day 50% of people were engineers and inventors. People with middling IQ can still contribute a lot! Maybe not to theoretical physics, but to many other areas! We hear a lot of gushing things about scientific geniuses, especially on this site and I think we discount the importance of everyday engineers and also people like lab techs and support staff, which are increasingly important as science becomes more multidisciplinary and collaborative.
I liked how in your AISS support talk you used history as a frame for thinking about this because it highlights the difficulty of achieving superhuman ethics. Human ethics (for instance as encoded in laws/rights/norms) is improving over time, but it’s been a very slow process that involves a lot of stumbling around and having to run experiments to figure out what works and what doesn’t. “The Moral Arc” by Michael Shermer is about the causes of moral progress… one of them is allowing free speech, free flow of ideas. Basically, it seems moral progress requires a culture that supports conjecture and criticism of many ideas—that way you are more likely to find the good ideas. How you get an AI to generate new ideas is anyone’s guess—“creativity” in AI is pretty shallow right now—I am not aware of any AI having invented anything useful. (There have been news reports about AI systems that have found new drugs, but the ones I’ve seen were actually later called out as just slight modifications of existing drugs that were in their training data and thus they were not super creative).
To be honest I only read sections I-III of this post.
I have a comment on this:An even more speculative thing to try would be auto-supervision. A language model can not only be asked to generate text about ethical dilemmas, it can also be asked to generate text about how good different responses to ethical dilemmas are, and the valence of the response can be used as a reinforcement signal on the object-level decision.
This is a nice idea. It’s easy to implement and my guess is it should improve consistency. I actually saw something similar done in computer vision—someone took the labels generated by a CNN on a previously unlabeled dataset and then used those to fine-tune the CNN. Surprisingly, the result was a slightly better model. I think what that process does is encourage consistency across a larger swatch of data. I’m having trouble finding the paper right now however and I have no idea if the result replicated. If you would like I can try to find it—I think it was in the medical imaging domain where data labeled with ground truth labels is scarce, so if you can train on autogenerated (“weak”) labels th
en that is super useful.
It’s a mixed bag. A lot of near term work is scientific, in that theories are proposed and experiments run to test them, but from what I can tell that work is also incredibly myopic and specific to the details of present day algorithms and whether any of it will generalize to systems further down the road is exceedingly unclear.
The early writings of Bostom and Yudkowsky I would classify as a mix of scientifically informed futurology and philosophy. As with science fiction, they are laying out what might happen. There is no science of psychohistory and while there are better and worse ways of forecasting the future (see “Superforecasting”) when it comes to forecasting how future technology will play out it’s especially impossible because future technology depends on knowledge we by definition don’t have right now. Still, the work has value even if it is not scientific, by alerting us to what might happen. It is scientifically informed because at the very least the futures they describe don’t violate any laws of physics. That sort of futurology work I think is very valubale because it explores the landscape of possible futures so we can identify the futures we don’t want so we we can takes steps to avoid those futures, even if the probability of any given future scenario is not clear.
A lot of the other work is pre-paradigmatic, as others have mentioned, but that doesn’t make it pseudoscience. Falsifiability is the key to demarcation. The work that borders on pseudoscience revolves heavily around the construction of what I call “free floating” systems. These are theoretical systems that are not tied into existing scientific theory (examples: laws of physics, theory of evolution, theories of cognition, etc) and also not grounded in enough detail that we can test whether the ideas / theories are useful/correct right now. They aren’t easily falsifiable. These free-floating sets of ideas tend to be hard for outsiders to learn since they involve a lot of specialized jargon and because sorting wheat from chaffe is hard because they don’t bother to subject their work to the rigors of peer review and publication in conferences / journals, which provide valuable signals to outsiders as to what is good or bad (instead we end up with a huge lists of Alignment Forum posts and other blog posts and PDFs with no easy way of figuring out what is worth reading). Some of this type of work blends into abstract mathematics. Safety frameworks like iterated distillation & debate, iterated amplification, and a lot of the MIRI work on self-modifying agents seem pretty free-floating to me (some of these ideas may be testable in some sort of absurdly simple toy environment today, but what these toy models tell us about more general scenarios is hard to say without a more general theory). A lot of the futurology stuff is also free floating (a hallmark of free floating stuff is zany large concept maps like here). These free floating things are not worthless but they also aren’t scientific.
Finally, there’s much that is philosophy. First, of course, there’s debates about ethics. Secondly there’s debates about how to define basic terms that are heavily used like intelligence, general vs narrow intelligence, information, explanation, knowledge, and understanding.
Interesting, thanks. 10x reduction in cost every 4 years is roughly twice what I would have expected. But it sounds quite plausible especially considering AI accelerators and ASICs.