Their business is death by thermonuclear fire, and they historically did their jobs pretty well…
While that might have been true in the beginning, currently they list their research areas as Applied Energy Programs, Civilian Nuclear Program, and Office of Science.
Their Office of Science does basic research like developing a Quantum Light Source.
But it’s not ASML, it’s not Bell labs, it’s not Deepmind, it’s not CATL… The magnitude of “progress” they contribute to is probably at least 1000 times smaller than the 4 I mentioned.**
They don’t seem to be as productive as the others, but in this context, the interesting question is why they aren’t. The researchers at the Los Alamos National Laboratory (and other similar DOE laboratories) don’t have to manage a bunch of grad students but they still seem to produce less research output.
While not being certain, I would expect that in the beginning the researchers at the Los Alamos National Laboratory where mainly funded out of the budget of Laboratory.
They don’t seem to be as productive as the others, but in this context, the interesting question is why they aren’t. The researchers at the Los Alamos National Laboratory (and other similar DOE laboratories) don’t have to manage a bunch of grad students but they still seem to produce less research output.
I picked the below because I think they are large organizations, doing research that has tangible benefits to humans at a wide scale.
ASML: they aren’t a general lab but are doing applied research to advance chip fabrication tech. They don’t get paid if their tools don’t work or are not a major advance. There are competitors who will take their business if they fall too far behind too long.
Bell Labs: you know this one
Deepmind: Googles private management thought they were being insufficiently productive, and fired 40 percent of the staff in October 2022, 1 month before chatGPT. Since then they obviously are now central to the core survival strategy for Google and presumably have many more resources, and are under pressure to deliver models that are competitive. This competition and large scale effort makes them similar to the other successes.
CATL: a massive company that just does fairly narrow scope research to fine tune the production process for battery cells, and has developed a sodium battery. Under heavy competition like the others. I mentioned this one because their research isn’t a graphene, it has immediate real value for humans, the cost of battery storage affects end users at a large scale, and obviously once it is cheap enough most homes will have a storage battery between the meter and electrical panel with solar input, and most cars and trucks will primarily use batteries.
Each success case is massive, with a lot of equipment, and has a clear goal they are optimizing for, where the organization itself supports them.
This is what I have noticed with the academic research I have seen : basically just a lack of funds and inefficient processes designed primarily to protect the funds means people can’t get prompt access to tools and materials. Each research avenue doesn’t have many people working on it, for example the large well known lab I personally saw is a bunch of separate labs that are mostly not collaborating.
Vs say a skunkworks or a large private effort that is aligned with the survival of the company. There is specialized equipment and specialized skilled staff available in massive quantities in special dedicated labs. I saw this at 2 chip company employers. Competitive pressure means that the goal is to achieve results today...or tonight...
I wrote all the above to say Los Alamos likely doesn’t have the organizational structure to contribute efficiently to private research, and it isn’t focused on a single area, which seems to be a common element above. Each success above has many billions of dollars and it’s all going to one specialized domain.
Googles private management thought they were being insufficiently productive, and fired 40 percent of the staff in October 2022
No, they cut staff costs by 40%. Not the same thing at all. (You would have noticed a lot more ex-DMers if they had fired half the place!)
They shut down the Edmonton office with Sutton*, so they clearly shed some people, but it’s not clear what percentage by headcount; because of how compensation works, a lot of that 40% could reflect, say, high stock grants and high share prices in the COVID tech bubble followed by slashing offered bonuses to return to baseline. The DM budget was unusually high for a while, I think, and interpreting the official public numbers is hard because of issues like the purchase of their extremely expensive London HQ.
Interpretation is also ambiguous because this was near-simultaneous with the merger with Google Brain; the general view is that GB was the one that lost out in the merger and was the one being dissolved due to insufficient productivity compared to DM. (And we do see a lot of ex-GBers now.)
* which is probably relevant to why Sutton is now partnering with Carmack’s Keen Technologies AGI startup.
Interpretation is also ambiguous because this was near-simultaneous with the merger with Google Brain; the general view is that GB was the one that lost out in the merger and was the one being dissolved due to insufficient productivity compared to DM. (And we do see a lot of ex-GBers now.)
Thanks for the details, to me the issue is that a large budget slash like this sounds pretty detrimental in EV. You could get this kind of savings during the Manhattan project if you decided to cut 2 of the 3 enrichment methods for example.
Sure we know in hindsight that all 3 methods worked, but the expected value of “bomb before the end of the war” drops a lot because everything is now riding on whichever method you kept.
I would assume now Deepmind is going to be focused on massive transformers and has much less to spare on any other routes.
This also, like you said, sends out many ‘B team’ members who still know almost everything the people not fired know, spreading the knowledge around to all the competition. (Imagine if the Manhattan project staff who were fired were able to join the Axis powers. They are bringing with them strategically relevant info, even if none of them are the most talented physicists)
It depends on what that ’40% staff cost’ means, really. Was it just accounting shenanigans related to RSUs and GOOG stock fluctuations? Then it means pretty much nothing of interest to us here at LW. Did it come from shedding a few superstars with multi-million-dollar compensation packages? Hard to say, depends on how much you think superstars matter at this point compared to researchers. Could be a very big deal: I remain convinced that search for LLMs may be the Next Big Thing and everyone who is reinventing RL from scratch for LLMs is botching the job, and so a few superstar researchers leaving DM could be critical. (But maybe you think the opposite because it’s now all about big pressgangs of researchers whipping a model into shape.) Did it come from shedding a lot of lower-level people who are obscure and unheard of? Inverse of the former.
If the cut is inflated by Edmonton people getting the axe, then I personally would consider this cut to be irrelevant: I have been largely unimpressed by their work, and I think Sutton’s ‘Edmonton plan’ or whatever he was calling it is not an interesting line of work compared to more mainstream RL scaling approaches. (In general, I think Sutton has completely missed the boat on deep learning & especially DL scaling. I realize the irony of saying this about the author of “The Bitter Lesson”, but if you look at his actual work, he’s committed to basically antiquated model-free tweaks and small models, rather than the future of large-scale model-based DRL—like all of his stuff on continual learning is a waste of time, when scaling just plain solves that!)
While that might have been true in the beginning, currently they list their research areas as Applied Energy Programs, Civilian Nuclear Program, and Office of Science.
Their Office of Science does basic research like developing a Quantum Light Source.
They don’t seem to be as productive as the others, but in this context, the interesting question is why they aren’t. The researchers at the Los Alamos National Laboratory (and other similar DOE laboratories) don’t have to manage a bunch of grad students but they still seem to produce less research output.
While not being certain, I would expect that in the beginning the researchers at the Los Alamos National Laboratory where mainly funded out of the budget of Laboratory.
I picked the below because I think they are large organizations, doing research that has tangible benefits to humans at a wide scale.
ASML: they aren’t a general lab but are doing applied research to advance chip fabrication tech. They don’t get paid if their tools don’t work or are not a major advance. There are competitors who will take their business if they fall too far behind too long.
Bell Labs: you know this one
Deepmind: Googles private management thought they were being insufficiently productive, and fired 40 percent of the staff in October 2022, 1 month before chatGPT. Since then they obviously are now central to the core survival strategy for Google and presumably have many more resources, and are under pressure to deliver models that are competitive. This competition and large scale effort makes them similar to the other successes.
CATL: a massive company that just does fairly narrow scope research to fine tune the production process for battery cells, and has developed a sodium battery. Under heavy competition like the others. I mentioned this one because their research isn’t a graphene, it has immediate real value for humans, the cost of battery storage affects end users at a large scale, and obviously once it is cheap enough most homes will have a storage battery between the meter and electrical panel with solar input, and most cars and trucks will primarily use batteries.
Each success case is massive, with a lot of equipment, and has a clear goal they are optimizing for, where the organization itself supports them.
This is what I have noticed with the academic research I have seen : basically just a lack of funds and inefficient processes designed primarily to protect the funds means people can’t get prompt access to tools and materials. Each research avenue doesn’t have many people working on it, for example the large well known lab I personally saw is a bunch of separate labs that are mostly not collaborating.
Vs say a skunkworks or a large private effort that is aligned with the survival of the company. There is specialized equipment and specialized skilled staff available in massive quantities in special dedicated labs. I saw this at 2 chip company employers. Competitive pressure means that the goal is to achieve results today...or tonight...
I wrote all the above to say Los Alamos likely doesn’t have the organizational structure to contribute efficiently to private research, and it isn’t focused on a single area, which seems to be a common element above. Each success above has many billions of dollars and it’s all going to one specialized domain.
No, they cut staff costs by 40%. Not the same thing at all. (You would have noticed a lot more ex-DMers if they had fired half the place!)
They shut down the Edmonton office with Sutton*, so they clearly shed some people, but it’s not clear what percentage by headcount; because of how compensation works, a lot of that 40% could reflect, say, high stock grants and high share prices in the COVID tech bubble followed by slashing offered bonuses to return to baseline. The DM budget was unusually high for a while, I think, and interpreting the official public numbers is hard because of issues like the purchase of their extremely expensive London HQ.
Interpretation is also ambiguous because this was near-simultaneous with the merger with Google Brain; the general view is that GB was the one that lost out in the merger and was the one being dissolved due to insufficient productivity compared to DM. (And we do see a lot of ex-GBers now.)
* which is probably relevant to why Sutton is now partnering with Carmack’s Keen Technologies AGI startup.
Thanks for the details, to me the issue is that a large budget slash like this sounds pretty detrimental in EV. You could get this kind of savings during the Manhattan project if you decided to cut 2 of the 3 enrichment methods for example.
Sure we know in hindsight that all 3 methods worked, but the expected value of “bomb before the end of the war” drops a lot because everything is now riding on whichever method you kept.
I would assume now Deepmind is going to be focused on massive transformers and has much less to spare on any other routes.
This also, like you said, sends out many ‘B team’ members who still know almost everything the people not fired know, spreading the knowledge around to all the competition. (Imagine if the Manhattan project staff who were fired were able to join the Axis powers. They are bringing with them strategically relevant info, even if none of them are the most talented physicists)
It depends on what that ’40% staff cost’ means, really. Was it just accounting shenanigans related to RSUs and GOOG stock fluctuations? Then it means pretty much nothing of interest to us here at LW. Did it come from shedding a few superstars with multi-million-dollar compensation packages? Hard to say, depends on how much you think superstars matter at this point compared to researchers. Could be a very big deal: I remain convinced that search for LLMs may be the Next Big Thing and everyone who is reinventing RL from scratch for LLMs is botching the job, and so a few superstar researchers leaving DM could be critical. (But maybe you think the opposite because it’s now all about big pressgangs of researchers whipping a model into shape.) Did it come from shedding a lot of lower-level people who are obscure and unheard of? Inverse of the former.
If the cut is inflated by Edmonton people getting the axe, then I personally would consider this cut to be irrelevant: I have been largely unimpressed by their work, and I think Sutton’s ‘Edmonton plan’ or whatever he was calling it is not an interesting line of work compared to more mainstream RL scaling approaches. (In general, I think Sutton has completely missed the boat on deep learning & especially DL scaling. I realize the irony of saying this about the author of “The Bitter Lesson”, but if you look at his actual work, he’s committed to basically antiquated model-free tweaks and small models, rather than the future of large-scale model-based DRL—like all of his stuff on continual learning is a waste of time, when scaling just plain solves that!)