“6. f6” should be “6. h3″.
Simon Fischer
Karma: 471
Microsoft is the sort of corporate bureaucracy where dynamic orgs/founders/researchers go to die. My median expectation is that whatever former OpenAI group ends up there will be far less productive than they were at OpenAI.
I’m a bit sceptical of that. You gave some reasonable arguments, but all of this should be known to Sam Altman, and he still chose to accept Microsoft’s offer instead of founding his own org (I’m assuming he would easily able to raise a lot of money). So, given that “how productive are the former OpenAI folks at Microsoft?” is the crux of the argument, it seems that recent events are good news iff Sam Altman made a big mistake with that decision.
I’m confused by this statement. Are you assuming that AGI will definitely be built after the research time is over, using the most-plausible-sounding solution?
Or do you believe that you understand NOW that a wide variety of approaches to alignment, including most of those that can be thought of by a community of non-upgraded alignment researchers (CNUAR) in a hundred years, will kill everyone and that in a hundred years the CNUAR will not understand this?
If so, is this because you think you personally know better or do you predict the CNUAR will predictably update in the wrong direction? Would it matter if you got to choose the composition of the CNUAR?
Another big source of potential volunteers: People who are going to be dead soon anyway. I’d probably volunteer if I knew that I’m dying from cancer in a few weeks anyway.
Typo: This should be .
after 17… dxc6 or 17. c6
This should probably be “after 17… cxd6 or 17… c6”.
I suspect Wave refers to this company: https://www.wave.com/en/ (they are connected to EA)
Planecrash is a glowfic co-written by Yudkowsky: https://glowficwiki.noblejury.com/books/planecrash
Seconding the recommendation of the rest in motion post, it has helped me with a maybe-similar feeling.
AISC team report: Soft-optimization, Bayes and Goodhart
I don’t believe these “practical” problems (“can’t try long enough”) generalize enough to support your much more general initial statement. This doesn’t feel like a true rejection to me, but maybe I’m misunderstanding your point.
I think I mostly agree with this, but from my perspective it hints that you’re framing the problem slightly wrong. Roughly, the problem with the outsourcing-approaches is our inability to specify/verify solutions to the alignment problem, not that specifying is not in general easier than solving yourself.
(Because of the difficulty of specifying the alignment problem, I restricted myself to speculating about pivotal acts in the post linked above.)
But you don’t need to be able to code to recognize that a software is slow and buggy!?
About the terrible UI part I agree a bit more, but even there one can think of relatively objective measures to check usability without being able to speak python.
In cases where outsourcing succeeds (to various degrees), I think the primary load-bearing mechanism of success in practice is usually not “it is easier to be confident that work has been done correctly than to actually do the work”, at least for non-experts.
I find this statement very surprising. Isn’t almost all of software development like this?
E.g., the client asks the developer for a certain feature and then clicks around the UI to check if it’s implemented / works as expected.
“This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform.”
This proves way too much.I agree, I think this even proves P=NP.
Maybe a more reasonable statement would be: You can not outsource cognitive labor if you don’t know how to verify the solution. But I think that’s still not completely true, given that interactive proofs are a thing. (Plug: I wrote a post exploring the idea of applying interactive proofs to AI safety.)
No, that’s not quite right. What you are describing is the NP-Oracle.
On the other hand, with the IP-Oracle we can (in principle, limited by the power of the prover/AI) solve all problems in the PSPACE complexity class.
Of course, PSPACE is again a class of decision problems, but using binary search it’s straightforward to extract complete answers like the designs mentioned later in the article.
Pivotal acts using an unaligned AGI?
Your reasoning here relies on the assumption that the learning mostly takes place during the individual organisms lifetime. But I think it’s widely accepted that brains are not “blank slates” at birth of the organism, but contain significant amount of information, akin to a pre-trained neural network. Thus, if we consider evolution as the training process, we might reach the opposite conclusion: Data quantity and training compute are extremely high, while parameter count (~brain size) and brain compute is restricted and selected against.
Thank you for writing about this! A minor point: I don’t think aerosolizing monkeypox suspensions using a nebulizer can be counted as gain of function research, not even “at least kind of”. (Or do I lack reading comprehension and misunderstood something?)
Hypothesis: If a part of the computation that you want your trained system to compute “factorizes”, it might be easier to evolve a modular system for this computation. By factorization I just mean that (part of) the computation can be performed using mostly independent parts / modules.
Reasoning: Training independent parts to each perform some specific sub-calculation should be easier than training the whole system at once. E.g. training n neural networks of size N/n should be easier (in terms of compute or data needed) than training one of size N, given the exponential size of the parameter space.This hypothesis might explain the appearance of modularity if the necessary initial conditions for this selective advantage to be used are regularly present.
(I’ve talked about this idea with Lblack already but wanted to spell it out a bit more and post it here for reference.)
I guess it’s hard to keep “they are experimenting with / building huge amounts of tanks” and “they are conducting combined arms exercises” secret from France and Russia, so they would have a lot of advance warning and could then also develop tanks.
But if you have lot more than a layman’s understanding of tank design / combined arms doctrine, you could still come out ahead in this.