For what it’s worth, the last big MIRI output I remember is Arbital, which unfortunately didn’t get a lot of attention. But since then? Publishing lightly edited conversations doesn’t seem like a substantial research output to me.
(Scott Garrabrant has done a lot of impressive foundational formal research, but it seems to me of little applicability to alignment, since it doesn’t operate in the usual machine learning paradigm. It reminds a bit of research in formal logic in the last century: People expected it to be highly relevant for AI, yet it turned out to be completely irrelevant. Not even “Bayesian” approaches to AI did go anywhere. My hopes for other foundational formal research today are similarly low, except for formal work which roughly fits into the ML paradigm, like statistical learning theory.)
I actually think his recent work on geometric rationality will be very relevant for thinking about advanced shard theories. Shards are selected using winner-take-all dynamics. Also, in worlds where ML alone does not in fact get you all the way to AGI, his work will become far more relevant than the alignment work you feel bullish on.
formal logic is not at all irrelevant for AI. the problem with it is that it only works once you’ve got low enough uncertainty weights to use it on. Once you do, it’s an incredible boost to a model. And deep learning folks have known this for a while.
For what it’s worth, the last big MIRI output I remember is Arbital, which unfortunately didn’t get a lot of attention. But since then? Publishing lightly edited conversations doesn’t seem like a substantial research output to me.
(Scott Garrabrant has done a lot of impressive foundational formal research, but it seems to me of little applicability to alignment, since it doesn’t operate in the usual machine learning paradigm. It reminds a bit of research in formal logic in the last century: People expected it to be highly relevant for AI, yet it turned out to be completely irrelevant. Not even “Bayesian” approaches to AI did go anywhere. My hopes for other foundational formal research today are similarly low, except for formal work which roughly fits into the ML paradigm, like statistical learning theory.)
I actually think his recent work on geometric rationality will be very relevant for thinking about advanced shard theories. Shards are selected using winner-take-all dynamics. Also, in worlds where ML alone does not in fact get you all the way to AGI, his work will become far more relevant than the alignment work you feel bullish on.
formal logic is not at all irrelevant for AI. the problem with it is that it only works once you’ve got low enough uncertainty weights to use it on. Once you do, it’s an incredible boost to a model. And deep learning folks have known this for a while.
Are there any modern models which use hardcoded rules written in formal logic?
sent dm.
It appears I didn’t get it?
Edit: Got it.