Sorry if I was unclear; my intended parsing was “accidentally (creating catastrophically-out-of-control AGIs)”. In other words, I don’t expect that people will try to create catastrophically-out-of-control AGIs. Therefore, if they create catastrophically-out-of-control AGIs, it would be by accident.
Emulating brains in order to increase capability is currently...an idea.
I think you’re overly confident that WBE would be irrelevant to the timeline of AGI capabilities research, but I think it’s a moot point anyway, since I don’t expect WBE before AGI, so I’m not really interested in arguing about it. :-P
Practically, progress requires doing both, i.e. better equipment to create and measure electricity is needed to understand it better, which helps understand how to direct, contain, and generate it better, etc.
I do in fact agree with you, but I think it’s not as clear-cut as you make it out to be in the WBE case, I think it takes a more detailed argument where reasonable people could disagree. In particular, there’s an argument on the other side that says “implementation-level understanding” is a different thing from “algorithm-level understanding”, and you only need the first one for WBE, not the second one.
So for example, if I give you a binary executable “factor.exe” that solves hard factorization problems, you would be able to run it on a computer much more easily than you could decompile it and understand how the algorithm works.
This example goes through because we have perfect implementation-level understanding about running executables on CPUs. In the brain case, Randal is arguing (and I agree) that we don’t have perfect implementation-level understanding, and we won’t get it by just studying the implementation level. The implementation-level is just very complicated—much more complicated than “dendrites are inputs, axons are outputs” etc. And it could involve subtle things that we won’t actually go measure and simulate unless we know that we need to go looking for them. So in practice, the only way to make up for our expected deficiencies in implementation-level understanding is to also have good algorithm-level understanding.
I think you’re overly confident that WBE would be irrelevant to the timeline of AGI capabilities research
Ah, I wrote this around the same time as another comment responding to something about ‘alignment work is a good idea even if the particular alignment method won’t work for a super intelligence’. (A positive utility argument is not a max utility argument.)
So, I wasn’t thinking about the timeline (and how relevant it would be) when I wrote that, just that it seems far out to me.
On reflection:
would be feasible to get WBE without incidentally first understanding brain algorithms well enough to code an AGI from scratch using similar algorithms.
I should have just responded to something like this (above).
I can see this being right (similar understanding required for both), although the idea that one must be easier than the other, I’m less sure of. Mostly in the sense that: I don’t know how small an AGI can be. Yes brains are big (and complicated), but I don’t know how much that can be avoided. So I think a working, understood, digital mind is a sufficiently large task that:
it seems far off (in terms of time and effort)
If a brain is harder, the added time and difficulty isn’t quite so big in comparison*
*an alternative would be that we’ll get an answer to this question before we get AGI:
As we start understanding minds, our view of the brain starts to recognize the difficulty. Like ‘we know something like X is required, but since evolution was proceeding in a very random/greedy way, in order to get something like X, a lot of unneeded complexity is added (because continuous improvement is a difficult constrain to fulfill, and relentless focus on improvement through that view is a path that does more ‘visiting every local maximum along the way’ than ‘goes for the global maximum’) and figuring this out will be a lot harder than figuring out (how to make) a digital** mind.
**I don’t know how much ‘custom/optimized hardware for architectures/etc.’ addresses the difficulties I mentioned. This might make your point about AGI before WBE a lot stronger—if the brain is an architecture optimized for minimizing power consumption in ways that make it way harder to emulate, timewise, than ‘AGI more optimized for ’computers″, then that could be a reason WBE would take longer.
I’d have thought that the main reason WBE would come up would be ‘understandability’ or ‘alignment’ rather than speed, though I can see why at first glance people would say ‘reverse engineering the brain (which exists) seems easier than making something new’ (even if that is wrong).
Strong agree, WBE seems far out to me too, like 100 years, although really who knows. By contrast “understanding the brain and building AGI using similar algorithms” does not seem far out to me—well, it won’t happen in 5 years, but I certainly wouldn’t rule it out in 20 or 30 years.
Yes brains are big (and complicated), but I don’t know how much that can be avoided.
I think the bigness and complicatedness of brains consists in large part of things that will not be necessary to include in the source code of a future AGI algorithm. See the first half of my post here, for example, for why I think that.
I’d have thought that the main reason WBE would come up would be ‘understandability’ or ‘alignment’ rather than speed, though I can see why at first glance people would say ‘reverse engineering the brain (which exists) seems easier than making something new’ (even if that is wrong).
There’s a normative question of whether it’s (A) good or (B) bad to have WBE before AGI, and there’s a forecasting question of whether WBE-before-AGI is (1) likely by default, vs (2) possible with advocacy/targeted funding/etc., vs (3) very unlikely even with advocacy/targeted funding/etc. For my part, I vote for (3), and therefore I’m not thinking very hard about whether (A) or (B) is right.
I’m not sure whether this part of your comment is referring to the normative question or the forecasting question.
I’m also not sure if when you say “reverse engineering the brain” you’re referring to WBE. For my part, I would say that “reverse-engineering the brain” = “understanding the brain at the algorithm level” = brain-inspired AGI, not WBE. I think the only way to get WBE before brain-inspired AGI is to not understand the brain at the algorithm level.
I’m not sure whether this part of your comment is referring to the normative question or the forecasting question.
Normative. Say that aligning something much smarter/more complicated than you (or your processing**) is difficult. The obvious fix would be: can we make people smarter?* If digital enhancement is easier (which seems like it could be likely, at least for crude ways—more computers more processing (though this may be more difficult than it sounds—serial has to be finished, parallel has to be communicated, etc.).)
This might help with analysis, or something like bandwidth. (Being able to process more information might make it easier to analyze the output of a process—if we wanted to evaluate something GPT like thing’s ability to generate poetry, then if it can generate ‘poetry’ faster than we can read or rate, then we’re the bottle neck.)
*Or algorithms simpler/easier to understand.
**Better ways of analyzing chess might help someone understand a (current) position in a chess game better (or just allow phones to compete with humans (though I don’t know how much of this is better algorithms, versus more powerful phones)).
Sorry if I was unclear; my intended parsing was “accidentally (creating catastrophically-out-of-control AGIs)”. In other words, I don’t expect that people will try to create catastrophically-out-of-control AGIs. Therefore, if they create catastrophically-out-of-control AGIs, it would be by accident.
I think you’re overly confident that WBE would be irrelevant to the timeline of AGI capabilities research, but I think it’s a moot point anyway, since I don’t expect WBE before AGI, so I’m not really interested in arguing about it. :-P
I do in fact agree with you, but I think it’s not as clear-cut as you make it out to be in the WBE case, I think it takes a more detailed argument where reasonable people could disagree. In particular, there’s an argument on the other side that says “implementation-level understanding” is a different thing from “algorithm-level understanding”, and you only need the first one for WBE, not the second one.
So for example, if I give you a binary executable “factor.exe” that solves hard factorization problems, you would be able to run it on a computer much more easily than you could decompile it and understand how the algorithm works.
This example goes through because we have perfect implementation-level understanding about running executables on CPUs. In the brain case, Randal is arguing (and I agree) that we don’t have perfect implementation-level understanding, and we won’t get it by just studying the implementation level. The implementation-level is just very complicated—much more complicated than “dendrites are inputs, axons are outputs” etc. And it could involve subtle things that we won’t actually go measure and simulate unless we know that we need to go looking for them. So in practice, the only way to make up for our expected deficiencies in implementation-level understanding is to also have good algorithm-level understanding.
Ah, I wrote this around the same time as another comment responding to something about ‘alignment work is a good idea even if the particular alignment method won’t work for a super intelligence’. (A positive utility argument is not a max utility argument.)
So, I wasn’t thinking about the timeline (and how relevant it would be) when I wrote that, just that it seems far out to me.
On reflection:
would be feasible to get WBE without incidentally first understanding brain algorithms well enough to code an AGI from scratch using similar algorithms.
I should have just responded to something like this (above).
I can see this being right (similar understanding required for both), although the idea that one must be easier than the other, I’m less sure of. Mostly in the sense that: I don’t know how small an AGI can be. Yes brains are big (and complicated), but I don’t know how much that can be avoided. So I think a working, understood, digital mind is a sufficiently large task that:
it seems far off (in terms of time and effort)
If a brain is harder, the added time and difficulty isn’t quite so big in comparison*
*an alternative would be that we’ll get an answer to this question before we get AGI:
As we start understanding minds, our view of the brain starts to recognize the difficulty. Like ‘we know something like X is required, but since evolution was proceeding in a very random/greedy way, in order to get something like X, a lot of unneeded complexity is added (because continuous improvement is a difficult constrain to fulfill, and relentless focus on improvement through that view is a path that does more ‘visiting every local maximum along the way’ than ‘goes for the global maximum’) and figuring this out will be a lot harder than figuring out (how to make) a digital** mind.
**I don’t know how much ‘custom/optimized hardware for architectures/etc.’ addresses the difficulties I mentioned. This might make your point about AGI before WBE a lot stronger—if the brain is an architecture optimized for minimizing power consumption in ways that make it way harder to emulate, timewise, than ‘AGI more optimized for ’computers″, then that could be a reason WBE would take longer.
I’d have thought that the main reason WBE would come up would be ‘understandability’ or ‘alignment’ rather than speed, though I can see why at first glance people would say ‘reverse engineering the brain (which exists) seems easier than making something new’ (even if that is wrong).
Thanks!
Strong agree, WBE seems far out to me too, like 100 years, although really who knows. By contrast “understanding the brain and building AGI using similar algorithms” does not seem far out to me—well, it won’t happen in 5 years, but I certainly wouldn’t rule it out in 20 or 30 years.
I think the bigness and complicatedness of brains consists in large part of things that will not be necessary to include in the source code of a future AGI algorithm. See the first half of my post here, for example, for why I think that.
There’s a normative question of whether it’s (A) good or (B) bad to have WBE before AGI, and there’s a forecasting question of whether WBE-before-AGI is (1) likely by default, vs (2) possible with advocacy/targeted funding/etc., vs (3) very unlikely even with advocacy/targeted funding/etc. For my part, I vote for (3), and therefore I’m not thinking very hard about whether (A) or (B) is right.
I’m not sure whether this part of your comment is referring to the normative question or the forecasting question.
I’m also not sure if when you say “reverse engineering the brain” you’re referring to WBE. For my part, I would say that “reverse-engineering the brain” = “understanding the brain at the algorithm level” = brain-inspired AGI, not WBE. I think the only way to get WBE before brain-inspired AGI is to not understand the brain at the algorithm level.
Normative. Say that aligning something much smarter/more complicated than you (or your processing**) is difficult. The obvious fix would be: can we make people smarter?* If digital enhancement is easier (which seems like it could be likely, at least for crude ways—more computers more processing (though this may be more difficult than it sounds—serial has to be finished, parallel has to be communicated, etc.).)
This might help with analysis, or something like bandwidth. (Being able to process more information might make it easier to analyze the output of a process—if we wanted to evaluate something GPT like thing’s ability to generate poetry, then if it can generate ‘poetry’ faster than we can read or rate, then we’re the bottle neck.)
*Or algorithms simpler/easier to understand.
**Better ways of analyzing chess might help someone understand a (current) position in a chess game better (or just allow phones to compete with humans (though I don’t know how much of this is better algorithms, versus more powerful phones)).