I am very sorry to hear about your experiences. I hope you’ve found peace and that the organizations can take your experiences on board.
On one hand you seem to want there to be more open discussion around mental health, whilst on the other you are criticising MIRI and CFAR for having people have mental health issues in their orbit. These seem somewhat in tension with each other.
I think one of the factors is that the mission itself is stressful. For example, air traffic control and the police are high stress careers, yet we need both.
Another issue is that rationality is in some ways more welcoming of (at least some subset) people whom society would seem weird. Especially since certain conditions can be paired with great insight or drive. It seems like the less a community appreciates the silver-lining of mental health issues, the better they’d score according to your metric.
Regarding secrecy, I’d prefer for AI groups to lean too much on the side of maintaining precautions about info-hazards than too little. (I’m only referring to technical research not misbehaviour). I think it’s perfectly valid for donors to decide that they aren’t going to give money without transparency, but there should also be respect for people who are willing to trust/make the leap of faith.
It’s not the environment for everyone, but then again, the same could be said about the famously secretive Apple or say, working in national security. That seems like less of a organizational problem than one of personal fit (other alignment organizations and grant opportunities seem less secretive). One question: Would your position have changed it you felt that they had been more upfront about the stresses of the job?
Another point: I put the probability of us being screwed without Eliezer very high. This is completely different from putting him on a pedestal and pretending he’s perfect or that no-one can ever exceed his talents. Rather, my model is that without him it likely likely would have taken a number of more years for AI safety research to really kick off. And in this game even a few years can really matter. Now other people will have a more detailed model of the history than me and may come to different conclusions. But it’s not really a claim that’s very far out there. Of course, I don’t know who said it or the context so its hard for me to update on this.
Regarding people being a better philosopher than Kant or not, we have to take into account that we have access to much more information today. Basic scientific knowledge and understanding of computers resolves many of the disputes that philosophers used to argue over. Further, even if someone hasn’t read Kant directly, they may very well have read people who have read Kant. So I actually think that there would be a huge number of people floating around who would be better philosophers than Kant given these unfair advantages. Of course, it’s an entirely different question if we gave Kant access to today’s knowledge, but then again, given population growth, it wouldn’t surprise me if there were still a significant number of them.
One point I strongly agree with you on is that rationalists should pay more attention to philosophy. In fact, my most recent theory of logical counterfactuals was in part influenced by Kant—although I haven’t had time to read his critique of pure reason yet.
I am very sorry to hear about your experiences. I hope you’ve found peace and that the organizations can take your experiences on board.
Thanks, I appreciate the thought.
On one hand you seem to want there to be more open discussion around mental health, whilst on the other you are criticising MIRI and CFAR for having people have mental health issues in their orbit. These seem somewhat in tension with each other.
I don’t see why these would be in tension. If there is more and better discussion then that reduces the chance of bad outcomes. (Partially, I brought up the mental health issues because it seemed like people were criticizing Leverage for having people with mental health issues in their orbit, but it seems like Leverage handled the issue relatively well all things considered.)
I think one of the factors is that the mission itself is stressful. For example, air traffic control and the police are high stress careers, yet we need both.
I basically agree.
It seems like the less a community appreciates the silver-lining of mental health issues, the better they’d score according to your metric.
I don’t think so. I’m explicitly saying that talking about weird perceptions people might have, such as mental subprocess implantation, is better than the alternative; this is more likely to realize the benefits of neuro-atypicality, by allowing people to recognize when non-neurotypicals are having accurate perceptions, and reduce the risk of psychiatric hospitalization or other bad outcomes.
Regarding secrecy, I’d prefer for AI groups to lean too much on the side of maintaining precautions about info-hazards than too much.
I don’t think it makes sense to run a “biased” expected value calculation that over-weights costs relative to benefits of information sharing. There are significant negative consequences when discussion about large topics is suppressed, which include failure to have the kind of high-integrity conversations that could lead to actual solutions.
Would your position have changed it you felt that they had been more upfront about the stresses of the job?
I don’t think the issue was just that it was stressful, but that it was stressful in really unexpected ways. I think me from when I started work would be pretty surprised reading this post.
Rather, my model is that without him it likely likely would have taken a number of more years for AI safety research to really kick off. And in this game even a few years can really matter.
That doesn’t seem wrong to me. However there’s a big difference between saying it saves a few years vs. causes us to have a chance at all when we otherwise wouldn’t. (I can’t rule out the latter claim, but it seems like most of the relevant ideas were already in the memespace, so it’s more an issue of timing and consolidation/problem-framing.)
Regarding people being a better philosopher than Kant or not, we have to take into account that we have access to much more information today.
I think when people are comparing philosophers they’re usually trying to compare novel contributions the person made relative to what came before, not how much raw philosophical knowledge they possess.
One point I strongly agree with you on is that rationalists should pay more attention to philosophy.
Yes, I’ve definitely noticed a trend where rationalists are mostly continuing from Hume and Turing, neglecting e.g. Kant as a response to Hume.
One point I strongly agree with you on is that rationalists should pay more attention to philosophy.
Yes, I’ve definitely noticed a trend where rationalists are mostly continuing from Hume and Turing, neglecting e.g. Kant as a response to Hume.
I’ve yet to see a readable explanation of what Kant had to say (in response to Hume or otherwise) that’s particularly worth paying attention to (despite my philosophy classes in college having covered Kant, and making some attempts later to read him). If you (or someone else) were to write an LW post about this, I think this might be of great benefit to everyone here.
I don’t know what Kant-insights Jessica thinks LW is neglecting, but I endorse Allen Wood’s introduction to Kant as a general resource.
(Partly because Wood is a Kant scholar who loves Kant but talks a bunch about how Kant was just being sloppy / inconsistent in lots of his core discussions of noumena, rather than assuming that everything Kant says reflects some deep insight. This makes me less worried about IMO one of the big failure modes of philosopher-historians, which is that they get too creative with their novel interpretations + treat their favorite historical philosophers like truth oracles.)
BTW, when it comes to transcendental idealism, I mostly think of Arthur Schopenhauer as ‘Kant, but with less muddled thinking and not-absolutely-horrible writing style’. So I’d usually rather go ask what Schopenhauer thought of a thing, rather than what Kant thought. (But I mostly disagree with Kant and Schopenhauer, so I may be the wrong person to ask about how to properly steel-man Kant.)
I’ve been working on a write-up on and off for months, which I might or might not ever get around to finishing.
The basic gist is that, while Hume assumes you have sense-data and are learning structures like causation from this sense-data, Kant is saying you need concepts of causation to have sense-data at all.
The Transcendental Aesthetic is a pretty simple argument if applied to Solomonoff induction. Suppose you tried to write an AI to learn about time, which didn’t already have time. How would it structure its observations, so it could learn about time from these different observations? That seems pretty hard, perhaps not really possible, since “learning” implies past observations affecting how future observations are interpreted.
In Solomonoff induction there is a time-structure built in, which structures observations. That is, the inductor assumes a priori that its observations are structured in a sequence.
Kant argues that space is also a priori this way. This is a somewhat suspicious argument given that vanilla Solomonoff induction doesn’t need a priori space to structure its observations. But maybe it’s true in the case of humans, since our visual cortexes have a notion of spacial observation already built in. (That is, when we see things, we see them at particular locations)
Other than time and space to structuring observations, what else has to be there? To see the same object twice there has to be a notion that two observations could be of the same object. But that is more structure than simply spacetime, there’s also a structure of connection between different observations so they can be of the same object.
Solomonoff induction might learn this through compression. Kant, unfortunately, doesn’t explicitly discuss compression all that much. However, even Solomonoff induction makes a priori assumptions beyond spacetime, namely, that the universe is a Turing machine. This is a kind of causal assumption. You couldn’t get “this runs on a Turing machine” by just looking at a bunch of data, without having some kind of prior that already contains Turing machines. It is, instead, assumed a priori that there’s a Turing machine causing your observations.
The book is mostly a lot of stuff like this, what thought structures we must assume a priori to learn from data at all.
The basic gist is that, while Hume assumes you have sense-data and are learning structures like causation from this sense-data, Kant is saying you need concepts of causation to have sense-data at all.
Hmm. Both of these ideas seem very wrong (though Kant’s, perhaps, more so). Is there anything else of value? If this (and similar things) are all that there is, then maybe rationalists are right to mostly ignore Kant…
I’ve yet to see a readable explanation of what Kant had to say (in response to Hume or otherwise) that’s particularly worth paying attention to
As an undergrad, instead of following the actual instructions and writing a proper paper on Kant, I thought it would be more interesting and valuable to simply attempt to paraphrase what he actually said, paragraph by paragraph. It’s the work of a young person with little experience in either philosophy or writing, but it certainly seems to have had a pretty big influence on my thinking over the past ten years, and I got an A. So, mostly for your entertainment, I present to you “Kant in [really not nearly as plain as I thought at the time] English”. (It’s just the bit on apperception.)
I think this is either basic psychology or wrong.¹
For one, Kant seems to be conflating the operation of a concept with its perception:
Since the concept of “unity” must exist for there to be combination (or “conjunction”) in the first place, unity can’t come from combination itself. The whole-ness of unified things must be a product of something beyond combination.
This seems to say that the brain cannot unify things unless it has a concept of combination. However, just as an example, reinforcement learning in AI shows this to be false: unification can happen as a mechanistic consequence of the medium in which experiences are embedded, and an understanding of unification—a perception as a concept—is wholly unnecessary.
Then okay, concepts are generalizations (compressions?) of sense data, and there’s an implied world of which we become cognizant by assuming that the inner structure matches the outer structure. So far, so Simple Idea Of Truth. But then he does the same thing again with “unity”, where he assumes that persistent identity-perception is necessary for judgment. Which I think any consideration of a nematode would disprove: judgment can also happen mechanistically.
I mean, I don’t believe that the self is objectively unified, so Kant’s view would be a problem for me. But I also just think that the model where most mental capabilities are caused by aspects of the medium and nonreflective computation, and consciousness only reacts to them in “hindsight”, seems a lot more convincing to me in light of my pop understanding of neuroscience and introspection of my own mind.
edit: In summary, I think Kant’s model cannot separate, and thus repeatedly mixes up, cognition and consciousness.
¹ Okay, let me be fair: I think this is spectacularly correct and insightful for its time. But I don’t think people who have read the Sequences will get closer to the truth from it.
I got through that page and… no, I really can’t summarize it. I don’t really have any idea what Kant is supposed to have been saying, or why he said any of those things, or the significance of any of it…
I appreciate Kant’s idea that certain things may arise from how we see and interpret the world. I think it’s plausible that this is an accurate high-level description of things like counterfactuals and probability.
(I’m a bit busy atm so I haven’t provided much detail)
They’re suggesting that you should have written ”...this is an accurate how-level description of things like...” It’s a minor point but I guess I agree.
there’s a big difference between saying it saves a few years vs. causes us to have a chance at all when we otherwise wouldn’t. [...] it seems like most of the relevant ideas were already in the memespace
I was struck by the 4th edition of AI: A Modern Approach quoting Nobert Weiner writing in 1960 (!), “If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively … we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”
It must not have seemed like a pressing issue in 1960, but Weiner noticed the problem! (And Yudkowsky didn’t notice, at first.) How much better off are our analogues in the worlds where someone like Weiner (or, more ambitiously, Charles Babbage) did treat it as a pressing issue? How much measure do they have?
Yeah, it’s quite plausible that it might have taken another decade (then again I don’t know if Bostrom thought super-intelligence was possible before encountering Eliezer)
I don’t think so. I’m explicitly saying that talking about weird perceptions people might have, such as mental subprocess implantation, is better than the alternative; this is more likely to realize the benefits of neuro-atypicality, by allowing people to recognize when non-neurotypicals are having accurate perceptions, and reduce the risk of psychiatric hospitalization or other bad outcomes.
I guess my point was that a community that excludes anyone who has mental health issues would score well on your metric, while a community that is welcoming would score poorly.
I think when people are comparing philosophers they’re usually trying to compare novel contributions the person made relative to what came before, not how much raw philosophical knowledge they possess.
Another possiblity is that they might be comparing their ability to form a correct philosophical opinion. This isn’t the same as raw knowledge, but I suspect that our epistemic position makes it much easier. Not only because of more information, but also because modern philosopher tends to be much clearer and explicit than older philosophy and so people can use it as an example to learn how to think clearly.
Regarding secrecy, I’d prefer for AI groups to lean too much on the side of maintaining precautions about info-hazards than too much.
Was one of the much’s in this sentence supposed to be a ‘little’?
(My guess is that you meant to say that you want orgs to err on the side of being overly cautious rather than being overly reckless, but wanted to double-check.)
I am very sorry to hear about your experiences. I hope you’ve found peace and that the organizations can take your experiences on board.
On one hand you seem to want there to be more open discussion around mental health, whilst on the other you are criticising MIRI and CFAR for having people have mental health issues in their orbit. These seem somewhat in tension with each other.
I think one of the factors is that the mission itself is stressful. For example, air traffic control and the police are high stress careers, yet we need both.
Another issue is that rationality is in some ways more welcoming of (at least some subset) people whom society would seem weird. Especially since certain conditions can be paired with great insight or drive. It seems like the less a community appreciates the silver-lining of mental health issues, the better they’d score according to your metric.
Regarding secrecy, I’d prefer for AI groups to lean too much on the side of maintaining precautions about info-hazards than too little. (I’m only referring to technical research not misbehaviour). I think it’s perfectly valid for donors to decide that they aren’t going to give money without transparency, but there should also be respect for people who are willing to trust/make the leap of faith.
It’s not the environment for everyone, but then again, the same could be said about the famously secretive Apple or say, working in national security. That seems like less of a organizational problem than one of personal fit (other alignment organizations and grant opportunities seem less secretive). One question: Would your position have changed it you felt that they had been more upfront about the stresses of the job?
Another point: I put the probability of us being screwed without Eliezer very high. This is completely different from putting him on a pedestal and pretending he’s perfect or that no-one can ever exceed his talents. Rather, my model is that without him it likely likely would have taken a number of more years for AI safety research to really kick off. And in this game even a few years can really matter. Now other people will have a more detailed model of the history than me and may come to different conclusions. But it’s not really a claim that’s very far out there. Of course, I don’t know who said it or the context so its hard for me to update on this.
Regarding people being a better philosopher than Kant or not, we have to take into account that we have access to much more information today. Basic scientific knowledge and understanding of computers resolves many of the disputes that philosophers used to argue over. Further, even if someone hasn’t read Kant directly, they may very well have read people who have read Kant. So I actually think that there would be a huge number of people floating around who would be better philosophers than Kant given these unfair advantages. Of course, it’s an entirely different question if we gave Kant access to today’s knowledge, but then again, given population growth, it wouldn’t surprise me if there were still a significant number of them.
One point I strongly agree with you on is that rationalists should pay more attention to philosophy. In fact, my most recent theory of logical counterfactuals was in part influenced by Kant—although I haven’t had time to read his critique of pure reason yet.
Thanks, I appreciate the thought.
I don’t see why these would be in tension. If there is more and better discussion then that reduces the chance of bad outcomes. (Partially, I brought up the mental health issues because it seemed like people were criticizing Leverage for having people with mental health issues in their orbit, but it seems like Leverage handled the issue relatively well all things considered.)
I basically agree.
I don’t think so. I’m explicitly saying that talking about weird perceptions people might have, such as mental subprocess implantation, is better than the alternative; this is more likely to realize the benefits of neuro-atypicality, by allowing people to recognize when non-neurotypicals are having accurate perceptions, and reduce the risk of psychiatric hospitalization or other bad outcomes.
I don’t think it makes sense to run a “biased” expected value calculation that over-weights costs relative to benefits of information sharing. There are significant negative consequences when discussion about large topics is suppressed, which include failure to have the kind of high-integrity conversations that could lead to actual solutions.
I don’t think the issue was just that it was stressful, but that it was stressful in really unexpected ways. I think me from when I started work would be pretty surprised reading this post.
That doesn’t seem wrong to me. However there’s a big difference between saying it saves a few years vs. causes us to have a chance at all when we otherwise wouldn’t. (I can’t rule out the latter claim, but it seems like most of the relevant ideas were already in the memespace, so it’s more an issue of timing and consolidation/problem-framing.)
I think when people are comparing philosophers they’re usually trying to compare novel contributions the person made relative to what came before, not how much raw philosophical knowledge they possess.
Yes, I’ve definitely noticed a trend where rationalists are mostly continuing from Hume and Turing, neglecting e.g. Kant as a response to Hume.
I’ve yet to see a readable explanation of what Kant had to say (in response to Hume or otherwise) that’s particularly worth paying attention to (despite my philosophy classes in college having covered Kant, and making some attempts later to read him). If you (or someone else) were to write an LW post about this, I think this might be of great benefit to everyone here.
I don’t know what Kant-insights Jessica thinks LW is neglecting, but I endorse Allen Wood’s introduction to Kant as a general resource.
(Partly because Wood is a Kant scholar who loves Kant but talks a bunch about how Kant was just being sloppy / inconsistent in lots of his core discussions of noumena, rather than assuming that everything Kant says reflects some deep insight. This makes me less worried about IMO one of the big failure modes of philosopher-historians, which is that they get too creative with their novel interpretations + treat their favorite historical philosophers like truth oracles.)
BTW, when it comes to transcendental idealism, I mostly think of Arthur Schopenhauer as ‘Kant, but with less muddled thinking and not-absolutely-horrible writing style’. So I’d usually rather go ask what Schopenhauer thought of a thing, rather than what Kant thought. (But I mostly disagree with Kant and Schopenhauer, so I may be the wrong person to ask about how to properly steel-man Kant.)
I’ve been working on a write-up on and off for months, which I might or might not ever get around to finishing.
The basic gist is that, while Hume assumes you have sense-data and are learning structures like causation from this sense-data, Kant is saying you need concepts of causation to have sense-data at all.
The Transcendental Aesthetic is a pretty simple argument if applied to Solomonoff induction. Suppose you tried to write an AI to learn about time, which didn’t already have time. How would it structure its observations, so it could learn about time from these different observations? That seems pretty hard, perhaps not really possible, since “learning” implies past observations affecting how future observations are interpreted.
In Solomonoff induction there is a time-structure built in, which structures observations. That is, the inductor assumes a priori that its observations are structured in a sequence.
Kant argues that space is also a priori this way. This is a somewhat suspicious argument given that vanilla Solomonoff induction doesn’t need a priori space to structure its observations. But maybe it’s true in the case of humans, since our visual cortexes have a notion of spacial observation already built in. (That is, when we see things, we see them at particular locations)
Other than time and space to structuring observations, what else has to be there? To see the same object twice there has to be a notion that two observations could be of the same object. But that is more structure than simply spacetime, there’s also a structure of connection between different observations so they can be of the same object.
Solomonoff induction might learn this through compression. Kant, unfortunately, doesn’t explicitly discuss compression all that much. However, even Solomonoff induction makes a priori assumptions beyond spacetime, namely, that the universe is a Turing machine. This is a kind of causal assumption. You couldn’t get “this runs on a Turing machine” by just looking at a bunch of data, without having some kind of prior that already contains Turing machines. It is, instead, assumed a priori that there’s a Turing machine causing your observations.
The book is mostly a lot of stuff like this, what thought structures we must assume a priori to learn from data at all.
Hmm. Both of these ideas seem very wrong (though Kant’s, perhaps, more so). Is there anything else of value? If this (and similar things) are all that there is, then maybe rationalists are right to mostly ignore Kant…
As an undergrad, instead of following the actual instructions and writing a proper paper on Kant, I thought it would be more interesting and valuable to simply attempt to paraphrase what he actually said, paragraph by paragraph. It’s the work of a young person with little experience in either philosophy or writing, but it certainly seems to have had a pretty big influence on my thinking over the past ten years, and I got an A. So, mostly for your entertainment, I present to you “Kant in [really not nearly as plain as I thought at the time] English”. (It’s just the bit on apperception.)
I think this is either basic psychology or wrong.¹
For one, Kant seems to be conflating the operation of a concept with its perception:
This seems to say that the brain cannot unify things unless it has a concept of combination. However, just as an example, reinforcement learning in AI shows this to be false: unification can happen as a mechanistic consequence of the medium in which experiences are embedded, and an understanding of unification—a perception as a concept—is wholly unnecessary.
Then okay, concepts are generalizations (compressions?) of sense data, and there’s an implied world of which we become cognizant by assuming that the inner structure matches the outer structure. So far, so Simple Idea Of Truth. But then he does the same thing again with “unity”, where he assumes that persistent identity-perception is necessary for judgment. Which I think any consideration of a nematode would disprove: judgment can also happen mechanistically.
I mean, I don’t believe that the self is objectively unified, so Kant’s view would be a problem for me. But I also just think that the model where most mental capabilities are caused by aspects of the medium and nonreflective computation, and consciousness only reacts to them in “hindsight”, seems a lot more convincing to me in light of my pop understanding of neuroscience and introspection of my own mind.
edit: In summary, I think Kant’s model cannot separate, and thus repeatedly mixes up, cognition and consciousness.
¹ Okay, let me be fair: I think this is spectacularly correct and insightful for its time. But I don’t think people who have read the Sequences will get closer to the truth from it.
Interesting, thanks—I will read this ASAP!
If you manage to get through that, maybe you can summarize it? Even Logan’s accessible explanation makes my eyes glaze over.
I got through that page and… no, I really can’t summarize it. I don’t really have any idea what Kant is supposed to have been saying, or why he said any of those things, or the significance of any of it…
I’m afraid I remain as perplexed as ever.
I appreciate Kant’s idea that certain things may arise from how we see and interpret the world. I think it’s plausible that this is an accurate high-level description of things like counterfactuals and probability.
(I’m a bit busy atm so I haven’t provided much detail)
‘how-level’ would be easier to parse
Oops, I meant “high-level”
“How-level”?
They’re suggesting that you should have written ”...this is an accurate how-level description of things like...” It’s a minor point but I guess I agree.
I was struck by the 4th edition of AI: A Modern Approach quoting Nobert Weiner writing in 1960 (!), “If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively … we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”
It must not have seemed like a pressing issue in 1960, but Weiner noticed the problem! (And Yudkowsky didn’t notice, at first.) How much better off are our analogues in the worlds where someone like Weiner (or, more ambitiously, Charles Babbage) did treat it as a pressing issue? How much measure do they have?
Yeah, it’s quite plausible that it might have taken another decade (then again I don’t know if Bostrom thought super-intelligence was possible before encountering Eliezer)
I guess my point was that a community that excludes anyone who has mental health issues would score well on your metric, while a community that is welcoming would score poorly.
Another possiblity is that they might be comparing their ability to form a correct philosophical opinion. This isn’t the same as raw knowledge, but I suspect that our epistemic position makes it much easier. Not only because of more information, but also because modern philosopher tends to be much clearer and explicit than older philosophy and so people can use it as an example to learn how to think clearly.
Was one of the much’s in this sentence supposed to be a ‘little’?
(My guess is that you meant to say that you want orgs to err on the side of being overly cautious rather than being overly reckless, but wanted to double-check.)
I’d prefer too much rather than too little.