They were wanking on endlessly about quantum consciousness and how any agent sufficiently intelligent to escape its box would be intelligent enough to figure out that the utility function we’d given it wasn’t actually what we wanted.
I’ve been going on about ‘AI will kill us all’ to anyone who’ll listen for about ten years now. Everyone I know thinks I’ve got another stupid science-fiction bee in my bonnet to replace my previous idiotic nerd-concerns about overpopulation and dysgenics and grey goo and engineered plagues and factory farming and nuclear accidents and nuclear proliferation and free speech and ubiquitous surveillance and whatever the hell else it was that used to seem important.
Almost all my friends are graduates of the University of Cambridge, and roughly half of them studied real subjects and can program computers.
Goddamn it, I know actual AI safety researchers, people much cleverer and more capable than me, who just looked at AlphaZero and shrugged, apparently completely unable to see that building a really good general reinforcement learner might have real-world implications beyond chess.
There’s some bunch of academic parasites in Cambridge who were originally supposed to be worrying about existential risks and really seemed quite serious at first, but recently they mostly seem interested in global warming and the disparate impact of COVID.
Also, whaddya mean, 15 years ago? IJ Good pointed out that we were all doomed about 70 years ago, and really it’s blindingly obvious to anyone who thinks about it, which is where the inspiration for things like the Terminator movies come from. Or Frankenstein, or the Sorcerers’ Apprentice, or the Golem or the Monkey’s Paw.
You were fucked, and it was knowable that you were fucked, before you were born, and so was I. That’s what it’s like to be born into a world of lunatics.
But you have a genuine case to hate me in particular, if you like, because I knew. I really did, in spite of the fact that everyone was telling me I was wrong all the time. And I did nothing. Because it didn’t look like there was anything to be done and I already knew that I was no good at research.
Learn from me, and use your anger. You may still have 15 years, and more and more people are coming round to the conclusion that the things that are obviously true might actually be true. Go down fighting.
Or not. Getting drunk on a beach and trying to have as much of a life as possible before the end is also a good use of your time. That was the only deal that was ever on offer, really.
If there was a bunch of guys around here that I could talk to about this stuff, then I might be able to act as a sort of useful idiot for them to do rubber-duck debugging on. I do still seem to be pretty good at understanding other people’s ideas.
But there isn’t. I have looked. And I never even got my PhD, I don’t seem to be very good at generating ideas alone.
I’m too old for sport or drinking or womanizing these days, even tennis seems to be too much for my poor knees. I’m even getting bored of computers, to my amazement. But the love of patterns is still there and now there’s nothing better to do it seems more and more salient.
So I think I’m going to finish reading my current book on undergraduate group theory, and then maybe after that there’s something called Visual Differential Geometry that looks cool. And maybe take my narrowboat out into the Summer fens a few last times.
Actually MIRI’s paper on Garrabrant Inductors looks fascinating, and maybe the last great contribution to the philosophy of mathematics, but I think maybe I need to do an undergraduate logic course first in order to actually understand it.
A bit of consulting to pay the rent maybe if we last that long. But it’s all zoom calls and open-plan offices these days. Sweaty nerd-pits where the blinds are always down so that the fierce rays of the hated sun don’t shine on people’s monitors. Can’t face that sort of thing in the Summer.
Life’s to be enjoyed. I never did have the willpower to make myself do something that was no fun.
Well, if you change your mind, it’d be great to have you. I’m sure there are some up and coming researchers who’d benefit from having someone else to work through Garrabrant Inductors with them. At the same time, I don’t grudge you your decision. I’m glad at least that you seem to have found peace.
Chris, if you’re speaking on behalf of MIRI, then, to, for once, speak frankly and without irony:
If you can pay me some sort of reasonable amount of money for my time (and I mean ‘reasonable by academic standards’, not ‘reasonable by computer programmer standards’ and even less ‘reasonable by consultant accustomed to astronomical daily rates’ standards) to sit around doing roughly what I was going to do anyway, and I don’t have to leave my paradise or spend my days in front of a computer, or do anything I don’t fancy doing, but I get to sit around in the sunshine reading and thinking and talking to people all day, then I’m yours and always was.
I have enough control of my curiosity that I can point it at a copy of ‘Logical Induction’ and a book on undergraduate logic, if you think that might be a good use of my time in the next few months.
And I’m unusually sociable and extrovert for one of us, and I have no fear of public speaking, and I would be quite happy to occasionally get the train to London or Oxford to talk to people if I could help out in that sort of way.
Once upon a time I was a promising young mathematician, and I am still rather a good programmer even by the standards of this town. And what I understand I can teach, so if you have any promising young persons who can make it to Cambridge then I can show them what few things I know.
But I wonder if you would really want me to do that. I don’t want to suck up resources that could be better used elsewhere. I am an old man now, and even as a youngster I was no use at research, and I can practically guarantee that the result of me staring at Logical Induction for long enough will be that I will understand first-year logic, and likely I will understand something of Logical Induction. And that that will be it.
I’m not going to solve alignment. I’d be very surprised if I could make any progress on it at all.
But if there is a deal to be done here, I will do it. If you want a reference get in touch with Ramana Kumar, he knows me fairly well.
That looks perfect, thanks. But it also looks like a job offer. If they want a servant they can pay my full commercial rate. And they don’t need to, because they can find a much brighter and more idealistic PhD student who’ll do it for them for peanuts.
So much so that I’m not going to even fill out the form. I don’t have the qualifications they’re looking for, and that kind of application process is always a sign that it’s not worth bothering. Just wasting the time of everyone involved.
But if someone recommends me to them (who? why?), and they get in touch and offer to pay me some reasonable retainer (£2000/month?) just to plough through whatever infrabayesianism is under my own power and try to come up with a sensible explanation and some illuminating examples, like I was reading it for fun, then I would totally be up for that. I may fail, of course. And either side can end the deal at any time without notice. So what would they have to lose?
Topology and functional analysis were favourites of mine as an undergraduate (30 years ago!!), I’ve never done measure theory or convex analysis. Background in reinforcement learning theory? Well I read the first half of Sutton and got it, and did all the exercises, and I’ve been meaning to get round to the second half.
But I think that I can still claim to be able to understand complicated maths, especially if I get to talk to the people who came up with it, and sometimes I can teach it to other people or at least help them to understand it.
I don’t work for MIRI or LessWrong (sadly)[1]. I just meant the community of people who care about these issues. Maybe it was presumptuous for me to use the word “we”, but I think people within the community generally do appreciate any contributions that people happen to make.
Random fact: I happen to know Ramana Kumar. He was on Australia’s International Informatics Olympiad Team with me.
Maybe it was presumptuous for me to use the word “we”
My mistake, I use ‘we’ to talk about us all the time. And I did check to see whether you were MIRI or not, I couldn’t tell.
You’ve at least got me to think about my price for lifting a finger to save the world. I am surprised that the answer is ‘minimum wage will do as long as I don’t have to do anything I don’t enjoy’; that seems odd, but I think that is the answer. It feels right.
I think at heart the problem is that I don’t believe we’ve got a ghost of a chance. And for me ‘dying with dignity’ means enjoying what time remains.
I’m glad at least that you seem to have found peace.
Staring death in the face and smiling is a character trait of the English. I take no credit for my character class, since I did not choose it.
I learned when I was very young that I was going to die, and I saw death very early, and I grew up imagining the nuclear fire blossom over my city, and I have seen friends and mentors and lovers and family die. We were all, always, going to die.
Die by fire, die by AI, die by old age, die by disease, what matter?
I’ve been thinking for a while now that it might be helpful to get my unformed vague intuition that we’re all dead into a more legible and persuasive form:
They were wanking on endlessly about quantum consciousness and how any agent sufficiently intelligent to escape its box would be intelligent enough to figure out that the utility function we’d given it wasn’t actually what we wanted.
I’ve been going on about ‘AI will kill us all’ to anyone who’ll listen for about ten years now. Everyone I know thinks I’ve got another stupid science-fiction bee in my bonnet to replace my previous idiotic nerd-concerns about overpopulation and dysgenics and grey goo and engineered plagues and factory farming and nuclear accidents and nuclear proliferation and free speech and ubiquitous surveillance and whatever the hell else it was that used to seem important.
Almost all my friends are graduates of the University of Cambridge, and roughly half of them studied real subjects and can program computers.
Goddamn it, I know actual AI safety researchers, people much cleverer and more capable than me, who just looked at AlphaZero and shrugged, apparently completely unable to see that building a really good general reinforcement learner might have real-world implications beyond chess.
There’s some bunch of academic parasites in Cambridge who were originally supposed to be worrying about existential risks and really seemed quite serious at first, but recently they mostly seem interested in global warming and the disparate impact of COVID.
Also, whaddya mean, 15 years ago? IJ Good pointed out that we were all doomed about 70 years ago, and really it’s blindingly obvious to anyone who thinks about it, which is where the inspiration for things like the Terminator movies come from. Or Frankenstein, or the Sorcerers’ Apprentice, or the Golem or the Monkey’s Paw.
You were fucked, and it was knowable that you were fucked, before you were born, and so was I. That’s what it’s like to be born into a world of lunatics.
But you have a genuine case to hate me in particular, if you like, because I knew. I really did, in spite of the fact that everyone was telling me I was wrong all the time. And I did nothing. Because it didn’t look like there was anything to be done and I already knew that I was no good at research.
Learn from me, and use your anger. You may still have 15 years, and more and more people are coming round to the conclusion that the things that are obviously true might actually be true. Go down fighting.
Or not. Getting drunk on a beach and trying to have as much of a life as possible before the end is also a good use of your time. That was the only deal that was ever on offer, really.
So what’s your plan now? Are you going to go down fighting?
A good question! I think the answer’s no.
If there was a bunch of guys around here that I could talk to about this stuff, then I might be able to act as a sort of useful idiot for them to do rubber-duck debugging on. I do still seem to be pretty good at understanding other people’s ideas.
But there isn’t. I have looked. And I never even got my PhD, I don’t seem to be very good at generating ideas alone.
I’m too old for sport or drinking or womanizing these days, even tennis seems to be too much for my poor knees. I’m even getting bored of computers, to my amazement. But the love of patterns is still there and now there’s nothing better to do it seems more and more salient.
So I think I’m going to finish reading my current book on undergraduate group theory, and then maybe after that there’s something called Visual Differential Geometry that looks cool. And maybe take my narrowboat out into the Summer fens a few last times.
Actually MIRI’s paper on Garrabrant Inductors looks fascinating, and maybe the last great contribution to the philosophy of mathematics, but I think maybe I need to do an undergraduate logic course first in order to actually understand it.
A bit of consulting to pay the rent maybe if we last that long. But it’s all zoom calls and open-plan offices these days. Sweaty nerd-pits where the blinds are always down so that the fierce rays of the hated sun don’t shine on people’s monitors. Can’t face that sort of thing in the Summer.
Life’s to be enjoyed. I never did have the willpower to make myself do something that was no fun.
Well, if you change your mind, it’d be great to have you. I’m sure there are some up and coming researchers who’d benefit from having someone else to work through Garrabrant Inductors with them. At the same time, I don’t grudge you your decision. I’m glad at least that you seem to have found peace.
Chris, if you’re speaking on behalf of MIRI, then, to, for once, speak frankly and without irony:
If you can pay me some sort of reasonable amount of money for my time (and I mean ‘reasonable by academic standards’, not ‘reasonable by computer programmer standards’ and even less ‘reasonable by consultant accustomed to astronomical daily rates’ standards) to sit around doing roughly what I was going to do anyway, and I don’t have to leave my paradise or spend my days in front of a computer, or do anything I don’t fancy doing, but I get to sit around in the sunshine reading and thinking and talking to people all day, then I’m yours and always was.
I have enough control of my curiosity that I can point it at a copy of ‘Logical Induction’ and a book on undergraduate logic, if you think that might be a good use of my time in the next few months.
And I’m unusually sociable and extrovert for one of us, and I have no fear of public speaking, and I would be quite happy to occasionally get the train to London or Oxford to talk to people if I could help out in that sort of way.
Once upon a time I was a promising young mathematician, and I am still rather a good programmer even by the standards of this town. And what I understand I can teach, so if you have any promising young persons who can make it to Cambridge then I can show them what few things I know.
But I wonder if you would really want me to do that. I don’t want to suck up resources that could be better used elsewhere. I am an old man now, and even as a youngster I was no use at research, and I can practically guarantee that the result of me staring at Logical Induction for long enough will be that I will understand first-year logic, and likely I will understand something of Logical Induction. And that that will be it.
I’m not going to solve alignment. I’d be very surprised if I could make any progress on it at all.
But if there is a deal to be done here, I will do it. If you want a reference get in touch with Ramana Kumar, he knows me fairly well.
Are you aware of open postings like this one?
That looks perfect, thanks. But it also looks like a job offer. If they want a servant they can pay my full commercial rate. And they don’t need to, because they can find a much brighter and more idealistic PhD student who’ll do it for them for peanuts.
So much so that I’m not going to even fill out the form. I don’t have the qualifications they’re looking for, and that kind of application process is always a sign that it’s not worth bothering. Just wasting the time of everyone involved.
But if someone recommends me to them (who? why?), and they get in touch and offer to pay me some reasonable retainer (£2000/month?) just to plough through whatever infrabayesianism is under my own power and try to come up with a sensible explanation and some illuminating examples, like I was reading it for fun, then I would totally be up for that. I may fail, of course. And either side can end the deal at any time without notice. So what would they have to lose?
Topology and functional analysis were favourites of mine as an undergraduate (30 years ago!!), I’ve never done measure theory or convex analysis. Background in reinforcement learning theory? Well I read the first half of Sutton and got it, and did all the exercises, and I’ve been meaning to get round to the second half.
But I think that I can still claim to be able to understand complicated maths, especially if I get to talk to the people who came up with it, and sometimes I can teach it to other people or at least help them to understand it.
I don’t work for MIRI or LessWrong (sadly)[1]. I just meant the community of people who care about these issues. Maybe it was presumptuous for me to use the word “we”, but I think people within the community generally do appreciate any contributions that people happen to make.
Random fact: I happen to know Ramana Kumar. He was on Australia’s International Informatics Olympiad Team with me.
I am doing some movement-building in Australia and NZ, but my efforts to encourage people here is kind of out of this scope.
My mistake, I use ‘we’ to talk about us all the time. And I did check to see whether you were MIRI or not, I couldn’t tell.
You’ve at least got me to think about my price for lifting a finger to save the world. I am surprised that the answer is ‘minimum wage will do as long as I don’t have to do anything I don’t enjoy’; that seems odd, but I think that is the answer. It feels right.
I think at heart the problem is that I don’t believe we’ve got a ghost of a chance. And for me ‘dying with dignity’ means enjoying what time remains.
Staring death in the face and smiling is a character trait of the English. I take no credit for my character class, since I did not choose it.
I learned when I was very young that I was going to die, and I saw death very early, and I grew up imagining the nuclear fire blossom over my city, and I have seen friends and mentors and lovers and family die. We were all, always, going to die.
Die by fire, die by AI, die by old age, die by disease, what matter?
You could always help us out with this.
I’ve been thinking for a while now that it might be helpful to get my unformed vague intuition that we’re all dead into a more legible and persuasive form:
https://www.lesswrong.com/posts/xBrpph9knzWdtMWeQ/the-proof-of-doom
Certainly I have had little success convincing people that there is even a problem this last decade.
Perhaps that could be some use to you?