Does the discussion of rationality techniques have a larger market than debates over Sleeping Beauty (I’m even beginning to miss those!)
Wow, I’d forgotten all about those. Those days were fun. We actually had to, well, think occasionally. Nothing remotely challenging has cropped up in a while!
Those days were fun. We actually had to, well, think occasionally. Nothing remotely challenging has cropped up in a while!
If you like thinking about challenging theoretical rationality problems, there are plenty of those left (logical uncertainty/bounded rationality, pascal’s wager/mugging/decision theory for running on error-prone hardware, moral/value uncertainty, nature of anticipation/surprise/disappointment/good and bad news, complexity/Occam’s razor).
I’ve actually considered writing a post titled “Drowning in Rationality Problems” to complain about how little we still know about the theory of rationality and how few LWers seem to be working actively on the subject, but I don’t know if that’s a good way to motivate people. So I guess what I’d like to know is (and not in a rhetorical sense), what’s stopping you (and others) from thinking about these problems?
Maybe I will, after I get a better idea why more people aren’t already working on these problems. One reason not to write it is that the feeling of being drowned in big problems is not a particularly good one, and possibly de-motivating. Sometimes I wish I could go back to 1998, when I thought Bayesianism was a pretty much complete solution to the problem of epistemology, except for this little issue of what to expect when you’re about to get copied twice in succession...
By the way, have you seen how I’ve been using MathOverflow recently? It seems that if you can reduce some problem to a short math question in standard terms, the default next action (after giving it your own best shot) should be posting it on MO. So far I’ve posted twoproblems that interested me, and both got solved within an hour.
So far I’ve posted two problems that interested me, and both got solved within an hour.
It’s all magic to me but it looks like a very effective human resource. Have you considered pushing MathOverflow to its limits and see if those people there might actually be able to make valuable contributions to open problems faced by Less Wrong or the SIAI?
I assume that the main obstacle in effectively exploiting such resources as MathOverflow is to formalize the problems that are faced by people working to refine rationality or create FAI. Once you know how to ask the the right questions one could spread them everywhere and see if there is someone who might be able to answer them, or if there is already a known solution.
Currently it appears to me that most of the important problems are not widely known, a lot of them being mainly discussed here on Less Wrong or on obscure mailing lists. By formalizing and spreading the gist of those problems one would be able to make people aware of Less Wrong and risks from AI and exploit various resources.
What I am thinking about is analogous to a huge roadside billboard with a short but succinct description of an important problem. Someone really smart or knowledgeable might drive-by and solve it. Not only would the solution be valuable but you would win a potential new human resource.
I’m all for exploiting resources to the limit! The bottleneck is formalizing the problems. It’s very slow and difficult work for me, and the SIAI people aren’t significantly faster at this task, as far as I can see.
One reason not to write it is that the feeling of being drowned in big problems is not a particularly good one, and possibly de-motivating
No! What is particularly demotivating for me is that I don’t know what heuristics I can trust and when I am better off trusting my intuitions (e.g. Pascal’s Mugging).
If someone was going to survey the rationality landscape and outline what we know and where we run into problems, it would help a lot by making people aware of the big and important problems.
I suspect that clearly defining open rationality problems would act as a focusing lens for action, not a demotivator. Please do publish your list of open rationality problems. Do for us what Hilbert did for mathematicians. But you don’t have to talk about ‘drowning.’ :)
So I guess what I’d like to know is (and not in a rhetorical sense), what’s stopping you (and others) from thinking about these problems?
Part of this issue for me, at least with respect to thinking about the problems in the context of lesswrong, is that the last efforts in that direction were stiffled rather brutally—to the extent that we lost one of the most technically orientated and prolific users. This isn’t to comment on the rightness or wrongness of that decision—just a description of an influence. Having big brother looming over the shoulder specifying what you may think makes it work instead of fun. And not work that I have any particular comparative advantage in! (I can actually remember thinking to myself “folks like Wei Dai are more qualified to tackle these sort of things efficiently, given their prior intellectual investments”.
Creating of a decision theory email list with a large overlap of LW posters also served to dilute attention, possibly reducing self-reinforcing curiosity to some degree.
But for me personally I have just had other things to be focussing my intellectual attention on. I felt (theoretical) rationality and decision theory get uncached from my brain as I loaded it up with biology and German. This may change in the near future. I’m heading over to Jasen’s training camp next month and that is likely to kick the mental focus around about.
I second cousin_it’s interest in your aforementioned post! It would actually be good to know which problems are not solved as opposed to which problems I just don’t know the solution to. Or, for that matter, which problems I think I know the solution to but really don’t.
Having big brother looming over the shoulder specifying what you may think makes it work instead of fun.
I do not really get this reaction. So what if Eliezer has a tendency to over-censor? I was once banned completely from a mailing list but it didn’t make me terribly upset or lose interest in the subject matter of the list. The Roko thing seems even less of a big deal. (I thought Roko ended up agreeing that it was a mistake to make the post. He could always post it elsewhere if he doesn’t agree. It’s not as if Eliezer has control over the whole Internet.)
And not work that I have any particular comparative advantage in! (I can actually remember thinking to myself “folks like Wei Dai are more qualified to tackle these sort of things efficiently, given their prior intellectual investments”.
I didn’t think I had any particular advantage when I first started down this path either. I began with what I thought was just fun little puzzle in an otherwise well-developed area, which nobody else was trying to solve because they didn’t notice it as a problem yet. So, I’m a bit weary about presenting “a list of really hard and important problems” and scaring people away. (Of course I may be scaring people away just through this discussion, but probably only a minority of LWers are listening to us.)
I second cousin_it’s interest in your aforementioned post! It would actually be good to know which problems are not solved as opposed to which problems I just don’t know the solution to. Or, for that matter, which problems I think I know the solution to but really don’t.
I guess another factor is that I have the expectation that if someone is really interested in this stuff (i.e., have a “burning need to know”), they would already have figured out which problems are not solved as opposed to which problems they just don’t know the solution to, because they would have tried every available method to find existing solutions to these problems. It seems unlikely that they’d have enough motivation to make much progress if they didn’t have at least that level of interest.
So I’ve been trying to figure out (without much success) how to instill this kind of interest in others, and again, I’m not sure presenting a list of important unsolved problems is the best way to do it.
So I’ve been trying to figure out (without much success) how to instill this kind of interest in others, and again, I’m not sure presenting a list of important unsolved problems is the best way to do it.
I’m not sure either. It would perhaps be a useful reference but not a massive motivator in its own right.
What I know works best as a motivator for me is putting up sample problems—presenting the subject matter in ‘sleeping hitchiker terrorist inna box’ form. When seeing a concrete (albeit extremely counterfactual) problem I get nerd sniped. I am being entirely literal when I say that takes a massive amount of willpower for me to stop myself from working on it. To the extent that there is less perceived effort for tackling the problem for 15 hours straight than there is for putting it aside. And that can be the start of a self reinforcement cycle at times.
The above is in contrast to just seeing the unsolved problems listed. That format is approximately inspiration neutral.
By the way, is that decision theory list still active? I was subscribed but haven’t seen anything appear of late.
What I know works best as a motivator for me is putting up sample problems—presenting the subject matter in ‘sleeping hitchiker terrorist inna box’ form.
That seems like a useful datum, thanks.
By the way, is that decision theory list still active? I was subscribed but haven’t seen anything appear of late.
It’s still active, but nobody has made a post for about a month.
I guess another factor is that I have the expectation that if someone is really interested in this stuff (i.e., have a “burning need to know”), they would already have figured out which problems are not solved as opposed to which problems they just don’t know the solution to, because they would have tried every available method to find existing solutions to these problems.
Discussing things that are already known can help in understanding them better. Also, the “burning need to know” occasionally needs to be ignited, or directed. I don’t study decision theory because I like studying decision theory in particular, even though it’s true that I always had a tendency to obsessively study something.
But decision theory ought to be a natural attractor for anyone with intellectual interests (any intellectual question → how am I supposed to answer questions like that? → epistemology → Bayesianism → nature of probability → decision theory). What’s stopping people from getting to the end of this path? Or am I just a freak in my tendency to “go meta”?
Yes, you’re a freak and nobody but you and a few other freaks can ever get any useful thinking done and didn’t we sort of cover this territory already?
I’m confused. Should I stop thinking about how exactly I’m “freaky” and how to possibly reproduce that “freakiness” in others? Has the effort already reached diminishing returns, or was it doomed from the start? Or do you think I’m just looking for ego-stroking or something?
Going meta takes resources. Resources could instead be applied directly to the problem in front of you. If not solving the problem right in front of you causes long term hard to recover from problems it makes sense to apply your resources directly to the problem at hand.
So:
(any intellectual question → how am I supposed to answer questions like that? → epistemology → Bayesianism → nature of probability → decision theory)
Seems rational when enough excess resources are available. To make more people follow this path you need:
To increase the resources of those you are trying to teach.
Lower the resource cost of following the path
Lesswrong.com and Lesswrong meetup groups teach life skills to increase the members resources. At the same time they gather people who know skills on the path with those who want to learn lowering the resource cost of following the path. Many other methods exist, I have just mentioned two. A road is being built it has just not reached where you are yet.
Perhaps you are ahead of the road marking the best routes, or clearing the ground, but not everyone have the resources to get so far without a well paved road.
Or morality! (Any action → but is that the right thing to do? → combinatorial explosion of extremely confusing open questions about cognitive science and decision theory and metaphysics and cosmology and ontology of agency and arghhhhhh.) It’s like the universe itself is a Confundus Charm and nobody notices.
How much of decision theory requires good philosophical intuition? If you could convince everyone at MathOverflow to familiarize themselves with it and work on it for a few months, would you expect them to make huge amounts of progress? If so, I admit I am surprised there aren’t more mathy folk sniping at decision theory just for meta’s sake.
Creating of a decision theory email list with a large overlap of LW posters also served to dilute attention, possibly reducing self-reinforcing curiosity to some degree.
I wasn’t aware this list existed, but would be very interested in reading its archives. Do you have a link?
I’ve been less engaged with the old topics for the last several months while trying to figure out an updateful way of thinking about decision problems (understand the role of observations, as opposed to reducing them to non-observations as UDT does; and construct an ADT-like explicit toy model). This didn’t produce communicable intermediate results (the best I could manage was this post, for which quite possibly nobody understood the motivation). Just a few days ago, I think I figured out the way of formalizing this stuff (which is awfully trivial, but might provide a bit of methodological guidance to future research).
In short, progress is difficult and slow where we don’t have a sufficient number of tools which would suggest actionable open problems that we could assign to metaphorical grad students. This also sucks out all motivation for most people who could be working on these topics, since there is little expectation of success and little understanding of what such success would look like. Even I actually work while expecting to most likely not produce anything particularly useful in the long run (there’s only a limited chance for limited success), but I’m a relatively strange creature. Academia additionally motivates people by rewarding the activity of building in known ways on existing knowledge without producing a lot of benefit, but producing visible, and possibly of high quality, if mostly useless results that gradually build up to systematic improvements.
Because it’s awfully trivial and it’s not easy to locate all the pieces of motivation and application that would make anyone enthusiastic about this. Like the fact that action and utility are arbitrary mathematical structures in ADT and not just integer outputs of programs.
Hm, I don’t see any trivial way of understanding observational knowledge except by treating it as part of the input-output map as UDT suggests. So if your idea is different, I’m still asking you to write it up.
In one sentence: Agent sees the world from within a logical theory in which observations are nonlogical symbols. I’ll of course try to write this up in time.
(the best I could manage was this post, for which quite possibly nobody understood the motivation)
I’m reasonably sure that’s because the problem you see doesn’t actually exist in your example and you only think it does because you misapplied UDT. If you think this is important why did you never get back to our discussion there as you promised? That might result in either a better understanding why this is so difficult for other people to grasp (if I was misunderstanding you or making a non-obvious mistake) or either a dissolution of the apparent problem or examples where it actually comes up (if I was right).
For some reason, I find it difficult to reason about these problems, and have never acquired a facility of easily seeing them all the way through, so it’s hard work for me to follow these discussions. I expect I was not making an error in understanding the problem the way it was intended, and figuring out the details of your way of parsing the problem was not a priority.
why did you never get back to our discussion there as you promised?
It feels emotionally difficult to terminate a technical discussion (where all participants invested nontrivial effort), while postponing it for a short time can be necessary, in which case there is an impulse to signal to others the lack of intention to actually stop the discussion, to signal the temporary nature of the present pause (but then, motivation to continue evaporates or gets revoked on reflection). I’ll try to keep in mind that making promises for continuing the discussion is a bad, no good way of communicating this (it happened recently again in a discussion with David Gerard about merits of wiki-managing policies; I edited out the promise in a few hours).
At this point, if you feel that you have a useful piece of knowledge which our discussion failed to communicate, I can only offer you a suggestion to write up your position as a (more self-contained) discussion post.
Yes. There are tons of open, difficult rationality/philosophical problems. If they haven’t ‘cropped up in a while’ on LW, it’s because those who are thinking about them aren’t taking the time to write about them. That’s quite understandable because writing takes a lot of time.
However, I tend to think that there are enough very smart rationalists on LW that if we can cogently bring everybody up to the cutting edge and then explain what the open problems are, progress will be made.
That’s really where I’m going with my metaethics sequence. I don’t have the hard problems of metaethics solved; only the easy ones. I’m hoping that bringing everybody up to the cutting edge and explaining what the open problems are will launch discussions that lead to incremental progress.
I have a vague intuition there’s something interesting that could happen with self-modifying AIs with creator and successor states knowably running on error-prone hardware while having pseudo-universal hypothesis generators that will of course notice the possibility of values corruption. I guess I’m still rooting for the ‘infinite reflection = contextually perfect morality’ deus ex machina. Utility functions as they’re normally idealized for imagining superintelligence behavior like in Basic AI Drives look an awful lot like self-protecting beliefs, which feels more and more decision theoretically wrong as time goes on. I trust the applicability of the symbols of expected utility theory less over time and trust common beliefs about the automatic implications of putting those symbols in a seed AI even less than that. Am I alone here?
The reason I am not attempting to tackle those problems is because I hang out with Steve Rayhawk and assume that if I was going to make any progress I’d have to be roughly as smart and knowledgeable as Steve Rayhawk, ’cuz if he hasn’t solved something yet that means I’d have to be smarter than him to solve it. I subconsciously intuit that as impossible so I try to specialize in pulling on less mathy yarns instead, which is actually a lot more possible than I’d anticipated but took me a long time to get passable at.
I trust the applicability of the symbols of expected utility theory less over time and trust common beliefs about the automatic implications of putting those symbols in a seed AI even less than that. Am I alone here?
The current theory is all fine—until you want to calculate utility based on something other than expected sensory input data. Then the current theory doesn’t work very well at all. The problem is that we don’t yet know how to code: “not what you are seeing, how the world really is” in a machine-readable format.
I’ve actually considered writing a post titled “Drowning in Rationality Problems” to complain about how little we still know about the theory of rationality and how few LWers seem to be working actively on the subject, but I don’t know if that’s a good way to motivate people.
I don’t think it’s necessary to frame the large number of problems you identify as a case of “drowning”. An alternative framing might be one about unexplored territory, something like “rationality is fertile ground for intellectual types who wish to acquire status by solving problems”.
As for why more people aren’t working on them, it could come down to simple herd effects, or something like that.
Could someone point me to an explanation of what is meant by ‘logical uncertainty’?
I’ve actually considered writing a post titled “Drowning in Rationality Problems” to complain about how little we still know about the theory of rationality...
This sounds incredible interesting, I would love to read it!
Logical uncertainty is uncertainty about the unknown outputs of known computations. For example, if you have a program for computing the digits of pi but don’t have enough time to run it, you have logical uncertainty about the billionth digit. You can express it with probabilities or maybe use some other representation. The mystery is how to formulate a decision process that makes provably “nice” decisions under logical uncertainty, and to precisely define the meaning of “nice”.
Wow, I’d forgotten all about those. Those days were fun. We actually had to, well, think occasionally. Nothing remotely challenging has cropped up in a while!
If you like thinking about challenging theoretical rationality problems, there are plenty of those left (logical uncertainty/bounded rationality, pascal’s wager/mugging/decision theory for running on error-prone hardware, moral/value uncertainty, nature of anticipation/surprise/disappointment/good and bad news, complexity/Occam’s razor).
I’ve actually considered writing a post titled “Drowning in Rationality Problems” to complain about how little we still know about the theory of rationality and how few LWers seem to be working actively on the subject, but I don’t know if that’s a good way to motivate people. So I guess what I’d like to know is (and not in a rhetorical sense), what’s stopping you (and others) from thinking about these problems?
I’d like you to write that post.
Maybe I will, after I get a better idea why more people aren’t already working on these problems. One reason not to write it is that the feeling of being drowned in big problems is not a particularly good one, and possibly de-motivating. Sometimes I wish I could go back to 1998, when I thought Bayesianism was a pretty much complete solution to the problem of epistemology, except for this little issue of what to expect when you’re about to get copied twice in succession...
By the way, have you seen how I’ve been using MathOverflow recently? It seems that if you can reduce some problem to a short math question in standard terms, the default next action (after giving it your own best shot) should be posting it on MO. So far I’ve posted two problems that interested me, and both got solved within an hour.
It’s all magic to me but it looks like a very effective human resource. Have you considered pushing MathOverflow to its limits and see if those people there might actually be able to make valuable contributions to open problems faced by Less Wrong or the SIAI?
I assume that the main obstacle in effectively exploiting such resources as MathOverflow is to formalize the problems that are faced by people working to refine rationality or create FAI. Once you know how to ask the the right questions one could spread them everywhere and see if there is someone who might be able to answer them, or if there is already a known solution.
Currently it appears to me that most of the important problems are not widely known, a lot of them being mainly discussed here on Less Wrong or on obscure mailing lists. By formalizing and spreading the gist of those problems one would be able to make people aware of Less Wrong and risks from AI and exploit various resources.
What I am thinking about is analogous to a huge roadside billboard with a short but succinct description of an important problem. Someone really smart or knowledgeable might drive-by and solve it. Not only would the solution be valuable but you would win a potential new human resource.
I’m all for exploiting resources to the limit! The bottleneck is formalizing the problems. It’s very slow and difficult work for me, and the SIAI people aren’t significantly faster at this task, as far as I can see.
No! What is particularly demotivating for me is that I don’t know what heuristics I can trust and when I am better off trusting my intuitions (e.g. Pascal’s Mugging).
If someone was going to survey the rationality landscape and outline what we know and where we run into problems, it would help a lot by making people aware of the big and important problems.
I suspect that clearly defining open rationality problems would act as a focusing lens for action, not a demotivator. Please do publish your list of open rationality problems. Do for us what Hilbert did for mathematicians. But you don’t have to talk about ‘drowning.’ :)
Second the need for a list of the most important problems.
Part of this issue for me, at least with respect to thinking about the problems in the context of lesswrong, is that the last efforts in that direction were stiffled rather brutally—to the extent that we lost one of the most technically orientated and prolific users. This isn’t to comment on the rightness or wrongness of that decision—just a description of an influence. Having big brother looming over the shoulder specifying what you may think makes it work instead of fun. And not work that I have any particular comparative advantage in! (I can actually remember thinking to myself “folks like Wei Dai are more qualified to tackle these sort of things efficiently, given their prior intellectual investments”.
Creating of a decision theory email list with a large overlap of LW posters also served to dilute attention, possibly reducing self-reinforcing curiosity to some degree.
But for me personally I have just had other things to be focussing my intellectual attention on. I felt (theoretical) rationality and decision theory get uncached from my brain as I loaded it up with biology and German. This may change in the near future. I’m heading over to Jasen’s training camp next month and that is likely to kick the mental focus around about.
I second cousin_it’s interest in your aforementioned post! It would actually be good to know which problems are not solved as opposed to which problems I just don’t know the solution to. Or, for that matter, which problems I think I know the solution to but really don’t.
I do not really get this reaction. So what if Eliezer has a tendency to over-censor? I was once banned completely from a mailing list but it didn’t make me terribly upset or lose interest in the subject matter of the list. The Roko thing seems even less of a big deal. (I thought Roko ended up agreeing that it was a mistake to make the post. He could always post it elsewhere if he doesn’t agree. It’s not as if Eliezer has control over the whole Internet.)
I didn’t think I had any particular advantage when I first started down this path either. I began with what I thought was just fun little puzzle in an otherwise well-developed area, which nobody else was trying to solve because they didn’t notice it as a problem yet. So, I’m a bit weary about presenting “a list of really hard and important problems” and scaring people away. (Of course I may be scaring people away just through this discussion, but probably only a minority of LWers are listening to us.)
I guess another factor is that I have the expectation that if someone is really interested in this stuff (i.e., have a “burning need to know”), they would already have figured out which problems are not solved as opposed to which problems they just don’t know the solution to, because they would have tried every available method to find existing solutions to these problems. It seems unlikely that they’d have enough motivation to make much progress if they didn’t have at least that level of interest.
So I’ve been trying to figure out (without much success) how to instill this kind of interest in others, and again, I’m not sure presenting a list of important unsolved problems is the best way to do it.
I’m not sure either. It would perhaps be a useful reference but not a massive motivator in its own right.
What I know works best as a motivator for me is putting up sample problems—presenting the subject matter in ‘sleeping hitchiker terrorist inna box’ form. When seeing a concrete (albeit extremely counterfactual) problem I get nerd sniped. I am being entirely literal when I say that takes a massive amount of willpower for me to stop myself from working on it. To the extent that there is less perceived effort for tackling the problem for 15 hours straight than there is for putting it aside. And that can be the start of a self reinforcement cycle at times.
The above is in contrast to just seeing the unsolved problems listed. That format is approximately inspiration neutral.
By the way, is that decision theory list still active? I was subscribed but haven’t seen anything appear of late.
That seems like a useful datum, thanks.
It’s still active, but nobody has made a post for about a month.
Ahh, there we go. Cousin_it just woke it up!
Discussing things that are already known can help in understanding them better. Also, the “burning need to know” occasionally needs to be ignited, or directed. I don’t study decision theory because I like studying decision theory in particular, even though it’s true that I always had a tendency to obsessively study something.
But decision theory ought to be a natural attractor for anyone with intellectual interests (any intellectual question → how am I supposed to answer questions like that? → epistemology → Bayesianism → nature of probability → decision theory). What’s stopping people from getting to the end of this path? Or am I just a freak in my tendency to “go meta”?
The wealth of interesting stuff located well before the end.
Seconding Eliezer. Also, please do more of the kind of thinking you do :-)
Yes, you’re a freak and nobody but you and a few other freaks can ever get any useful thinking done and didn’t we sort of cover this territory already?
I’m confused. Should I stop thinking about how exactly I’m “freaky” and how to possibly reproduce that “freakiness” in others? Has the effort already reached diminishing returns, or was it doomed from the start? Or do you think I’m just looking for ego-stroking or something?
Going meta takes resources. Resources could instead be applied directly to the problem in front of you. If not solving the problem right in front of you causes long term hard to recover from problems it makes sense to apply your resources directly to the problem at hand.
So:
Seems rational when enough excess resources are available. To make more people follow this path you need:
To increase the resources of those you are trying to teach.
Lower the resource cost of following the path
Lesswrong.com and Lesswrong meetup groups teach life skills to increase the members resources. At the same time they gather people who know skills on the path with those who want to learn lowering the resource cost of following the path. Many other methods exist, I have just mentioned two. A road is being built it has just not reached where you are yet.
Perhaps you are ahead of the road marking the best routes, or clearing the ground, but not everyone have the resources to get so far without a well paved road.
Or morality! (Any action → but is that the right thing to do? → combinatorial explosion of extremely confusing open questions about cognitive science and decision theory and metaphysics and cosmology and ontology of agency and arghhhhhh.) It’s like the universe itself is a Confundus Charm and nobody notices.
How much of decision theory requires good philosophical intuition? If you could convince everyone at MathOverflow to familiarize themselves with it and work on it for a few months, would you expect them to make huge amounts of progress? If so, I admit I am surprised there aren’t more mathy folk sniping at decision theory just for meta’s sake.
I wasn’t aware this list existed, but would be very interested in reading its archives. Do you have a link?
I second jimrandomh’s interest in the mailing list? Can I be signed up for it? Are there archives?
decision-theory-workshop.googlegroups.com
I’m not sure who admins (and so can confirm new subscribers). It’s a google group so the archive may well survive heat death.
Thanks.
(BTW Google seems to be messing with the structure of the URLs for groups, the address that currently works is https://groups.google.com/group/decision-theory-workshop/ )
I’ve been less engaged with the old topics for the last several months while trying to figure out an updateful way of thinking about decision problems (understand the role of observations, as opposed to reducing them to non-observations as UDT does; and construct an ADT-like explicit toy model). This didn’t produce communicable intermediate results (the best I could manage was this post, for which quite possibly nobody understood the motivation). Just a few days ago, I think I figured out the way of formalizing this stuff (which is awfully trivial, but might provide a bit of methodological guidance to future research).
In short, progress is difficult and slow where we don’t have a sufficient number of tools which would suggest actionable open problems that we could assign to metaphorical grad students. This also sucks out all motivation for most people who could be working on these topics, since there is little expectation of success and little understanding of what such success would look like. Even I actually work while expecting to most likely not produce anything particularly useful in the long run (there’s only a limited chance for limited success), but I’m a relatively strange creature. Academia additionally motivates people by rewarding the activity of building in known ways on existing knowledge without producing a lot of benefit, but producing visible, and possibly of high quality, if mostly useless results that gradually build up to systematic improvements.
Uhh, so why don’t I know about it? Could you send an email to me or to the list?
Because it’s awfully trivial and it’s not easy to locate all the pieces of motivation and application that would make anyone enthusiastic about this. Like the fact that action and utility are arbitrary mathematical structures in ADT and not just integer outputs of programs.
Hm, I don’t see any trivial way of understanding observational knowledge except by treating it as part of the input-output map as UDT suggests. So if your idea is different, I’m still asking you to write it up.
In one sentence: Agent sees the world from within a logical theory in which observations are nonlogical symbols. I’ll of course try to write this up in time.
I’m reasonably sure that’s because the problem you see doesn’t actually exist in your example and you only think it does because you misapplied UDT. If you think this is important why did you never get back to our discussion there as you promised? That might result in either a better understanding why this is so difficult for other people to grasp (if I was misunderstanding you or making a non-obvious mistake) or either a dissolution of the apparent problem or examples where it actually comes up (if I was right).
For some reason, I find it difficult to reason about these problems, and have never acquired a facility of easily seeing them all the way through, so it’s hard work for me to follow these discussions. I expect I was not making an error in understanding the problem the way it was intended, and figuring out the details of your way of parsing the problem was not a priority.
It feels emotionally difficult to terminate a technical discussion (where all participants invested nontrivial effort), while postponing it for a short time can be necessary, in which case there is an impulse to signal to others the lack of intention to actually stop the discussion, to signal the temporary nature of the present pause (but then, motivation to continue evaporates or gets revoked on reflection). I’ll try to keep in mind that making promises for continuing the discussion is a bad, no good way of communicating this (it happened recently again in a discussion with David Gerard about merits of wiki-managing policies; I edited out the promise in a few hours).
At this point, if you feel that you have a useful piece of knowledge which our discussion failed to communicate, I can only offer you a suggestion to write up your position as a (more self-contained) discussion post.
Yes. There are tons of open, difficult rationality/philosophical problems. If they haven’t ‘cropped up in a while’ on LW, it’s because those who are thinking about them aren’t taking the time to write about them. That’s quite understandable because writing takes a lot of time.
However, I tend to think that there are enough very smart rationalists on LW that if we can cogently bring everybody up to the cutting edge and then explain what the open problems are, progress will be made.
That’s really where I’m going with my metaethics sequence. I don’t have the hard problems of metaethics solved; only the easy ones. I’m hoping that bringing everybody up to the cutting edge and explaining what the open problems are will launch discussions that lead to incremental progress.
I have a vague intuition there’s something interesting that could happen with self-modifying AIs with creator and successor states knowably running on error-prone hardware while having pseudo-universal hypothesis generators that will of course notice the possibility of values corruption. I guess I’m still rooting for the ‘infinite reflection = contextually perfect morality’ deus ex machina. Utility functions as they’re normally idealized for imagining superintelligence behavior like in Basic AI Drives look an awful lot like self-protecting beliefs, which feels more and more decision theoretically wrong as time goes on. I trust the applicability of the symbols of expected utility theory less over time and trust common beliefs about the automatic implications of putting those symbols in a seed AI even less than that. Am I alone here?
The reason I am not attempting to tackle those problems is because I hang out with Steve Rayhawk and assume that if I was going to make any progress I’d have to be roughly as smart and knowledgeable as Steve Rayhawk, ’cuz if he hasn’t solved something yet that means I’d have to be smarter than him to solve it. I subconsciously intuit that as impossible so I try to specialize in pulling on less mathy yarns instead, which is actually a lot more possible than I’d anticipated but took me a long time to get passable at.
The current theory is all fine—until you want to calculate utility based on something other than expected sensory input data. Then the current theory doesn’t work very well at all. The problem is that we don’t yet know how to code: “not what you are seeing, how the world really is” in a machine-readable format.
I don’t think it’s necessary to frame the large number of problems you identify as a case of “drowning”. An alternative framing might be one about unexplored territory, something like “rationality is fertile ground for intellectual types who wish to acquire status by solving problems”.
As for why more people aren’t working on them, it could come down to simple herd effects, or something like that.
Could someone point me to an explanation of what is meant by ‘logical uncertainty’?
This sounds incredible interesting, I would love to read it!
Logical uncertainty is uncertainty about the unknown outputs of known computations. For example, if you have a program for computing the digits of pi but don’t have enough time to run it, you have logical uncertainty about the billionth digit. You can express it with probabilities or maybe use some other representation. The mystery is how to formulate a decision process that makes provably “nice” decisions under logical uncertainty, and to precisely define the meaning of “nice”.
So basically the stuff you don’t know because you don’t have logical omniscience.