I feel like I have a broad distribution over worlds and usually answer questions with probability distributions, that I have a complete mental universe (which feels to me like it outputs answers to a much broader set of questions than Eliezer’s, albeit probabilistic ones, rather than bailing with “the future is hard to predict”).
Sometimes I’ll be tracking a finite number of “concrete hypotheses”, where every hypothesis is ‘fully fleshed out’, and be doing a particle-filtering style updating process, where sometimes hypotheses gain or lose weight, sometimes they get ruled out or need to split, or so on. In those cases, I’m moderately confident that every ‘hypothesis’ corresponds to a ‘real world’, constrained by how well as I can get my imagination to correspond to reality. [A ‘finite number’ depends on the situation, but I think it’s normally something like 2-5, unless it’s an area I’ve built up a lot of cache about.]
Sometimes I’ll be tracking a bunch of “surface-level features”, where the distributions on the features don’t always imply coherent underlying worlds, either on their own or in combination with other features. (For example, I might have guesses about the probability that a random number is odd and a different guess about the probability that a random number is divisible by 3 and, until I deliberately consider the joint probability distribution, not have any guarantee that it’ll be coherent.)
Normally I’m doing something more like a mixture of those, which I think of as particles of incomplete world models, with some features pinned down and others mostly ‘surface-level features’. I can often simultaneously consider many more of these; like, when I’m playing Go, I might be tracking a dozen different ‘lines of attack’, which have something like 2-4 moves clearly defined and the others ‘implied’ (in a way that might not actually be consistent).
Are any of those like your experience? Or is there some other way you’d describe it?
different ideas about what kinds of standards discourse should aspire to
Have you written about this / could you? I’d be pretty excited about being able to try out discoursing with people in a Paul-virtuous way.
I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
The main complexity feels like the thing you point out where it’s impossible to make them fully fleshed out, so you build a bunch of intuitions about what is consistent (and could be fleshed out given enough time) and then refine those intuitions only periodically when you actually try to flesh something out and see if it makes sense. And often you go even further and just talk about relationships amongst surface level features using intuitions refined from a bunch of samples.
I feel like a distinctive feature of Eliezer’s dialog w.r.t. foom / alignment difficulty is that he has a lot of views about strong regularities that should hold across all of these worlds. And then disputes about whether worlds are plausible often turn on things like “is this property of the described world likely?” which is tough because obviously everyone agrees that every particular world is unlikely. To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
This is all a bit of a guess though / rambling speculation.
I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
Oh whoa, you don’t remember your samples from before? [I guess I might not either, unless I’m concentrating on keeping them around or verbalized them or something; probably I do something more expert-iteration-like where I’m silently updating my generating distributions based on the samples and then resampling them in the future.]
To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
Yeah, this seems likely; this makes me more interested in the “selectively ignoring variables” hypothesis for why Eliezer running this strategy might have something that would naturally be called a mainline. [Like, it’s very easy to predict “number of apples sold = number of apples bought” whereas it’s much harder to predict the price of apples.] But maybe instead he means it in the ‘startup plan’ sense, where you do actually assign basically no probability to your mainline prediction, but still vastly more than any other prediction that’s equally conjunctive.
Sometimes I’ll be tracking a finite number of “concrete hypotheses”, where every hypothesis is ‘fully fleshed out’, and be doing a particle-filtering style updating process, where sometimes hypotheses gain or lose weight, sometimes they get ruled out or need to split, or so on. In those cases, I’m moderately confident that every ‘hypothesis’ corresponds to a ‘real world’, constrained by how well as I can get my imagination to correspond to reality. [A ‘finite number’ depends on the situation, but I think it’s normally something like 2-5, unless it’s an area I’ve built up a lot of cache about.]
Sometimes I’ll be tracking a bunch of “surface-level features”, where the distributions on the features don’t always imply coherent underlying worlds, either on their own or in combination with other features. (For example, I might have guesses about the probability that a random number is odd and a different guess about the probability that a random number is divisible by 3 and, until I deliberately consider the joint probability distribution, not have any guarantee that it’ll be coherent.)
Normally I’m doing something more like a mixture of those, which I think of as particles of incomplete world models, with some features pinned down and others mostly ‘surface-level features’. I can often simultaneously consider many more of these; like, when I’m playing Go, I might be tracking a dozen different ‘lines of attack’, which have something like 2-4 moves clearly defined and the others ‘implied’ (in a way that might not actually be consistent).
Are any of those like your experience? Or is there some other way you’d describe it?
Have you written about this / could you? I’d be pretty excited about being able to try out discoursing with people in a Paul-virtuous way.
I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
The main complexity feels like the thing you point out where it’s impossible to make them fully fleshed out, so you build a bunch of intuitions about what is consistent (and could be fleshed out given enough time) and then refine those intuitions only periodically when you actually try to flesh something out and see if it makes sense. And often you go even further and just talk about relationships amongst surface level features using intuitions refined from a bunch of samples.
I feel like a distinctive feature of Eliezer’s dialog w.r.t. foom / alignment difficulty is that he has a lot of views about strong regularities that should hold across all of these worlds. And then disputes about whether worlds are plausible often turn on things like “is this property of the described world likely?” which is tough because obviously everyone agrees that every particular world is unlikely. To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
This is all a bit of a guess though / rambling speculation.
Oh whoa, you don’t remember your samples from before? [I guess I might not either, unless I’m concentrating on keeping them around or verbalized them or something; probably I do something more expert-iteration-like where I’m silently updating my generating distributions based on the samples and then resampling them in the future.]
Yeah, this seems likely; this makes me more interested in the “selectively ignoring variables” hypothesis for why Eliezer running this strategy might have something that would naturally be called a mainline. [Like, it’s very easy to predict “number of apples sold = number of apples bought” whereas it’s much harder to predict the price of apples.] But maybe instead he means it in the ‘startup plan’ sense, where you do actually assign basically no probability to your mainline prediction, but still vastly more than any other prediction that’s equally conjunctive.