Steve_Rayhawk

Karma: 1,048

Steve_Rayhawk Jul 10, 2012, 4:44 PM
8 points
in reply to: private_messaging’s comment on: An Intuitive Explanation of Solomonoff Induction
The subtleties I first had in mind were the ones that should have (but didn’t) come up in the original earlier dicussion of MWI, having to do with the different numbers of bits in different parts of an observation-predicting program based on a physical theory, and which of those parts should have their bits be charged against the prior or likelihood of the physical theory itself and which of the parts should have their bits be taken for granted as intrinsic parts of the anthropic reasoning that any agent would need to be capable of (even if some physical theories didn’t use part of that anthropic reasoning “software”).

(I didn’t specify what the subtleties were, and you seem to have picked a reading of which subtleties I must have been referring to and what I must have meant by “resolve” that together made what I was saying make no sense at all. This might be another example of the proposed tendency of “not looking very hard to see whether other people could have reasonably supposed” etc. (whether or not other people in the same reference class from your point of view as me weren’t signaling that they understood the point either).)

Steve_Rayhawk Jul 10, 2012, 4:34 PM
2 points
in reply to: private_messaging’s comment on: An Intuitive Explanation of Solomonoff Induction
I’m sorry; I was referring to what I had perceived as a general pattern, from seeing snippets of discussions involving you while I was lurking off-and-on. The “pre-emptive” was meant to refer to within single exchanges, not to refer all the way back to (in this case) the original discussion about MWI (which I’m still hunting down). Now that I look more closely at your history, this has only been at all frequent within the past few months.

I don’t have any specific recollection of you from before that comment on the “detecting rationalization” post, but looking back through your comment history of that period, I’m mystified by what happened there too. It’s possible that someone thought you were Thomas covertly giving a self-justifying speech about a random red-herring explanation he’d invented to tell himself for why other people disagreed with him, and they wished to discourage him from thinking other people agreed with that explanation.

Steve_Rayhawk Jul 10, 2012, 1:57 PM
1 point
in reply to: private_messaging’s comment on: An Intuitive Explanation of Solomonoff Induction
You discussed this over here too, with greater hostility:

also someone somehow thinks that Solomonoff induction finds probabilities for theories, while it just assigns 2^-length as probability for software code of such length, which is obviously absurd when applied to anything but brute force generated shortest pieces of code,

I’m trying to figure out whether, when you say “someone”, you mean someone upthread or one of the original authors of the post. Because if it’s the post authors, then I get to accuse you of not caring enough about refraining from heaping abuse on writers who don’t deserve it to actually bother to check whether they made the mistake you mocked them for. The post includes discussion of ideals vs. approximations in the recipe metaphor and in the paragraph starting “The actual process above might seem a little underwhelming...”:

We just check every single hypothesis? Really? Isn’t that a little mindless and inefficient? This will certainly not be how the first true AI operates. But don’t forget that before this, nobody had any idea how to do ideal induction, even in principle. Developing fundamental theories, like quantum mechanics, might seem abstract and wasteful. [...] such theories and models change the world, as quantum mechanics did with modern electronics. In the future, [...] ways to approximate Solomonoff induction [...] Perhaps they will develop methods to eliminate large numbers of hypotheses all at once. Maybe hypotheses will be broken into distinct classes. Or maybe they’ll use methods to statistically converge toward the right hypotheses.

Everyone knows there are practical difficulties. Some people expect that someone will almost certainly find a way to mitigate them. You tend to treat people as idiots for going ahead with the version of the discussion that presumes that.

With respect to your objections about undiscoverable short programs, presumably people would look for ways to calibrate the relationship between the description lengths of the theories their procedures do discover (or the lengths of the parts of those theories) and the typical probabilities of observations, perhaps using toy domains where everything is computable and Solomonoff probabilities can be computed exactly. Of course, by diagonalization there are limits to the power of any finite such procedure, but it’s possible to do better than assuming that the only programs are the ones you can find explicitly. Do you mean to make an implicit claim by your mockery which special-cases to the proposition that what makes the post authors deserve to be mocked is that they didn’t point out ways one might reasonably hope to get around this problem, specifically? (Or, alternatively, is the cause of your engaging in mockery something other than a motivation to express propositions (implicit or explicit) that readers might be benefited by believing? For example, is it a carelessly executed side-effect of a drive to hold your own thinking to high standards, and so to avoid looking for ways to make excuses for intellectual constructs you might be biased in favor of?)

Steve_Rayhawk Jul 10, 2012, 9:24 AM
30 points
in reply to: Kawoomba’s comment on: An Intuitive Explanation of Solomonoff Induction
He identifies subtleties, but doesn’t look very hard to see whether other people could have reasonably supposed that the subtleties resolve in a different way than he thinks they “obviously” do. Then he starts pre-emptively campaigning viciously for contempt for everyone who draws a different conclusion than the one from his analysis. Very trigger-happy.

This needlessly pollutes discussion… that is to say, “needless” in the moral perspective of everyone who doesn’t already believe that most people who first appear wrong by that criterion that way in fact are wrong, and negligently and effectively incorrigibly so, such that there’d be nothing to lose by loosing broadside salvos before the discussion has even really started. (Incidentally, it also disincentivizes the people who could actually explain the alternative treatment of the subtleties from engaging with him, by demonstrating a disinclination to bother to suppose that their position might be reasonble.) This perception of needlessness, together with the usual assumption that he must already be on some level aware of other peoples’ belief in that needlessness but is disregarding that belief, is where most of the negative affect toward him comes from.

Also, his occasional previous lack of concern for solid English grammar didn’t help the picture of him as not really caring about the possibility that the people he was talking to might not deserve the contempt for them that third parties would inevitably come away with the impression that he was signaling.

(I wish LW had more people who were capable of explaining their objections understandably like this, instead of being stuck with a tangle of social intuitions which they aren’t capable of unpacking in any more sophisticated way than by hitting the “retaliate” button.)

Steve_Rayhawk Jun 8, 2012, 10:06 PM
5 points
on: Thoughts and problems with Eliezer’s measure of optimization power
A concept I’ve played with, coming off of Eliezer’s initial take on the problem of formulating optimization power, is: Suppose something generated N options randomly and then chose the best. Given the observed choice, what is the likelihood function for N?

For continuously distributed utilities, this can be computed directly using beta distributions. Beta(N, 1) is the probability density for the highest of N uniformly distributed unit random numbers. This includes numbers which are cumulative probabilities for a continuous distribution at values drawn from that distribution, and therefore numbers which are cumulative probabilities at the goodness of an observed choice. (N doesn’t have to be an integer, because beta distributions are defined for non-integer parameters.)

(The second step of this construction, where you attach a beta distribution to another distribution’s CDF, I had to work out by myself; it’s not directly mentioned in any discussions of extreme value statistics that I could find. The Mathworld page on order statistics, one step of generalization away, uses the formula for a beta CDF transformed by another CDF, but it still doesn’t refer to beta distributions by name.)

If the utilities are discretely distributed, you have to integrate the beta density over the interval of cumulative probabilities that invert to the observed utility.

To handle choices of mixtures, I guess you could modify this slightly, and ask about the likelihood function for N given the observed outcome, marginalizing over (as User:Kindly also suggests) the (unobserved) choice of option. This requires a distribution over options and a conditional distribution over observations given options. This would also cover situations with composite options where you only observe one of the aspects of the chosen option.

Opposed optimization might be very crudely modeled by increasing the number on the opposite side of the beta distribution from N. Somewhat related to this is Warren Smith’s “Fixed Point for Negamaxing Probability Distributions on Regular Trees”, which examines the distributions of position values that result when two opponents take turns choosing the worst option for each other.

Alternatively, instead of a likelihood function for N, you could have a likelihood function for an exponential weighting λ on the expected utility of the option:

Pr(A was chosen)/Pr(B was chosen) ∝ exp(λ(U(A)-U(B))).

Higher values of λ would be hypotheses in which better options were more strongly randomly selected over worse ones. (This is something like a logit-response model, for which λ (or “β”, or “1/μ”) would be the “rationality” parameter. It might be more familiar as a Gibbs distribution.) But this would fail when the expected utility from the null distribution was heavy-tailed, because then for some λ≠0 the distribution of optimized expected utilities would be impossible to normalize. Better would be for the cumulative probability at the expected utility of the chosen option to be what was exponentially weighted by λ. In that case, in the limit λ = (N-1) >> 1, the two models give the same distribution.

All of these statistics, as well as Eliezer’s original formulation, end up encoding equivalent information in the limit where the utility of each option is an independent sum of many identical light-tailed-distributed components and you’re predicting a marginal distribution of utilities for one of those components. In this limit you can safely convert everything to a statistical mechanics paradigm and back again.

Of course, the real criterion for a good formulation of optimization power is whether it helps people who use it in an argument about things that might be optimizers, or who hear it used in such an argument, to come to truthful conclusions.

In this respect, likelihood functions can have the problem that most people won’t want to use them: they’re hard to compute with, or communicate, unless they belong to a low-dimensional family. The likelihood functions I suggested won’t do that except under very simple conditions. I’m not sure what the best way would be to simplify them to something lower-dimensional. I guess you could just communicate a maximum-likelihood estimate and precision for the optimization power parameter. Or, if you chose a reference prior over optimization power, you could communicate its posterior mean and variance.

All of this presupposes that the problems with the unoptimized probability measure can be dealt with. Maybe it would work better to describe the optimization power of a system in terms of a series of levels of simpler systems leading up to that system, where each level’s new amount of optimization was only characterized approximately, and only relative to something like a distribution of outputs from the previous level. (This would at least patch the problem where, if thermodynamics is somehow involved in the results of an action, that action can count as very powerful relative to the uniform measure over the system’s microstates.) If optimizer B is sort of like choosing the best of N options generated by optimizer A, and optimizer C is sort of like choosing the best of M options generated by optimizer B, that might not have to mean that optimizer C is much like choosing the best of N*M options generated by optimizer A.

Steve_Rayhawk Mar 7, 2012, 7:22 PM
15 points
in reply to: wedrifid’s comment on: AI Risk and Opportunity: A Strategic Analysis
That said, I think his fear of culpability (for being potentially passively involved in an existential catastrophe) is very real. I suspect he is continually driven, at a level beneath what anyone’s remonstrations could easily affect, to try anything that might somehow succeed in removing all the culpability from him. This would be a double negative form of “something to protect”: “something to not be culpable for failure to protect”.

If this is true, then if you try to make him feel culpability for his communication acts as usual, this will only make his fear stronger and make him more desperate to find a way out, and make him even more willing to break normal conversational rules.

I don’t think he has full introspective access to his decision calculus for how he should let his drive affect his communication practices or the resulting level of discourse. So his above explanations for why he argues the way he does are probably partly confabulated, to match an underlying constraining intuition of “whatever I did, it was less indefensible than the alternative”.

(I feel like there has to be some kind of third alternative I’m missing here, that would derail the ongoing damage from this sort of desperate effort by him to compel someone or something to magically generate a way out for him. I think the underlying phenomenon is worth developing some insight into. Alex wouldn’t be the only person with some amount of this kind of psychology going on—just the most visible.)

Steve_Rayhawk Mar 4, 2012, 11:44 AM
15 points
in reply to: XiXiDu’s comment on: AI Risk and Opportunity: A Strategic Analysis
Currently you suspect that there are people, such as yourself, who have some chance of correctly judging whether arguments such as yours are correct, and of attempting to implement the implications if those arguments are correct, and of not implementing the implications if those arguments are not correct.

Do you think it would be possible to design an intelligence which could do this more reliably?

Steve_Rayhawk Mar 4, 2012, 9:49 AM
5 points
in reply to: Will_Newsome’s comment on: People who “don’t rationalize”? [Help Rationality Group figure it out]
I wish there was a more standard term for this than “kinesthetic thinking”, that other people would be able to look up and understand what was meant.

(A related term is “motor cognition”, but that doesn’t denote a thinking style. Motor cognition is a theoretical paradigm in cognitive psychology, according to which most cognition is a kind of higher-order motor control/planning activity, connected in a continuous hierarchy with conventional concrete motor control and based on the same method of neural implementation. (See also: precuneus (reflective cognition?); compare perceptual control theory.) Another problem with the term “motor cognition” is that it doesn’t convey the important nuance of “higher-order motor planning except without necessarily any concurrent processing of any represented concrete motions”. (And the other would-be closest option, “kinesthetic learning”, actively denotes the opposite.)

Plausibly, people could be trained to introspectively attend to the aspect of cognition which was like motor planning with a combination of TCMS, to inhibit visual and auditory imagery, and cognitive tasks which involved salient constraints and tradeoffs. Maybe the cognitive tasks would also need to have specific positive or negative consequences for apparent execution of recognizable scripts of sequential actions typical of normally learned plans for the task. Some natural tasks, which are not intrinsically verbal or visual, with some of these features would be social reasoning, mathematical proof planning, or software engineering.)

when I am thinking kinesthetically I basically never rationalize as such

I think kinesthetic thinking still has things like rationalization. For example, you might have to commit to regarding a certain planned action a certain way as part of a complex motivational gambit, with the side effect that you commit to pretend that the action will have some other expected value than the one you would normally assign. If this ability to make commitments that affect perceived expected value can be used well, then by default this ability is probably also being used badly.

Could you give more details about the things like rationalization that you were thinking of, and what it feels like deciding not to do them in kinesthetic thinking?
What links here?
- Will_Newsome's comment on Rationality Quotes June 2013 by Thomas (Jun 10, 2013, 12:19 PM; 14 points)

Steve_Rayhawk Jan 19, 2012, 7:27 AM
9 points
on: Q&A with Abram Demski on risks from AI

If the human-level AGI

0) is autonomous (has, or forms, long-term goals)
1) is not socialized

#1 is important because a self-modifying system will tend to respond to negative reinforcement concerning sociopathic behaviors resulting from #3-- though, it must be admitted, this will depend on how deeply the ability to self-modify runs. Not all architectures will be capable of effectively modifying their goals in response to social pressures. (In fact, rigid goal-structure under self-modification will usually be seen as an important design-point.)

Abram: Could you make this more precise?

From the way you used the concept of “negative reinforcement”, it sounds like you have a particular family of agent architectures in mind, which is constrained enough that we can predict that the agent will make human-like generalizations about a reward-relevant boundary between “sociopathic” behaviors and “socialized” ones. It also sounds like you have a particular class of possible emergent socially-enforced concepts of “sociopathic” in mind, which is constrained enough that we can predict that the behaviors permitted by that sociopathy concept wouldn’t still be an existential catastrophe from our point of view. But you haven’t said enough to narrow things down that much.

For example, you could have two initially sociopathic agents, who successfully manipulate each other into committing all their joint resources to a single composite agent having a weighted combination of their previous utility functions. The parts of this combined agent would be completely trustworthy to each other, so that the agents could be said to have modified their goals in response to the “social pressures” of their two-member society. But the overall agent could still be perfectly sociopathic toward other agents who were not powerful enough to manipulate it into making similar concessions.

Steve_Rayhawk Jan 10, 2012, 1:22 PM
2 points
in reply to: Manfred’s comment on: Q&A with experts on risks from AI #2
In fact, I’d prefer it if Q8 started out with the less-shibbolethy “How much have you read about, or used the concepts of...” or something like that, which replaces a dichotomy with a continuum.

Yeah… I wanted to make the suggested question less loaded, but it would have required more words, and I was unthinkingly preoccupied with worry about a limit on the permitted complexity of a single-sentence question. Maybe I should have split the question across more sentences.

The signaling uses of Q8 seem like a bad idea to me, although it seems a worthwhile thing to ask for Steve Rayhawk’s reasons.

My reasons for suggesting Q8 were mostly:
- First, I wanted to make it easier to narrow down hypotheses about the relationship between respondents’ opinions about AI risk and their awareness of progress toward formal, machine-representable concepts of optimal AI design (also including, I guess, progress toward practically efficient mechanized application of those concepts, as in Schmidhuber’s Speed Prior and AIXI-tl).
- Second, I was imagining that many respondents would be AI practitioners who thought mostly in terms of architectures with a machine-learning flavor. Those architectures usually have a very specific and limited structure in their hypothesis space or policy space by construction, such that it would be clearly silly to imagine a system with such an architecture self-representing or self-improving. These researchers might have a conceptual myopia by which they imagine “progress in AI” to mean only “creation of more refined machine-learning-style architectures”, of a sort which of course wouldn’t lead towards passing a threshold of capability of self-improvement anytime soon. I wanted to put in something of a conceptual speed bump to that kind of thinking, to reduce unthinking dismissiveness in the answers, and counter part of the polarizing/consistency effects that merely receiving and thinking about answering the survey might have on the recipients’ opinions. (Of course, if this had been a survey which were meant to be scientific and formally reportable, it would be desirable for the presence of such a potentially leading question to be an experimentally controlled variable.)
With those reasons on the table, someone else might be able to come up with a question that fulfills them better. I also agree with paulfchristiano’s comment.

Steve_Rayhawk Jan 9, 2012, 6:32 AM
2 points
in reply to: Steve_Rayhawk’s comment on: Q&A with experts on risks from AI #1

However, it’s harder to find uncontroversial questions which would be diagnostic of these errors.

Perhaps an expert’s beliefs about the costs of better information and the costs of delay might be assessed with a willingness-to-pay question, such as a tradeoff involving a hypothetical benefit to everyone now living on Earth which could be sacrificed to gain hypothetical perfect understanding of some technical unknowns related to AI risks, or a hypothetical benefit gained at the cost of perfect future helplessness against AI risks. However, even this sort of question might seem to frame things hyperbolically.

Steve_Rayhawk Jan 9, 2012, 6:19 AM
20 points
on: Q&A with experts on risks from AI #1
I think experts’ opinions on the possibility of AI self-improvement may covary with their awareness of work on formal, machine-representable concepts of optimal AI design, particularly Solomonoff induction, including its application to reinforcement learning as in AIXI, and variations of Levin search such as Hutter’s algorithm M and Gödel machines. If an expert is unaware of those concepts, this unawareness may serve to explain away the expert’s belief that there are no approaches to engineering self-improvement-capable AI on any foreseeable horizon.

If it’s not too late, you should probably include a question to judge the expert’s awareness of these concepts in your questionnaires, such as:

“Qn: Are you familiar with formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter’s algorithm M, AIXI, or Gödel machines?”

...bearing in mind that the presence of such a question may affect their other answers.

(This was part of what I was getting at with my analysis of the AAAI panel interim report: “What cached models of the planning abilities of future machine intelligences did the academics have available [...]?” “What fraction of the academics are aware of any current published AI architectures which could reliably reason over plans at the level of abstraction of ‘implement a proxy intelligence’?”)

Other errors which might explain away an expert’s unconcern for AI risk are:
- incautious thinking about the full implications of a given optimization criterion or motivational system;
- when considering AI self-improvement scenarios, incautious thinking about parameter uncertainty and structural uncertainty in economic descriptions of computational complexity costs and efficiency gains over time (particularly given that a general AI will be motivated to investigate many different possible structures for the process for self-improvement, including structures one may not oneself have considered, in order to choose a process whose economics are as favorable as possible); and
- incomplete reasoning about options for gathering information about technical factors affecting AI risk scenarios, when considering the potential relative costs of delaying AI safety projects until better information is available (on the implicit expectation that, in the event that the technical factors turn out to imply safety, delaying will have prevented the cost of the AI safety projects, and (more viscerally) that having advocated delay will prevent one’s own loss of prestige, unthinkingly taken as a proxy for correctness, whereas failure to have advocated an immediate start to AI safety projects could not result in loss of one’s own prestige in any event).
However, it’s harder to find uncontroversial questions which would be diagnostic of these errors.
What links here?
- XiXiDu's comment on Q&A with experts on risks from AI #2 by XiXiDu (Jan 9, 2012, 8:08 PM; 14 points)
- [Template] Questions regarding possible risks from artificial intelligence by XiXiDu (Jan 10, 2012, 11:59 AM; 10 points)

Steve_Rayhawk Dec 30, 2011, 5:12 AM
4 points
in reply to: Mitchell_Porter’s comment on: Welcome to Less Wrong!
I’ll stake $500 if eligible.

When would the answer need to be known by?

Steve_Rayhawk Dec 10, 2011, 7:17 AM
0 points
on: AIXI and Existential Despair
[...] assume that the universe and its initial conditions can be described succinctly and inferred by A, and that the sequence of bits sent over W1 and W2 can be defined using an additional 10000 bits once a description of the universe is in hand.

Do you mean, “once a full description of the universe is in hand”, and that the 10000 bits are the complexity of locating W1 and W2 in the full description?
- A’s outputs are fed to the output wire W2, the rest of the universe (including A itself) behaves according to physical law, and A is given the values from input wire W1 as its input. (Model 1)
- A’s outputs are ignored, the rest of the universe behaves according to physical law, and A is given the values from W1 as its input. (Model 2)
Model 2 still has to locate W1. What might be the complexity of locating W2 conditional on having located W1? It seems plausible that this extra complexity would be within a constant of the complexity of describing the AIXI algorithm and something like counterfactual physical reasoning about alternative inputs, to check whether the known behavior of AIXI is consistent with W2 given counterfactual input at W1.

Steve_Rayhawk Nov 30, 2011, 6:43 PM
8 points
in reply to: brilee’s comment on: Log-odds (or logits)
The natural unit of ratio, the neper (Np), is easier to interpret for small ratio contributions, where the derivative of exp(x) is ≈1:

0.1Np = exp( 0.1) ∶ 1 ≈ 1.1 ∶ 1
-0.1Np = exp(-0.1) ∶ 1 ≈ 0.9 ∶ 1

This could make for an easy upgrade path to use of nepers or centinepers instead of percents in comparatives involving rates, which would reduce semantic confusion. “50% faster” can mean “gets 150% as far” (so .41Np faster, or 41 cNp, or perhaps 41Np%) or “takes 50% as much time” (so .69Np faster, or 69cNp, or 69Np%). That’s an argument for using nepers as a standard base outside communications of probability.

(trivia: Nepers and radians are each other turned sideways, being respectively the real and imaginary parts of eigenvalues of linear differential equation systems.)

Steve_Rayhawk Nov 7, 2011, 2:15 PM
8 points
on: AI Challenge: Ants
I would not expect to win without a method that I could see to approximate or to exceed linear-programming formulations of multi-agent path planning, for example here (2005/2008) or here (2001). Those methods don’t fully cover the stochasticity of resource gathering or the adversariality of combat, but they partly cover resource-finding and patrolling (when up-to-date visual information about unknowns is an objective) and deployment, and they provide a framework into which to fit tradeoffs among options evaluated by more specialized models, such as for combat, exploration, or gathering.

One approximation would be a relaxation) of the ant indicator function to a basis) of wavelet-like vectors (for example diffusion kernels) on the graph of connected land squares. (The basis vectors should overlap smoothly, because under a rigorous relaxation, non-overlapping basis vectors would allow approximated ants to teleport across the support) of basis elements.)
What links here?
- gwern's comment on AI Challenge: Ants—Post Mortem by D_Alex (Jan 14, 2012, 7:29 PM; 2 points)

Steve_Rayhawk Sep 19, 2011, 7:46 AM
4 points
in reply to: Vladimir_Nesov’s comment on: The Cognitive Science of Rationality
Related: Ken Thompson’s “Reflections on Trusting Trust” (his site, Wikipedia#Reflections_on_Trusting_Trust)), Richard Kennaway’s comment on “The Finale of the Ultimate Meta Mega Crossover”.

Steve_Rayhawk May 22, 2011, 6:02 AM
15 points
in reply to: PhilGoetz’s comment on: Rationalists don’t care about the future

This is a critical topic, but LessWrong hates it. Matthew 7:6 comes to mind.

For the record, I dispute your causal model of the audience’s response.

In particular, I dispute your model of the audience’s moral reasoning as to what is inevitably being approved of or disapproved of by expressions of approval or disapproval of your actions relating to the post.

I also dispute your model of the audience’s factual and moral reasoning about the gravity of the problem you suggest. I dispute specifically your model of the audience’s process of choosing to suppose that non-exponential weighting functions could be considered sufficiently indicative of potential solutions as to justify relative unconcern. (This is because I dispute your model of the utility function structures initially familiar to the audience. As part of this, I dispute your model of their descriptions of discounting functions, according to which it apparently would be impossible for them to intend to refer to a function which was to be applied on a prespecified absolute timescale, without being translated to start at an agent’s present time. If that was not your model, then I dispute your confusing apparent claim that such functions, if non-exponential, must be dynamically inconsistent.)

I am concerned that the errors in your model of the audience, if left unchallenged, will only serve to reinforce in you the apparent resentful, passive-aggressive self-righteousness which would have largely been itself the cause of the misinterpretations which led to those errors originally. This self-reinforcing effect might create needless mutual epistemic alienation.

Steve_Rayhawk May 19, 2011, 7:04 AM
15 points
on: Rationalists don’t care about the future
You keep constructing scenarios whose intent, as far as I can tell, is to let you argue that in those scenarios any currently imaginable non-human system would be incapable of choosing a correct or defensible course of action. By comparison, however, you must also be arguing that some human system in each of those scenarios would be capable of choosing a correct or defensible course of action. How?

And: Suppose you knew that someone was trying to understand the answer to this question, and create the field of “Artificial Ability to Choose Correct and Defensible Courses of Action The Way Humans Apparently Can”. What kinds of descriptions do you think they might give of the engineering problem at the center of their field of study, of their criteria for distinguishing between good and bad ways of thinking about the problem, and of their level of commitment to any given way in which they’ve been trying to think about the problem? Do those descriptions differ from Eliezer’s descriptions regarding “Friendly AI” or “CEV”?

You seem to be frustrated about some argument(s) and conclusions that you think should be obvious to other people. The above is an explanation of how some conclusions that seem obvious to you could seem not obvious to me. Is this explanation compatible with your initial model of my awareness of arguments’ obviousnesses?

Steve_Rayhawk May 15, 2011, 9:31 PM
15 points
in reply to: PhilGoetz’s comment on: Rationalists don’t care about the future
Any temporal discounting other than temporal is provably inconsistent

The conditions of the proof are applicable only to reinforcement agents which, as a matter of architecture, are forced to integrate anticipated rewards using a fixed weighting function whose time axis is constantly reindexed to be relative to the present. If we could self-modify to relax that architectural constraint—perhaps weighting according to some fixed less temporally indexical schedule, or valuing something other than weighted integrals of reward—would you nonetheless hold that rational consistency would require us to continue to engage in exponential temporal discounting? Whether or not the architectural constraint had previously been a matter of choice? (And who would be the “us” who would thus be required by rational consistency, so that we could extract a normative discount rate from them? Different aspects of a person or civilization exhibit discount functions with different timescales, and our discount functions and architectural constraints can themselves partially be traced to decision-like evolutionary and ecological phenomena in the biosphere, whose “reasoning” we may wish to re-examine.)

(ETA: Maybe I should be less uncharitable about your implied position, since you may not have been aware of the conditions of the proof you cited, or not thought to consider a wider range of agent motivational architectures. But if that was the sort of thing you didn’t know, and it was crucial to your original case, you should have known to state your case in more measured and careful language. If you commit strongly to a hostile conclusion that seems unjustifiable, I unthinkingly respond by exploiting the unjustifiability and strength of commitment to make the hostile conclusion look bad, using lines of modus tollens reasoning that wouldn’t be able to rhetorically connect if your commitment had been weaker.)

To my current thinking, preferences would be one form of information about desirability of events, and any information about desirability of events would be timeless—even if the events that were desirable were within time, and even if the information about their desirability must have been acquired within time. There’s no direct reason why questions of “when you learned about the desirability” or “when you had to act on the desirability” should enter into it.

Why were you so certain [...] that exponential temporal discounting behavior was a matter of intrinsic value rather than instrumental value [...]?

Perhaps I should have left out the distraction of the term “exponential”, and asked: “Why were you so certain that temporal discounting in behavior was a matter of intrinsic value rather than instrumental value?” In part my comment was to argue that:
- discounting behavior can be generated for instrumental reasons;
- we may reach different conclusions as to whether discounting behavior is a matter of intrinsic or instrumental value, depending on the level of analysis at which we identify agency (and/or instrumental agency);
- there are reasons to expect that, in interpreting utility functions from preference claims, we may easily become confused and inappropriately assign intrinsicality to values or rules of valuation which were actually instrumental.
I should have argued more explicitly that:
- Instrumental exponential discounting is conditional, not eternal; it lasts only as long as the exponentially growing opportunity costs which motivate it.
- Inappropriate hypotheses of intrinsicality of values can lead to paradoxes, which the corresponding hypotheses of instrumentality may avoid. This is because instrumental values have effect conditionally while intrinsic values have effect unconditionally. Thus, if you observe an apparent paradox during an analysis that assumes intrinsicality, you should put more weight on competing analyses that assume instrumentality, on the theory that you missed a relevant condition which prevents a conflicting value from extending to the paradoxical case.
(My argument was meant to cover non-exponential discounting as well, and show that exponential discounting behavior can be caused by a same mechanism as non-exponential discounting behavior, since I did not specify that market rates of return were constant.)

My comment was also to argue that we are simply confused about the right way to extract utility functions from information about behavior or reported preferences, and therefore that apparent paradoxes do not necessarily mean that the premises are wrong which they appear to mean are wrong.