@Roland
I suppose you could google “(arrogant OR arrogance OR modesty) eliezer yudkowsky” and have plenty to digest. Note that the arrogance at issue is neither dishonest nor unwarranted, but it is an impairment, and a consequence of trade-offs which, from within a broader context, probably wouldn’t be taken in the same way.
That’s as far as I’m willing to entertain this line of inquiry, which ostensibly neutral request for facts appears to belie an undercurrent of offense.
Eliezer, I’ve been watching you with interest since 1996 due to your obvious intelligence and “altruism.” From my background as a smart individual with over twenty years managing teams of Ph.D.s (and others with similar non-degreed qualifications) solving technical problems in the real world, you’ve always struck me as near but not at the top in terms of intelligence. Your “discoveries” and developmental trajectory fit easily within the bounds of my experience of myself and a few others of similar aptitudes, but your (sheltered) arrogance has always stood out. I wish you continued progress, not so much in ever-sharper analysis, but in ever more effective synthesis of the leading-edge subjects you pursue.
I see this discussion over the last several months bouncing around, teasingly close to a coherent resolution of the ostensible subjective/objective dichotomy applied to ethical decision-making. As a perhaps pertinent meta-observation, my initial sentence may promulgate the confusion with its expeditious wording of “applied to ethical decision-making” rather than a more accurate phrasing such as “applied to decision-making assessed as increasingly ethical over increasing context.”
Those who in the current thread refer to the essential element of empathy or similarity (of self models) come close. It’s important to realize that any agent always only expresses its nature within its environment—assessments of “rightness” arise only in the larger context (of additional agents, additional experiences of the one agent, etc.)
Our language and our culture reinforce an assumption of an ontological “rightness” that pervades our thinking on these matters. An even greater (perceived) difficulty is that to relinquish ontological “rightness” entails ultimately relinquishing an ontological “self”. But to relinquish such ultimately unfounded beliefs is to gain clarity and coherence while giving up nothing actual at all.
“Superrationality” is an effective wrapper around these apparent dilemmas, but even proponents such as Hofstadter confused description with prescription in this regard. Paradox is always only a matter of insufficient context. In the bigger picture all the pieces must fit. [Or as Eliezer has taken to saying recently: “It all adds up to normalcy.”
Apologies if my brief pokings and proddings on this topic appear vague or even mystical. I can only assert within this limited space and bandwidth that my background in science, engineering and business is far from that of one who could harbor vagueness, relativism, mysticism, or postmodernist patterns of thought. I appreciate the depth and breadth of Eliezer’s written explorations of this issue whereas I lack the time to do so myself.
Watching the ensuing commentary, I’m drawn to wishfully imagine a highly advanced Musashi, wielding his high-dimensional blade of rationality such that in one stroke he delineates and separates the surrounding confusion from the nascent clarity. Of course no such vorpal katana could exist, for if it did, it would serve only to better clear the way for its successors.
I see a preponderance of viewpoints representing, in effect, the belief that “this is all well and good, but how will this guide me to the one true prior, from which Archimedian point one might judge True Value?”
I see some who, given a method for reliably discarding much which is not true, say scornfully in effect “How can this help me? It says nothing whatsoever about Truth itself!”
And then there are the few who recognize we are each like leaves of a tree rooted in reality, and while we should never expect exact agreement between our differing subjective models, we can most certainly expect increasing agreement—in principle—as we move toward the root of increasing probability, pragmatically supporting, rather than unrealistically affirming, the ongoing growth of branches of increasing possibility. [Ignoring the progressive enfeeblement of the branches necessitating not just growth but eventual transformation.]
Eliezer, I greatly appreciate the considerable time and effort you must put into your essays. Here are some suggested topics that might help reinforce and extend this line of thought:
Two communities, separated by a chasm Would it be seen as better (perhaps obviously) to build a great bridge between them, or to consider the problem in terms of an abstract hierarchy of values, for example involving impediments to transfer of goods, people, … ultimately information, for which building a bridge is only a special-case solution? In general, is any goal not merely a special case (and utterly dependent on its specifiability) of values-promotion?
Fair division, etc. Probably nearly all readers of Overcoming Bias are familiar with a principled approach to fair division of a cake into two pieces, and higher order solutions have been shown to be possible with attendant computational demands. Similarly, Rawles proposed that we ought to be satisfied with social choice implemented by best-known methods behind a veil of ignorance as to specific outcomes in relation to specific beneficiaries. Given the inherent uncertainty of specific future states within any evolving system of sufficient complexity to be of moral interest, what does this imply about shifting moral attention away from expected consequences, and toward increasingly effective principles reasonably optimizing our expectation of improving, but unspecified and indeed unspecifiable, consequences? Bonus question: How might this apply to Parfit’s Repugnant Conclusion and other well-recognized “paradoxes” of consequentialist utilitarianism?
Constraints essential for meaningful growth Widespread throughout the “transhumanist” community appears the belief that considerable, if not indefinite progress can be attained via the “overcoming of constraints.” Paradoxically, the accelerating growth of possibilities that we experience arises not with overcoming constraints, but rather embracing them in ever-increasing technical detail. Meaningful growth is necessarily within an increasingly constrained possibility space—fortunately there’s plenty of fractal interaction area within any space of real numbers—while unconstrained growth is akin to a cancer. An effective understand of meaningful growth depends on an effective understanding of the subjective/objective dichotomy.
Thanks again for your substantial efforts.
Eliezer, it’s a pleasure to see you arrive at this point. With an effective understanding of the subjective/objective aspects supporting a realistic metaethics, I look forward to your continued progress and contributions in terms of the dynamics of increasingly effective evolutionary (in the broadest sense) development for meaningful growth, promoting a model of(subjective) fine-grained, hierarchical values with increasing coherence over increasing context of meaning-making, implemts principles of (objective) instrumental action increasingly effective over increasing scope of consequences. Wash, rinse, repeat...
There’s no escape from the Red Queen’s race, but despite the lack of objective milestones or markers of “right”, there’s real progress to be made in the direction of increasing rightness.
Society has been doing pretty well at the increasingly objective model of instrumental action known commonly known as warranted scientific knowledge. Now if we could get similar focus on the challenges of values-elicitation, inductive biases, etc., leading to an increasingly effective (and coherent) model of agent values...
Jef
In other words, probability is not likelihood.
Anon wrote: “Any question of ethics is entirely answered by arbitrarily chosen ethical system, therefore there are no “right” or “better” answers.”
Matters of preference are entirely subjective, but for any evolved agent they are far from arbitrary, and subject to increasing agreement to the extent that they reflect increasingly fundamental values in common.
Once again we’ve highlighted the immaturity of present-day moral thinking—the kind that leads inevitably to Parfit’s Repugnant Conclusion. But any paradox is merely a matter of insufficient context; in the bigger picture all the pieces must fit.
Here we have people struggling over the relative moral weight of torture versus dust specks, without recognizing that there is no objective measure of morality, but only objective measures of agreement on moral values.
The issue at hand can be modeled coherently in terms of the relevant distances (regardless of how highly dimensional, or what particular distance metric) between the assessor’s preferred state and the assessor’s perception of the alternative states. Regardless of the particular (necessarily subjective) model and evaluation function, there must be some scalar distance between the two states within the assessor’s model (since a rational assessor can have only a single coherent model of reality, and the alternative states are not identical.) Furthermore introducing a multiplier on the order of a googolplex overwhelms any possible scale in any realizable model, leading to an effective infinity, forcing one (if one’s reasoning is to be coherent) to view that state as dominant.
All of this (as presented by Eliezer) is perfectly rational—but merely a special case and inappropriate to decision-making within a complex evolving context where actual consequences are effectively unpredictable.
If one faces a deep and wide chasm impeding desired trade with a neighboring tribe, should one rationally proceed to achieve the desired outcome: an optimum bridge?
Or should one focus not on perceived outcomes, but rather on most effectively expressing one’s values-complex: Ie, valuing not the bridge, but effective interaction (including trade), and proceeding to exploit best-known principles promoting interaction, for example communications, air transport, replication rather than transport...and maybe even a bridge?
The underlying point is that within a complex evolutionary environment, specific outcomes can’t be reliably predicted. Therefore to the extent that the system (within its environment of interaction) cannot be effectively modeled, an optimum strategy is one that leads to discovering the preferred future through the exercise of increasingly scientific (instrumental) principles promoting an increasingly coherent model of evolving values.
In the narrow case of a completely specified context, it’s all the same. In the broader, more complex world we experience, it means the difference between coherence and paradox.
The Repugnant Conclusion fails (as does all consequentialist ethics when extrapolated) because it presumes to model a moral scenario incorporating an objective point of view. Same problem here.
“I think people would be more comfortable with your conclusion if you had some way to quantify it; right now all we have is your assertion that the math is in the dust speck’s favor.”
The actual tipping point depends on your particular subjective assessment of relative utility. The actual tipping point doesn’t matter; what matters is that there is crossover at some point, therefore such reasoning about preferences, like San Jose --> San Francisco --> Oakland --> San Jose is incoherent.
This form of reasoning, while correct within a specified context, is dangerously flawed with regard to application within contexts sufficiently complex that outcomes cannot be effectively modeled. This includes much of moral interest to humans. In such cases, as with evolutionary computation, an optimum strategy exploits best-known principles synergisticly promoting a maximally-coherent set of present values, rather than targeting illusory, realistically unspecifiable consequences. Your “rationality” is correct but incomplete. This speaks as well to the well-known paradoxes of all consequentialist ethics.
I see the importance of context generally overlooked in this discussion, as in most discussion of rationality (encompassing discussion of rational methods.) The elegance and applicability of Bayesian inference is to me undeniable, but I look forward to broader discussion of its application within systems of effective decision-making entailing prediction within a context which is not only uncertain but evolving. In other words, consideration of principles of effective agency where the game itself is inherently uncertain.
In doing so, I think we are driven to re-frame our thinking away from goals to be achieved and onto values to be promoted, away from rules to be followed and onto increasingly coherent principles to be applied, and away from maximizing expected utility and onto maximizing potential synergies within a game that is consistent but inherently incomplete. I see Bayes as necessary but not sufficient for this more encompassing view.
A good post on a profoundly beautiful subject, and a nice bit of jujutsu the way it works against the backdrop of the meta-commentary.
A minor quibble: Have you considered that use of “Law”, like “The Way”, while perhaps appropriately elevating, might work against your message by obscuring appreciation of increasingly general “principles”?
I simply want to express my great appreciation for Eliezer’s substantial efforts to share his observations of the journey, his willingness (in principle) to update his beliefs, and his presently ongoing integration of the epistemologically undeniable “subjective” with the hardcore reductionist “objective.” I’m joyfully anticipating what comes next!
“A brilliant idea, Jef! I volunteer you to test it out. Start blowing dust around your house today.”
Although only one person, I’ve already begun, and have entered in my inventor’s notebook some apparently novel thinking on not only dust, but mites, dog hair, smart eyedrops, and nanobot swarms!
With so many so deep in reductionist thinking, I’m compelled to stir the pot by asking how one justifies the assumption that the SPECK is a net negative at all, aggregate or not, extended consequences or not? Wouldn’t such a mild irritant, over such a vast and diverse population, act as an excellent stimulus for positive adaptations (non-genetic, of course) and likely positive extended consequences?
“Regarding (1), we pretty much always have excellent reason to mistrust our judgments, and then we have to choose anyway; inaction is also a choice. The null plan is a plan. As Russell and Norvig put it, refusing to act is like refusing to allow time to pass.”
This goes to the crux of the matter, why to the extent the future is uncertain, it is better to decide based on principles (representing wisdom encoded via evolutionary processes over time) rather than on the flat basis of expected consequences.
Eliezer wrote “Wow. People sure are coming up with interesting ways of avoiding the question.”
I posted earlier on what I consider the more interesting question of how to frame the problem in order to best approach a solution.
If I were to simply provide my “answer” to the problem, with the assumption that the dust in the eyes is likewise limited to 50 years, then I would argue that the dust is to be preferred to the torture, not on a utilitarian basis of relative weights of the consequences as specified, but on the bigger-picture view that my preferred future is one in which torture is abhorrent in principle (noting that this entails significant indirect consequences not specified in the problem statement.)
Fascinating, and scary, the extent to which we adhere to established models of moral reasoning despite the obvious inconsistencies. Someone here pointed out that the problem wasn’t sufficiently defined, but then proceeded to offer examples of objective factors that would appear necessary to evaluation of a consequentialist solution. Robin seized upon the “obvious” answer that any significant amount of discomfort, over such a vast population, would easily dominate, with any conceivable scaling factor, the utilitarian value of the torture of a single individual. But I think he took the problem statement too literally; the discomfort of the dust mote was intended to be vanishingly small, over a vast population, thus keeping the problem interesting rather than “obvious.”
But most interesting to me is that no one pointed out that fundamentally, the assessed goodness of any act is a function of the values (effective, but not necessarily explicit) of the assessor. And assessed morality as a function of group agreement on the “goodness” of an act, promoting the increasingly coherent values of the group over increasing scope of expected consequences.
Now the values of any agent will necessarily be rooted in an evolutionary branch of reality, and this is the basis for increasing agreement as we move toward the common root, but this evolving agreement in principle on the direction of increasing morality should never be considered to point to any particular destination of goodness or morality in any objective sense, for that way lies the “repugnant conclusion” and other paradoxes of utilitarianism.
Obvious? Not at all, for while we can increasingly converge on principles promoting “what works” to promote our increasingly coherent values over increasing scope, our expression of those values will increasingly diverge.
Addendum (Out of concern for appearing too vague and mystical):
What is interesting about a brick wall?
Is is in the texture of a brick or its color or its cracks? Is it in the alignment pattern of the rows, or the deviations within the rows? Is it in the statistical regularities of the rows and columns forming the wall? Is it in the demonstration of rectangular tiling in the plane, properties of containment and division? Is it in the ecological aspects of the wall in relation to it’s environment? …
What is interesting in Borges’s Library of Babel, in all it’s vast algorithmic complexity?
What’s commonly lacking from scientific accounts of the world is the essential role of the observer, not in the world itself, but in any accounting of it.
The benefit of staring at a wall is to become aware of the observer, beyond that it’s relatively pointless.
For me, a highlight of each year is a multi-day gathering of about 40 individuals selected for their intelligence, integrity and passion to make the world a better place. We share our current thinking and projects and actively refine and synergize plans for the year ahead. Nearly everyone there displays perceptiveness, creativity, joy of life, “sparkle”, well above the norm, but—these qualities are NOT highly predictive of effectiveness outside the individual’s preferred environment.