Johnicholas

Karma: 2,008

Johnicholas Jul 12, 2012, 1:35 PM
1 point
on: Reply to Holden on The Singularity Institute
Thanks for posting this!

I am also grateful to Holden for provoking this—as far as I can tell, the only substantial public speech from SIAI on LessWrong. SIAI often seems to be far more concerned with internal projects than communicating with its supporters, such as most of us on LessWrong.

Johnicholas Jun 14, 2012, 2:56 AM
5 points
in reply to: Eliezer Yudkowsky’s comment on: Reply to Holden on ‘Tool AI’
I don’t think Strange7 is arguing Strange7′s point strongly; let me attempt to strengthen it.

A button that does something dangerous, such as exploding bolts that separate one thing from another thing, might be protected from casual, accidental changes by covering it with a lid, so that when someone actually wants to explode those bolts, they first open the lid and then press the button. This increases reliability if there is some chance that any given hand motion is an error, but the errors of separate hand motions are independent. Similarly ‘are you sure’ dialog boxes.

In general, if you have several components, each of a given reliability, and their failure modes are somewhat independent, then you can craft a composite component of greater reliability than the individuals. The rings that Strange7 brings up are an example of this general pattern (there may be other reasons why layers-of-rings architectures are chosen for reliability in practice—this explanation doesn’t explain why the rings are ordered rather than just voting or something—this is just one possible explanation).

Johnicholas Jun 13, 2012, 12:15 PM
8 points
in reply to: Strange7’s comment on: Reply to Holden on ‘Tool AI’
The distinction between hardwiring and softwiring is, at above the most physical, electronic aspects of computer design, a matter of policy—something in the programmer’s mind and habits, not something out in the world that the programmer is manipulating. From any particular version of the software’s perspective, all of the program it is running is equally hard (or equally soft).

It may not be impossible to handicap an entity in some way analogous to your suggestion, but holding fiercely to the concept of hardwiring will not help you find it. Thinking about mechanisms that would accomplish the handicapping in an environment where everything is equally hardwired would be preferable.

There’s some evidence that chess AIs ‘personality’ (an emergent quality of their play) is related to a parameter of their evaluation function called ‘contempt’, which is something like (handwaving wildly) how easy the opponent is to manipulate. In general, AIs with higher contempt seek to win-or-lose more, and seek to draw less. What I’m trying to say is, your idea is not without merit, but it may have unanticipated consequences.

Johnicholas Jun 12, 2012, 10:23 AM
2 points
on: Reply to Holden on ‘Tool AI’
The thing that is most like an agent in the Tool AI scenario is not the computer and software that it is running. The agent is the combination of the human (which is of course very much like an agent) together with the computer-and-software that constitutes the tool. Holden’s argument is that this combination agent is safer somehow. (Perhaps it is more familiar; we can judge intention of the human component with facial expression, for example.)

The claim that Tool AI is an obvious answer to the Friendly AI problem is a paper tiger that Eliezer demolished. However, there’s a weaker claim, that SIAI is not thinking about Tool AI much if at all, and that it would be worthwhile to think about (e.g. because it already routinely exists), which Eliezer didn’t really answer.

Johnicholas May 16, 2012, 10:24 AM
1 point
on: General purpose intelligence: arguing the Orthogonality thesis
Minor text correction;

“dedicated committee of human-level AIs dedicated” repeats the same adjective in a small span.

More wide-ranging:

Perhaps the paper would be stronger if it explained why philosophers might feel that convergence is probable. For example, in their experience, human philosophers / philosophies converge.

In a society, where the members are similar to one another, and much less powerful than the society as a whole, the morality endorsed by the society might be based on the memes that can spread successfully. That is, a meme like ‘everyone gets a vote’ or the golden rule is a meme that is symmetrical in a way that ‘so-and-so gets the vote’ isn’t. The memes that spread successfully might be more likely to be symmetrical. There could be convergent evolution of memes, and this could explain human philosophers experience of convergence.

Johnicholas May 16, 2012, 10:08 AM
−2 points
in reply to: drnickbone’s comment on: General purpose intelligence: arguing the Orthogonality thesis
There was an incident of censorship by EY relating to acausal trading—the community’s confused response (chilling effects? agreement?) to that incident explains why there is no overall account.

Johnicholas May 12, 2012, 1:27 PM
8 points
in reply to: NancyLebovitz’s comment on: Thoughts on the Singularity Institute (SI)
There’s two uses of ‘utility function’. One is analogous to Daniel Dennett’s “intentional stance” in that you can choose to interpret an entity as having a utility function—this is always possible but not necessarily a perspicuous way of understanding an entity—because you might end up with utility functions like “enjoys running in circles but is equally happy being prevented from running in circles”.

The second form is as an explicit component within an AI design. Tool-AIs do not contain such a component—they might have a relevance or accuracy function for evaluating answers, but it’s not a utility function over the world.

Johnicholas Apr 30, 2012, 12:15 PM
0 points
in reply to: Hul-Gil’s comment on: Recognizing memetic infections and forging resistance memes
You’re right, it’s infeasible to care about individual memes (or for that matter, the vast majority of individual animals) the way we care about other humans. I don’t have an answer to your question, I’m trying to break a logjam of humancentric ethical thinking.

Forgive me for passing on my confusion here, but I’m not certain that consciousness/sentience, is anything more than ‘recognizably human’. You and I have a common brain architecture and one of our faculties is picking that out from trees and rocks. Perhaps there are plenty of evolved, competent alien minds that would pose a challenge to our ape-like-me recognition systems simply because they’re so alien.

But if and when humans upload, then they will become effectively memes. We need to solve the question of how to care about non-sentient life, because a datafile could be you or me or a descendent.

Johnicholas Apr 27, 2012, 2:03 PM
1 point
in reply to: JoachimSchipper’s comment on: Recognizing memetic infections and forging resistance memes
No. This is regarding the ‘possibly irrelevant rant’ which I marked explicitly as a ‘possibly irrelevant rant’. The concepts in the rant seemed nearby and inspirational to the main article in my mind when I wrote it, which is why I included it, but I cannot articulate a direct connection.

Johnicholas Apr 27, 2012, 11:38 AM
5 points
in reply to: faul_sname’s comment on: Recognizing memetic infections and forging resistance memes
Analogous in that people once discriminated against other races, other sexes, but over time with better ethical arguments, we decided it was better to treat other races, other sexes as worthy members of the “circle of compassion”. I predict that if and when we interact with another species with fairly similar might (for example if and when humans speciate) then humancentrism will be considered as terrible as racism or sexism is now.

Moral realism (if I understand it correctly) is the position that moral truths like ‘eating babies is wrong’ are out in the world something like the law of gravitation. Yudkowsky has argued convincingly in the Baby-Eater sequence against moral realism (and I agree with him). However, he implied a false fork that, if moral realism is false, then humancentrism is the answer. Yes, our sense of morality is based on our history. No, our history is not the same as our species.

DNA is one residue of our history, but libraries are also a similar residue. There are two instances in our history of allying with a very alien form of life: Viral eukaryogenesis, and the alliance with memes.

Does this help at all? I feel like I’m saying the same thing over again just with more words.

Johnicholas Apr 26, 2012, 9:29 PM
0 points
in reply to: JGWeissman’s comment on: Recognizing memetic infections and forging resistance memes
Thank you. Exactly.

Johnicholas Apr 26, 2012, 9:27 PM
−1 points
in reply to: Vaniver’s comment on: Recognizing memetic infections and forging resistance memes
Thanks, I’ll replace the sentence with “Humancentrism is analogous to racism or sexism”. It’s no defense, but I’m not intentionally trying to use dark arts of persuasion.

Johnicholas Mar 15, 2012, 5:35 PM
3 points
in reply to: fubarobfusco’s comment on: Schelling fences on slippery slopes
That’s true, it is a funny sort of deontology.

Note that Kant actually claimed that he was not preferring one consequence over another, he was finding a self-contradiction in one consequence, and no self-contradiction in the other.

That is, “you should not steal” because, at the end of the slippery slope, there is a self-contradiction, something like “if everybody ought to steal, what does theft even mean?”. (I’m trying to give Kant a fair shake, though I think he’s wrong.)

Johnicholas Jan 19, 2012, 7:27 PM
2 points
in reply to: wedrifid’s comment on: Histocracy: Open, Effective Group Decision-Making With Weighted Voting
I am not an expert, I just think it’s a big field, with a lot of work done in it. There’s an entire journal called “Information Fusion” for example (though I don’t think highly of Dempster-Schafer stuff).

The best exemplar I can point to is Freund and Schapire’s “Adaptive game playing using multiplicative weights”: http://cseweb.ucsd.edu/~yfreund/papers/games_long.pdf

Another possibly-decent review is Blum and Mansour’s chapter on regret minimization: www.cs.cmu.edu/~avrim/Papers/regret-chapter.pdf

Johnicholas Jan 18, 2012, 7:32 PM
5 points
on: Histocracy: Open, Effective Group Decision-Making With Weighted Voting
This post sounds to me like the author almost invented prediction markets or other well-known expert-fusion information-aggregation techniques, but then fell into “not invented here” syndrome, rationalizing and hypothesizing unproven advantages to the tiny differences between histocracy and the other similar ideas.

Possibly this points out a flaw in our allocation of prestige—in order to correctly incent adoption, we need to spread prestige from the first or most well-known advocate of an idea to the early adopters. See Derek Siver’s idea of the First Follower: http://sivers.org/ff

Johnicholas Dec 29, 2011, 12:26 AM
7 points
on: For-Profit Rationality Training
I think one of the worries I would have is—is this a cult, feel-good motivational-speaker sort of thing, a self-propagating meme predator attacking me?

Possibly an attempt to align your financial incentives with the students would help—for example, by offering some sort of money back if not satisfied after such-and-so period of time, or accepting student labor as payment (while still paying them at their current rate—that is, hiring them out at a higher, post-training rate).

I also think that because this is an information good (videos, software, text), you might not be able to charge what you initially might expect to be able to charge. Possibly generating content continuously, giving away a fraction of it (the oldest fraction) and charging for the cutting-edge updates could work.

Johnicholas Dec 20, 2011, 8:47 PM
0 points
in reply to: TruePath’s comment on: Value of Information: Four Examples
Interesting..

So the original tree T models a tree of “states of affairs”, and the original partition or subset T’ models each node being either under the decision-maker’s control or not. Then the function f would go from elements of T’ to successors of those elements—nodes of T, to be sure, but there’s a side condition there. Then the probability measure P is a somewhat more powerful way of attaching probabilities to the non-choice nodes—that is, if you have a distribution over the successors of each non-choice node, then you can obtain a probability measure, but a probability measure over paths would allow some additional correlations. The function u can be understood (in a finite tree) as labelling leaves with utilities, because paths of maximum length in a finite tree are isomorphic with leaves—but by describing it the way you did, you leave the door open to applying this formalism to an infinite tree. UP(f) would be the utility of a particular strategy (f), and U(f|\sigma) would be… the utility given a certain initial sequence of events? So \sigma is a finite path segment?

I don’t understand the grammar of “Now either …. and …”—should it be “Now either … or …”? Or is it really “Now assume … and …”?

When you use U(f) later, I am guessing that’s either UP(f) with the P elided, or U(f | the empty path segment) - regardless, we’re going to have to fix a P in order to get a utility, right?

Then the phrase “f(sigma)=f(tau) if the set of x such that \sigma(x) is not equal to tau(x) is a subset of S”—If I understand correctly, sigma and tau are finite path segments, which are isomorphic to nodes in the tree—but are they functions from nodes? If they are functions, wouldn’t they be from, say, initial sections of the integers 0...n TO nodes? If I understand correctly, they’re going to diverge at at most one point—once diverged, since it’s a tree, they’re not going to be able to rejoin. Or were you saying tree and thinking ‘dag’?

I worry about the types of these things; coding it up in something like Haskell or Ocaml might make everything sharper and perhaps suggest simplifications. I’m sure that you can carry through the basic intuition.

Thanks for thinking about this.

Johnicholas Dec 1, 2011, 8:58 PM
13 points
in reply to: [deleted]’s comment on: Hack Away at the Edges
There’s another story that compromises between the “Genius” story (which is almost certainly false in general; I can’t say whether it’s true in this case) and the Bazaar / “everyone and nobody did it” story, which neglects the crucial importantances of people like Tao and Gowers.

How do trade hubs arise? There’s a network effect, where if NYSE or Jita is a deep, liquid market, then NYSE or Jita becomes more desirable trade location, and therefore becomes even more liquid. How do we standardize on currencies? There’s a network effect, where a dollar or a cowrie is valuable because it is widely accepted in trade, and so becomes more valuable.

However, if there is an initially symmetric situation, then there’s a coordination problem of “where will the hub be?”. The geometric centrality of Jita or New York (where “geometry” includes things like velocity of travel and political risks) is crucial for them to win the initial struggle with other candidate hubs. The value-density (lightness, ease of transport) fungibility, and difficulty of forging paper money, and perhaps the association with national identity and taxation was crucial for paper money winning out as the standard for trade.

Gowers and Tao’s prestige might have acted more as a symmetry-breaker, identifying THIS as the hub that is likely going to eventually be the place to be.

Johnicholas Nov 22, 2011, 4:48 PM
0 points
on: Value of Information: Four Examples
I’m a little confused about “value of information” as a precise concept.

Suppose that you have a tree with two kinds of interior nodes in it, and leaves labeled with utilities. One interior node is a choice node, and the other is a nature node, with an associated distribution over its subtrees. It’s fairly obvious that you can work backwards up the tree, and find both an optimum strategy and an expected value of the entire tree.

However, I don’t see where “value of information” shows up in this framework anywhere. Do I need to distinguish some choice nodes as “information gathering”, and apply a default strategy to them as “don’t gather any information”, and then compute value of information as the difference between best I can do with my eyes closed and the best I can do flat out?

What if there is no natural partition of some actions as information gathering and not-information-gathering?

Is there some homomorphism from the first tree (which is normally a tree of evidence or belief states) to an “externally visible” tree, where two nodes are identified if the only difference is inside my head?

Johnicholas Oct 23, 2011, 3:48 PM
10 points
on: Better Disagreement
The first few levels of the hierarchy, DH0, DH1, DH2, DH3, are not useless, if you’re concerned with getting to the truth efficiently. Pointing out that the author has no credentials in the area they are speaking in would allow people to limit their attention to credentialed speakers (due to bounded rationality). The senator example is similar—if you can’t listen to everything and have to prune someone, choosing to forgo listening to obviously biased speakers might be a reasonable choice.

DH2 and DH3 are sometimes valuable as well, because we can do some inference about the world, taking a piece of communication as evidence. For example, we may infer something about the author’s personality or beliefs from their tone; and then (bounded rationality again) decide whether to read further based on this (limited, probabilistic) estimate of the author.

Contradiction works similarly; it’s a performance, putting the reputation of the speaker behind the negated claim. If a third party hears a college freshman make a point, and the professor flatly contradicts it, then the third party may be able to decide who to believe, shortcutting detailed analysis (scarce processing again). In some cases, this kind of contradiction argument can even be supported solely on tone, without leaning on reputation. A scholarly-sounding contradiction to a casual claim may be sufficient to sway me in some situations—perhaps the stakes are low.