In addition to Prase’s comment on the possibility of an unbounded chain of strategies (and building off of what I think shminux is saying), I’m also wondering (I’m not sure of this) if bounded cognitive strategies are strictly monotonically increasing? i.e.( For all strategies X and Y, X>Y or Y>X). It seems like lateral moves could exist given that we need to use bounded strategies—certain biases can only be corrected to a certain degree using feasible methods, and mediation of biases rests on adopting certain heuristics that are going to be better optimized for some minds than others. Given two strategies A and B that don’t result in a Perfect Bayesian, it certainly seems possible to me that EU(Adopt A) = EU(Adopt B) and A and B dominate all other feasible strategies by making a different set of tradeoffs at equal cost (relative to a Perfect Bayesian).
Zetetic
Not to mention the question of just how friendly a heavily enhanced human will be. Do I want an aggressive king maker with tons of money to spend on upgrades to increase their power by massively amplifying their intelligence? How about a dictator who had been squirreling away massive, illegally obtained funds?
Power corrupts, and even if enhancements are made widely available, there’s a good possibility of an accelerating (or at least linearly increasing) gap in cognitive enhancements (I have the best enhancement, ergo I can find a quicker path to improving my own position, including inventing new enhancements if the need arises—thereby securing my position at the top long enough to seize an awful lot of control). An average person may end up with a greatly increased intelligence that is miniscule relative to what’s possible to attain if they had the resources to do so.
In a scenario where someone who does have access to lots of resources can immediately begin to control the game at a level of precision far beyond what is obtainable for all but a handful of people, this may be a vast improvement over a true UFAI let loose on an unsuspecting universe, but it’s still a highly undesirable scenario. I would much rather have an FAI (I suspect some of these hypothetical persons would decide it to be in their best interest to block any sort of effort to build something that outstrips their capacity for controlling their environment—FAI or no).
If its ok to kill animals, then there’s no reason to say it’s not ok to kill blicket-less humans.
I just want to point out this alternative position: Healthy (mentally and otherwise) babies can gain sufficient mental acuity/self-awareness to outstrip animals in their normal trajectory—i.e. babies become people after a while.
Although I don’t wholeheartedly agree with this position, it seems consistent. The stance that such a position would imply is that babies with severe medical conditions (debilitating birth defects, congenital diseases etc.) could be killed with parental consent, and fetuses likely to develop birth defects can be aborted, but healthy fetuses cannot be aborted, and healthy babies cannot be killed. I bring this up in particular because of your other post about the family with the severely disabled 6-year-old.
I think it becomes a little more complicated when we’re talking about situations in which we have the ability to impart self-awareness that was previously not there. On the practical level I certainly wouldn’t want to force a family to either face endless debt from an expensive procedure or a lifetime of grief from a child that can’t function in day to day tasks. It also brings up the question of whether to make animals self-aware, which is… kind of interesting but probably starting to drift off topic.
Have you looked into Univalent foundations at all? There was an interesting talk on it a while ago and it seems as though it might be relevant to your pursuits.
I’ve read your post on Syntacticism and some of your replies to comments. I’m currently looking at the follow up piece (The Apparent Reality of Physics).
I might be misunderstanding you, but it looks like you’re just describing fragments of infinitary logic which has a pretty solid amount of research behind it. Barwise actually developed a model theory for it, you can find a (slightly old) overview here (in part C).
Infinitary logic admits precisely what you’re talking about. For instance; it models sentences with universal and existential quantifiers in N using conjunctions and disjunctions (respectively) over the index ω.
As far as
when we have all that Gödel and Löb and Tarski stuff flying around, some people are tempted to say “Oh, mathematics can’t really prove things, therefore not-Platonism and not-infinities”, which I think is a mistake.
I don’t think the results of Gödel, Löb and Tarski are necessary to conclude that Platonism is at least pretty questionable. I don’t know where the “mathematics can’t really prove things” bit is coming from—we can prove things in mathematics, and I’ve never really seen people claim otherwise. Are you implicitly attributing something like Absolute Truth to proofs?
Anyway, I’ve been thinking about a naturalistic account of anti-realism for a little while. I’m not convinced it’s fruitful, but Platonism seems totally incomprehensible to me anymore. I can’t see a nice way for it to mesh with what I know about brains and how we form concepts—and the accounts I read can’t give any comprehensible account of what exactly the Platonic realm is supposed to be, nor how we access it. It looks like a nice sounding fiction with a big black box called the Platonic realm that everything we can’t really explain gets stuck into. Nor can I give any kind of real final account of my view of mathematics, because it would at least require some new neuroscience (which I mention in that link).
I will say that I don’t think that mathematics has any special connection with The Real Truth, and I also think that it’s a terrible mistake to think that this is a problem. Truth is evidence based, we figure out the truth by examining our environment. We have extremely strong evidence that math performs well as a framework for organizing information and for reasoning predictively and counterfactually—that it works extremely well in modelling the world around us. I definitely don’t think that it’s reasonable to think that math grants us license to epistemic certainty.
I wouldn’t mind being convinced otherwise, but I’d need an epistemically sound account of Platonism first. I don’t find Tegmark to be super convincing, though I do like that he restricts the MUH to computable structures—but I’m also not sure if that strictly qualifies as Platonism like you seem to be using it. At the very least it gives a moderately constructive account of exactly what is being claimed, which is something I would require from any Platonic theory.
Yeah, I’d say that’s a fair approximation. The AI needs a way to compress lots of input data into a hierarchy of functional categories. It needs a way to recognize a cluster of information as, say, a hammer. It also needs to recognize similarities between a hammer and a stick or a crow bar or even a chair leg, in order to queue up various policies for using that hammer (if you’ve read Hofstadter, think of analogies) - very roughly, the utility function guides what it “wants” done, the statistical inference guides how it does it (how it figures out what actions will accomplish its goals). That seems to be more or less what we need for a machine to do quite a bit.
If you’re just looking to build any AGI, he hard part of those two seems to be getting a nice, working method for extracting statistical features from its environment in real time. The (significantly) harder of the two for a Friendly AI is getting the utility function right.
A couple of things come to mind, but I’ve only been studying the surrounding material for around eight months so I can’t guarantee a wholly accurate overview of this. Also, even if accurate, I can’t guarantee that you’ll take to my explanation.
Anyway, the first thing is that brain form computing probably isn’t a necessary or likely approach to artificial general intelligence (AGI) unless the first AGI is an upload. There doesn’t seem to be good reason to build an AGI in a manner similar to a human brain and in fact, doing so seems like a terrible idea. The issues with opacity of the code would be nightmarish (I can’t just look at a massive network of trained neural networks and point to the problem when the code doesn’t do what I thought it would).
The second is that consciousness is not necessarily even related to the issue of AGI, the AGI certainly doesn’t need any code that tries to mimick human thought. As far as I can tell, all it really needs (and really this might be putting more constraints than are necessary) is code that allows it to adapt to general environments (transferability) that have nice computable approximations it can build by using the data it gets through it’s sensory modalities (these can be anything from something familiar, like a pair of cameras, or something less so like a geiger counter or some kind of direct feed from thousands of sources at once).
Also, a utility function that encodes certain input patterns with certain utilities, some [black box] statistical hierarchical feature extraction [/black box] so it can sort out useful/important features in its environment that it can exploit. Researchers in the areas of machine learning and reinforcement learning are working on all of this sort of stuff, it’s fairly mainstream.
As far as computing power—the computing power of the human brain is definitely measurable so we can do a pretty straightforward analysis of how much more is possible. As far as raw computing power, I think we’re actually getting quite close to the level of the human brain, but I can’t seem to find a nice source for this. There are also interesting “neuromorphic” technologies geared to stepping up the massively parallel processing (many things being processed at once) and scale down hardware size by a pretty nice factor (I can’t recall if it was 10 or 100), such as the SyNAPSE project. In addition, with things like cloud/distributed computing, I don’t think that getting enough computing power together is likely to be much of an issue.
Bootstrapping is a metaphor referring to the ability of a process to proceed on its own. So a bootstrapping AI is one that is able to self-improve along a stable gradient until it reaches superintelligence. As far as “how does it know what bits to change”, I’m going to interpret that as “How does it know how to improve itself”. That’s tough :) . We have to program it to improve automatically by using the utility function as a guide. In limited domains, this is easy and has already been done. It’s called reinforcement learning. The machine reads off its environment after taking an action an updates its “policy” (the function it uses to pick its actions) after getting feedback (positive or negative or no utility).
The tricky part is having a machine that can self-improve not just by reinforcement in a single domain, but in general, both by learning and by adjusting its own code to be more efficient, all while keeping its utility function intact—so it doesn’t start behaving dangerously.
As far as SIAI, I would say that Friendliness is the driving factor. Not because they’re concerned about friendliness, but because (as far as I know) they’re the first group to be seriously concerned with friendliness and one of the only groups (the other two being headed by Nick Bostrom and having ties to SIAI) concerned with Friendly AI.
Of course the issue is that we’re concerned that developing a generally intelligent machine is probable, and if it happens to be able to self improve to a sufficient level it will be incredibly dangerous if no one put in some serious, serious effort into thinking about how it could go wrong and solving all of the problems necessary to safeguard against that. If you think about it, the more powerful the AGI is, the more needs to be considered. An AGI that has access to massive computing power, can self improve and can get as much information (from the internet and other sources) as it wants, could easily be a global threat. This is, effectively, because the utility function has to take into account everything the machine can affect in order to guarantee we avoid catastrophe. An AGI that can affect things at a global scale needs to take everyone into consideration, otherwise it might, say, drain all electricity from the Eastern seaboard (including hospitals and emergency facilities) in order to solve a math problem. It won’t “know” not to do that, unless it’s programed to (by properly defining its utility function to make it take those things into consideration). Otherwise it will just do everything it can to solve the math problem and pay no attention to anything else. This is why keeping the utility function intact is extremely important. Since only a few groups, SIAI, Oxford’s FHI and the Oxford Martin Programme on the Impacts of Future Technologies, seem to be working on this, and it’s an incredibly difficult problem, I would much rather have SIAI develop the first AGI than anywhere else I can think of.
Hopefully that helps without getting too mired in details :)
Kind of curious about how many LWers live in the greater Cincinnati area. I posted a while ago looking for some but I only had maybe one response. It looks like there are at least three of us plus a few others from the area.
I live in Cincinnati and I’d definitely be willing to consider it.
Nor is it clear in what way they’re objects in the first place. I mean, arguably there’s no such thing in reality as an apple—there’s a bunch of atoms which smoothly coexist with other atoms we think of as air or table, and any cutoff point is inherently arbitrary.
That’s a very important point. What you’re talking about is known as object individuation and there are actually quite a few interesting studies on it.
In fact, that’s my theory, or perhaps a proto-theory: that what’s needed to develop mathematics is not so much an evolved faculty of syntax, but something more basic: an ability to conceptualize something that’s different from something else.
So in effect your proto-theory is that object individuation is sufficient to develop mathematics? I’m not sure that I buy that. Non-human apes seem to have a capacity for object individuation and even categorization, yet lack mathematics. I will say this: the writers of that article do issue the caveat that the matter of whether this is full blown object individuation rather than some sort of tracking is not yet settled. Nor does it seem that we fully understand the mechanisms for object individuation. So you could be right, but I think that while object individuation is a necessary condition for the development of mathematics, it doesn’t seem to be sufficient.
Also, to further clarify what I mean by “syntax”, I would include in (as the key feature) an evolved faculty of syntax the ability to think in combinatorial/recursive terms, but again I readily concede that in order for this to work in the way that it does in humans, you certainly seem to need object individuation. I just don’t see how you can have mathematics without the further ability to think in combinatorial terms once you have this discrete mapping of the world object individuation gives you. At the very least, I don’t see how object individuation alone can give rise to mathematics.
If I’m reading you correctly, you deny any ontological status to “10”. But then I’m not sure how you square that with the following experiment: show 10 apples to two people separately, one will say “ten apples”, the other will say “dix pommes”. Repeat many times with other people similarly, establish unambiguously the correspondence between ‘ten’ and ‘dix’; where does it come from? Linguistically, it doesn’t feel very different from establishing the correspondence between ‘red’ and ‘rouge’. If you’re saying that the common thing is “a set of statistical regularities”, that seems to just delay the question—what exactly are those and why do they match in humans with different cultures and mutually unintelligible language?
First, I want to be clear that the “statistical set of regularities” I’m talking about isn’t at the level of language or grammar recognition/learning. When I say statistical set of regularities I’m talking about the “object” level. Whatever allows you to discretize apple as something in particular in your environment.
Also, I don’t know if I’m going as far as denying ontological status to “10”, and even if I were, I’m having trouble figuring out the point of your experiment or how it relates to my position. I might just be totally missing your point here. Language isn’t sufficient for indicating the ontology of a group of people, if that’s an underlying assumption here. There are indigenous peoples that have no word for the color blue, yet still have blue-green cone cells. I think maybe part of the confusion is that I’m stripping this language/grammar down to a sort of combinatorial/recursive modality of thought, this is effectively the universal grammar thesis. There is some additional research suggesting that the confluence of recursive thought and object individuation still would not be sufficient for mathematics. I tend to interpret this to mean that the boost in computing power you get when you can actually communicate ideas to multiple minds is necessary to get very far with creating mathematics, though you might still have the capacity for mathematical thought without this additional ability to communicate highly complex ideas.
As far as you last question:
If you’re saying that the common thing is “a set of statistical regularities”, that seems to just delay the question—what exactly are those and why do they match in humans with different cultures and mutually unintelligible language?
The way I’m reading you, you don’t give any ontological status to “apple” since, as you said,
I mean, arguably there’s no such thing in reality as an apple—there’s a bunch of atoms which smoothly coexist with other atoms we think of as air or table, and any cutoff point is inherently arbitrary.
So you have more or less the same problem, correct? I don’t really feel like I have a nice answer to it. How do you account for this issue? You basically seem to be asking me how object individuation works, which is something that doesn’t seem to be well understood yet, so I don’t see how I could answer this. Am I reading you right or am I misinterpreting you (on the question or on the issue with the apple)?
A Platonist may think that we’ve evolved the ability to query and study the world of platonic ideas.
I think that’s the Penrose argument, from Emperor’s New Mind? I’m really unclear on what that would mean, because I don’t know what it would mean to “query” the “Platonic realm”, nor do I know what the “world of platonic ideas” is supposed to be or what its actual relation to the physical world is. It sounds like an extreme form of dualism, specified to mathematical thought rather than subjective feeling or qualia. I mean, what if I were to try to explain subjective experience by suggesting that we evolved the ability to access the realm of qualia? How is that different?
If you think it’s sensible and that I’ve totally misunderstood it, could you please explain it to me or send me a relevant link?
To say that human beings “invented numbers”—or invented the structure implicit in numbers—seems like claiming that Neil Armstrong hand-crafted the Moon. The universe existed before there were any sentient beings to observe it, which implies that physics preceded physicists. This is a puzzle, I know; but if you claim the physicists came first, it is even more confusing because instantiating a physicist takes quite a lot of physics. Physics involves math, so math—or at least that portion of math which is contained in physics—must have preceded mathematicians. Otherwise, there would have no structured universe running long enough for innumerate organisms to evolve for the billions of years required to produce mathematicians.
This is something I’ve been thinking about quite a lot lately. Mathematics, as we do it, seems to be a feature of the physical world that requires something like a combinatorial grammar and something like the parietal lobe that can manipulate and change visual representations based on that grammar.
Categories exist, in a sense, but they don’t exist in the form we tend to attribute to them. “Number” is, in a way, a fuzzy sort of concept; on one hand numbers are a linguistic convention that allows us to convey socially relevant information and to organize information for our own benefit. This allows us to construct a surrounding syntax that allows us to reason about them at the level of language. On the other they seem to have a rigid structure and meaning. Of course this latter issue can be explained away as the result of a user illusion – our consciousness is itself lossy, fuzzy, and not at all what we think that it is before having further training in areas such as cognitive neuroscience, why should we suppose that one part of conscious access breaks this pattern and how could it do so?
It seems like counting is really just a method for attaching a token that points at a space of possible meanings, or more realistically, a token that picks out a fuzzy parameter representing a subjective impression of a scenario. I count 10 apples, but there is only one of each thing, none of the objects are actually identical and I don’t think they could be in principle (though I could be confused here). When I say “there are 10 apples here”, I’m denoting the fact that another being with an ontology sufficiently similar to my own will recognize that there are 10 apples (though I don’t think about it that way, this is an implicit assumption). The apples fit a certain set of statistical regularities that lead me to classify them as such (having something like a well-trained neural network for such tasks), and the combinatorial, lexical aspect of my thinking appropriates a label known as a base 10 Arabic numeral to the scenario. This is useful to me because it allows me to think at the level of syntax – I know that “apples” (things that I classify as such) tend to function in a certain way, and salient aspects of apples – primary features in my classification algorithm – allow me to map a finite set of future courses of action involving them.
Viewing mathematics in this way, as a set of evolved cognitive phenomena that works because being able to extract, organize and communicate information about you/your environment is evolutionarily advantageous, seems to make it tough to be a Platonist.
Successful use would count as evidence for the laws of probabilities providing “good” values right? So if we use these laws quite a bit and they always work, we might have P(Laws of Probability do what we think they do) = .99999 We could discount our output using this. We could also be more constructive and discount based on the complexity of the derivation using the principle “long proofs are less likely to be correct” in the following way: Each derivation can be done in terms of combinations of various sub-derivations so we could get probability bounds for new, longer derivations from our priors over other derivations from which it is assembled. (derivations being the general form of the computation rather than the value specific one).
ETA: Wait, were you sort of diagonalizing on Bayes Theorem because we need to use that to update P(Bayes Theorem)? If so I might have misread you.
According to the first step of the algorithm, A will play chicken with the universe and return 1, making S inconsistent. So if S is consistent, that can’t happen.
Is this right? I’m wondering if you’re assuming soundness relative to the natural semantics about A, because in step one, it isn’t clear that there is a contradiction in S rather than a failure of S to be sound with respect to the semantics it’s supposed to model (what actions A can take and their utility). I might be confused, but wouldn’t this entail contradiction of the soundness of S rather than entailing that S is inconsistent? S would only be inconsistent if it can prove both A() = a and A()≠a, but unless you have further hidden assumptions about S I don’t see why A returning a would entail that S proves A() = a.
This is how I want to interpret this: S is some deduction system capable of talking about all actions A() can make, and proving some range of utility results about them. S is also consistent and sound.
Play chicken with the universe: if S proves that A()≠a for some action a, then return a.
If S proves for all a there is some u such that [A() = a ⇒ U() = u] , output argmax (a) else exit and output nothing.
Since proving A can’t take an action a ( that is, A()≠a ) entails that S is not sound (because A will take such an action in step 1), S can’t prove any such result. Also, since proving that an action has two distinct utility values leads to A≠a, the soundness and consistency of S entails that this can’t happen. Does this seem right?
Also, step two seems too strong. Wouldn’t it suffice to have it be:
1) For all actions a, if there is some u such that [A() = a ⇒ U() = u] and u > 0, (), else add [A() = a ⇒ U() = 0] to the axioms of S (only for the duration of this decision)
2) output argmax (a)
My thought is that there could be some possible actions the agent can take that might not have provable utility and it seems like you should assign an expected utility of 0 to them (no value being privileged, it averages out to 0), but if you can prove that at least one action has positive utility, then you maximize expected utility by choosing the one with the highest positive utility.
This is weaker than the current step two but still seems to have the desired effect. Does this hold water or am I missing something?
I think the real motivator that Runzel and Chatz provided is the opportunity for students to signal their dislike of Justin Beiber by donating.
I’m not sure, at the very least this could be inverted—they caused a situation where many groups of students would tend to pressure each other to donate in order to avoid the negative utility that would come from acting against their peer group.
If every student is signaling their dislike by donating, donating becomes the status quo. In that case, donating wouldn’t be a status raising action, but not signaling would deviate from the status quo, which is perceived as high risk so you’re willing to pay to avoid that. In the end, everyone but the groups for which the signaling is status neutral is at a net loss.
No problem! It’s an interesting topic with lots of surrounding results that are somewhat surprising (at least to me).
A more specific statement of the result from the paper is:
Lemma 1 Suppose that the theory T is omega consistent with repsect to some formula P(y) and that T has a finitely axiomatizable subtheory S satisfying
(i) Each recursive relation is definable in S
(ii) For every m = 0, 1, ….,
)Then each subtheory of T is of complete degree
Also:
Corallary 6 Let T be the theory ZF or any extension of ZF. If T is omega consistent with respect to the formula then the degrees of subtheories of T are exactly the complete degrees.
ETA: Is this what you were referring to in:
Is there a formal system (not talking about the standard integers, I guess)
I’m not sure, I don’t see a related reply/comment. Either way, I’m not 100% sure I’m following all of the arguments in the papers, but it appears that the theories that are of intermediate degree are necessarily very unusual and complicated, and I’m not sure how feasible it would be to construct one explicitly.
ETA2: I found yet another interesting paper that seems to state that finding a natural example of a problem of intermediate degree is a long standing open problem.
I found another paper with an interesting result (unfortunately behind a pay wall). It asks what range of degrees occur for subtheories of arithmetic and concludes that a degree a is associated with these theories iff it’s a complete degree. What does that mean?
Definition 10 A degree of recursive unsolvability a is said to be complete if there exists a degree b such that b’ = a (According to Freiberg’s theorem [6, p.265] a degree a is complete iff 0′<= a).
That makes sense because b’ is basically a Turing jump from b, but it’s bad because 0′ is the degree of the halting problem. The first Turing jump from deciable theories is of degree equal to the halting problem. So it looks like any undecidable (in distinction from Presberger and other decidable theories) first order subsystems of arithmetic, which to me would be the most intuitive theories, are going to be at least as strong as the halting problem. That is, if I’m reading this right—proof theory is sort of a hobby and I’m mostly self-taught.
Something is going on with my comment, the middle section keeps disappearing.
Is there a formal system (not talking about the standard integers, I guess) whose provability oracle is strictly weaker than the halting oracle, but still uncomputable?
I did some reading and it looks like while there are some proofs of the existence of a dense set of intermediate Turing degrees, it’s a bit difficult to pin down definite problems of intermediate degree. I found one paper that postulates a couple of possibly intermediate degree problems.
This article (same author) talks about some of the surrounding issues and some the difficulties with proving the existence of an intermediate computational process.
Given those difficulties, it isn’t clear to me that intermediate proof oracles for formal systems exist and if they do it seems like that might be non-trivial, but I’m definitely not the best person to ask.
I’ve been on Mount Stupid a lot, maybe enough to be past Mount Stupid’s Mount Stupid. I’ve had a lot of interests that I’ve developed over the (relatively short) 22 years and I’ve been caught standing atop Mount Stupid (by others and by myself) enough that I often feel it in the pit of my stomach - a sort of combination of embarrassment and guilt—when I start shouting from there. Especially if no one corrects me and I realize my mistake. The worst is the feeling I get when I’ve established some authority in someone’s eyes and give them wrong information.
The best cure I’ve found for getting stuck atop Mount Stupid is to start learning a subject that’s been the long interest of an honest friend (someone who’s default mode of communication—at least among good friends—is significantly closer to Crocker’s rules than to ordinary conversation). That seems to have really been the most helpful thing for me. If you take up a subject, you get past Mount Stupid a lot quicker when there’s someone to push you tumbling off the top (or at least point out your vast ignorance). It also builds up a reflex for noticing your ignorance—you start to know what it feels like to have a shallow understanding of something and you start to recognize when you’re speaking out of your depth. You’ll have been conditioned by having been called out in the past. Of course, you’re still going to shout from high atop Mount Stupid a lot, but you’ll realize what you’re doing much more easily.
Caveat: I might still be atop Mount Stupid’s Mount Stupid, don’t forget that.