I internally debated this question myself. Ideally, I’d completely agree with you. But I needed the shorter publishing and feedback cycle for a number of reasons. Sorry, but a longer one may not have happened at all.
Edit: future readers will have the benefit of a link to part 2
In the model there’s the distribution p, which determines how the world is changing. In the chess example this would include: a) how the agent’s action changes the state of the game + b) some distribution we assume (but which we may or may not actually know) about the opponent’s action and the resulting state of the game. In a physics example, p should include the relevant laws of physics, together with constants which tell the rate (and manner) in which the world is changing. Any changing parameters should be part of the state.
It seems that you’re saying that it may be difficult to know what p is. Then you are very much correct. You probably couldn’t infer the laws of physics from the current wave function of the universe, or the rules of chess from the current state of the game. But at this point we’re only assuming that such laws exist, not that we know how to learn them.
p and q are probability distributions, which is where we allow for randomness in the process. But note that randomness becomes a tricky concept if you go deep enough into physics.
As for the “quantum mind” theory, as far as I can tell it’s fringe science at best. Personally, I’m very skeptical. Regardless, such a model can still have the Markov property, if you include the wave function in your state.
There’s supposed to be some way to do so partially, if anyone knows what it is.
This should work in Markdown, but it seems broken :(
Edit: test
Thanks, Vincent, it works!
And how do you strikeout your comment?
I’m not sure what you mean. It looks fine to me, and I can’t find where to check / change such a setting.
Edit:
Very strange. Fixed, I hope.
Thanks!
You are expressing a number of misconceptions here. I may address some in future posts, but in short:
By information I mean the Shannon information (see also links in OP). Your example is correct.
By the action of powering the electromagnet you are not increasing your information on the state of the world. You are increasing your information on the state of the coin, but through making it dependent on the state of the electromagnet which you already knew. This point is clearly worth a future post.
There is no “entropy in environment”. Entropy is subjective to the viewer.
I realize now that an example would be helpful, and yours is a good one.
Any process can be described on different levels. The trick is to find a level of description that is useful. We make an explicit effort to model actions and observation so as to separate the two directions of information flow between the agent and the environment. Actions are purely “active” (no information is received by the agent) while observations are purely “passive” (no information is sent by the agent). We do this because these two aspects of the process have very different properties, as I hope to make clear in future posts.
The process of “figuring out where the table is” involves information flowing in both directions, and so is neither an action nor an observation. Some researchers call such a thing “a subgoal”. We should break it down further, for example we could have taps as actions and echoes as observations, as you suggest.
If you want to argue that no information is lost by tapping, then fine, I won’t be pedantic and point out the tiny bits of information that do get lost. The point is that some information being lost is a representing feature of the process of taking an action. Over time, if you don’t take in new information through observations, your will have less and less information about the world, even if some actions you take have a high probability of not losing any information.
Excellent point. It will be a few posts (if the audience is interested) before I can answer you in a way that is both intuitive and fully convincing.
The technical answer is that the belief update caused by an action is deterministically contractive. It never increases the amount of information.
A more intuitive answer (but perhaps not yet convincing) is that, proximally, your action of asking your friend did not change the location of your laptop, only your friend’s mental state. And the effect it had on your friend’s mental state is that you are now less sure of what it is. You took some of the things you knew about it (“she was probably going to keep sitting around”) and made them no longer true.
Regarding information about the future (of the location of your laptop), it is always included in information about the present. Your own mental state is independent of the future given the present. Put another way, you can’t update without more observations. In this case, the location of your laptop merely becomes entangled with information that you already had on your friend’s mental state (“this is where she thinks my desk is”).
Having a word [...] is a more compact code precisely in those cases where we can infer some of those properties from the other properties. (With the exception perhaps of very primitive words, like “red” [...]).
Remember that mutual information is symmetric. If some things have the property of being red, then “red” has the property of being a property of those things. Saying “blood is red” is really saying “remember that visual experience that you get when you look at certain roses, apples, peppers, lipsticks and English buses and phone booths? The same happens with blood.” If I give you the list above, can you find (“infer”) more red things? Then “red” is a good word.
But do note that this is a dual sense to the one in which “human” is a good word. Most of the properties of humans are statistically necessary for being human: remove any one of them, and the thing is much less likely to be human. “Human” is a good word because these properties are positively correlated. On the other hand, most of the red things are statistically sufficient for being red: take any one of them, and the thing is much more likely to be red. “Red” is a good word because these things are negatively correlated—they are a bunch of distinct things with a shared aspect.
The Harsanyi paper is very enlightening, but he’s not really arguing that people have shared priors. Rather, he’s making the following points (section 14):
It is worthwhile for an agent to analyze the game as if all agents have the same prior, because it simplifies the analysis. In particular, the game (from that agent’s point of view) then becomes equivalent to a Bayesian complete-information game with private observations.
The same-prior assumption is less restrictive than it may seem, because agents can still have private observations.
A wide family of hypothetical scenarios can be analyzed as if all agents have the same prior. Other scenarios can be easily approximated by a member of this family (though the quality of the approximation is not studied).
All of this is mathematically very pleasing, but it doesn’t change my point. That’s mainly because in the context of the Harsanyi paper “prior” means before any observation, and in the context of this post “prior” means before the shared observation (but possibly after private observations).
I’m aware of this result. It specifically requires the two Beyesians to have the same prior. My point is exactly that this doesn’t have to be the case, and in reality is sometimes not the case.
EDIT: The original paper by Aumann references a paper by Harsanyi which supposedly addresses my point. Aumann himself is careful in interpreting his result as supporting my point (since evidently there are people who disagree despite trusting each other). I’ll report here my understanding of the Harsanyi paper once I get past the paywall.
Traditional Rationalists can agree to disagree. Traditional Rationality doesn’t have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence.
This is also true of Bayesians. The probability estimate given the evidence is a property of the map, not the territory (hence “estimate”). One correct posterior implies one correct prior. What is this “Ultimate Prior”? There isn’t one.
Possibly, you meant that there’s one correct posterior given the evidence and the prior. That’s correct, but it doesn’t prevent Bayesians from disagreeing, because they do have different priors.
Alternatively, one can point out that the “given evidence” operator is, in expectation, always non-expansive, and contractive when the priors disagree. This means that the beliefs of Perfect Bayesians with shared observations converge (with probability 1) into a single posterior. But this convergence is too slow for humans. Agreeing to disagree is sometimes our only option.
Incidentally, it’s Traditional Rationalists who believed they should never agree to disagree: the set of hypotheses which aren’t “ruled out” by confirmed and repeatable experiments, they argued, is a property of the territory.
A GAI with the utility of burning itself? I don’t think that’s viable, no.
What do you mean by “viable”?
Intelligence is expensive. More intelligence costs more to obtain and maintain. But the sentiment around here (and this time I agree) seems to be that intelligence “scales”, i.e. that it doesn’t suffer from diminishing returns in the “middle world” like most other things; hence the singularity.
For that to be true, more intelligence also has to be more rewarding. But not just in the sense of asymptotically approaching optimality. As intelligence increases, it has to constantly find new “revenue streams” for its utility. It must not saturate its utility function, in fact its utility must be insatiable in the “middle world”. A good example is curiosity, which is probably why many biological agents are curious even when it serves no other purpose.
Suicide is not such a utility function. We can increase the degree of intelligence an agent needs to have to successfully kill itself (for example, by keeping the gun away). But in the end, it’s “all or nothing”.
But anyway it can’t be that Godelian reasons prevent intelligences from wanting to burn themselves, because people have burned themselves.
Gödel’s theorem doesn’t prevent any specific thing. In this case I was referring to information-theoretic reasons. And indeed, suicide is not a typical human behavior, even without considering that some contributing factors are irrelevant for our discussion.
Do you count the more restrictive technology by which humans operate as a constraint which artificial agents may be free of?
Why not? Though of course it may turn out that AI is best programmed on something unlike our current computer technology.
In that sense, I completely agree with you. I usually don’t like making the technology distinction, because I believe there’s more important stuff going on in higher levels of abstraction. But if that’s where you’re coming from then I guess we have resolved our differences :)
Not at all. If you insist, let’s take it from the top:
I wanted to convey my reasoning, let’s call it R.
I quoted a claim of the form “because P is true, Q is true”, where R is essentially “if P then Q”. This was a rhetorical device, to help me convey what R is.
I indicated clearly that I don’t know whether P or Q are true. Later I said that I suspect P is false.
Note that my reasoning is, in principle, falsifiable: if P is true and Q is false, then R must be false.
While Q may be relatively easy to check, I think P is not.
I expect to have other means of proving R.
I feel that I’m allowed to focus on conveying R first, and attempting to prove or falsify it at a later date. The need to clarify my ideas helped me understand them better, in preparation of future proof.
I stated clearly and repeatedly that I’m just conveying an idea here, not providing evidence for it, and that I agree with readers who choose to doubt it until shown evidence.
Do you still think I’m at fault here?
EDIT: Your main objection to my presentation was that Q could be false. Would you like to revise that objection?
I’ll try to remember that, if only for the reason that some people don’t seem to understand contexts in which the truth value of a statement is unimportant.
a GAI with [overwriting its own code with an arbitrary value] as its only goal, for example, why would that be impossible? An AI doesn’t need to value survival.
A GAI with the utility of burning itself? I don’t think that’s viable, no.
I’d be interested in the conclusions derived about “typical” intelligences and the “forbidden actions”, but I don’t see how you have derived them.
At the moment it’s little more than professional intuition. We also lack some necessary shared terminology. Let’s leave it at that until and unless someone formalizes and proves it, and then hopefully blogs about it.
could you clarify your position, please?
I think I’m starting to see the disconnect, and we probably don’t really disagree.
You said:
This sounds unjustifiably broad
My thinking is very broad but, from my perspective, not unjustifiably so. In my research I’m looking for mathematical formulations of intelligence in any form—biological or mechanical.
Taking a narrower viewpoint, humans “in their current form” are subject to different laws of nature than those we expect machines to be subject to. The former use organic chemistry, the latter probably electronics. The former multiply by synthesizing enormous quantities of DNA molecules, the latter could multiply by configuring solid state devices.
Do you count the more restrictive technology by which humans operate as a constraint which artificial agents may be free of?
I’m not sure why you say this.
Please remember that this introduction is non-standard, so you may need to be an expert on standard RL to see the connection. And while some parts are not in place yet, this post does introduce what I consider to be the most important part of the setting of RL.
So I hope we’re not arguing over definitions here. If you expand on your meaning of the term, I may be able to help you see the connection. Or we may possibly find that we use the same term for different things altogether.
I should also explain why I’m giving a non-standard introduction, where a standard one would be more helpful in communicating with others who may know it. The main reason is that this will hopefully allow me to describe some non-standard and very interesting conclusions.