Tor Økland Barstad

Karma: 167

Tor Økland Barstad Mar 22, 2023, 7:39 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: AGI is uncontrollable, alignment is impossible
I do have arguments for that, and I have already mentioned some of them earlier in our discussion (you may not share that assesment, despite us being relatively close in mind-space compared to most possible minds, but oh well).
Some of the more relevant comments from me are on one of the posts that you deleted.
As I mention here, I think I’ll try to round off this discussion. (Edit: I had a malformed/misleading sentence in that comment that should be fixed now.)

Tor Økland Barstad Mar 22, 2023, 7:35 AM
7 points
6
in reply to: Donatas Lučiūnas’s comment on: God vs AI scientifically
Every assumption is incorrect unless there is evidence.
Got any evidence for that assumption? 🙃
Answer to all of them is yes. What is your explanation here?
Well, I don’t always “agree”^[1] with ChatGPT, but I agree in regards to those specific questions.
...
I saw a post where you wanted people to explain their disagreement, and I felt inclined to do so :) But it seems now that neither of us feel like we are making much progress.
Anyway, from my perspective much of your thinking here is very misguided. But not more misguided than e.g. “proofs” for God made by people such as e.g. Descartes and other well-known philiophers :) I don’t mean that as a compliment, but more so as to neutralize what may seem like anti-compliments :)
Best of luck (in your life and so on) if we stop interacting now or relatively soon :)
I’m not sure if I will continue discussing or not. Maybe I will stop either now or after a few more comments (and let you have the last word at some point).
1. ^
  I use quotation-marks since ChatGPT doesn’t have “opinions” in the way we do.
What links here?
- Tor Økland Barstad's comment on AGI is uncontrollable, alignment is impossible by Donatas Lučiūnas (Mar 22, 2023, 7:39 AM; 1 point)

Tor Økland Barstad Mar 22, 2023, 7:30 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: AGI is uncontrollable, alignment is impossible
Do you think you can deny existence of an outcome with infinite utility?
To me, according to my preferences/goals/inclinations, there are conceivable outcomes with infinite utility/disutility.
But I think it is possible (and feasible) for a program/mind to be extremely capable, and affect the world, and not “care” about infinite outcomes.
The fact that things “break down” is not a valid argument.
I guess that depends on what’s being discussed. Like, it is something to take into account/consideration if you want to prove something while referencing utility-functions that reference infinities.

Tor Økland Barstad Mar 22, 2023, 7:18 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: God vs AI scientifically
About universally compelling arguments?

First, a disclaimer: I do think there are “beliefs” that most intelligent/capable minds will have in practice. E.g. I suspect most will use something like modus ponens, most will update beliefs in accordance with statistical evidence in certain ways, etc. I think it’s possible for a mind to be intelligent/capable without strictly adhering to those things, but for sure I think there will be a correlation in practice for many “beliefs”.
Questions I ask myself are:
- Would it be impossible (in theory) to wire together a mind/program with “belief”/behavior x, and having that mind be very capable at most mental tasks?
- Would it be infeasible (for humans) to wire together a mind/program with “belief”/behavior x, and having that mind be very capable at most mental tasks?
And in the case of e.g. caring about “goals” I don’t see good reasons to think that the answer is “no”.
Like, I think it is physically and practically possible to make minds that act in ways that I would consider “completely stupid”, while still being extremely capable at most mental tasks.
Another thing I sometimes ask myself:
1. “Is it possible for an intelligent program to surmise what another intelligent mind would do if it had goal/preferences/optimization-target x?”
2. “Would it be possible for another program to ask about #1 as a question, or fetch that info from the internals of another program?”
If yes and yes, then a program could be written where #2 surmised from #1 what such a mind would do (with goal/preferences/optimization-target x), and carries out that thing.
I could imagine information that would make me doubt my opinion / feel confused, but nothing that is easy to summarize. (I would have to be wrong about several things—not just one.)

Tor Økland Barstad Mar 22, 2023, 7:01 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: God vs AI scientifically
With all the interactions we had, I’ve got an impression that you are more willing to repeat what you’ve heard somewhere instead of thinking logically.
Some things I’ve explained in my own words. In other cases, where someone else already has explained something thing well, I’ve shared an URL to that explanation.
more willing to repeat what you’ve heard somewhere instead of thinking logically
This seems to support my hypothesis of you “being so confident that we are the ones who “don’t get it” that it’s not worth it to more carefully read the posts that are linked to you, more carefully notice what we point to as cruxes, etc”.
Universally compelling arguments are not possible” is an assumption
Indeed. And it’s a correct assumption.
Why would there be universally compelling arguments?
One reason would be that the laws of physics worked in such a way that only minds that think in certain ways are allowed at all. Meaning that if neurons or transistors fire so as to produce beliefs that aren’t allowed, some extra force in the universe intervenes to prevent that. But, as far as I know, you don’t reject physicalism (that all physical events, including thinking, can be explained in terms of relatively simple physical laws).
Another reason would be that minds would need “believe”^[1] certain things in order to be efficient/capable/etc (or being the kind of efficient/capable/etc thinking machine that humans may be able to construct). But that’s also not the case. It’s not even needed for logical consistency^[2].
1. ^
  Believe is not quite the right word, since we also are discussing what minds are optimized for / what they are wired to do.
2. ^
  And logical consistency is also not a requirement in order to be efficient/capable/etc. As a rule of thumb it helps greatly of course. And this is a good rule of thumb, as rules of thumbs go. But it would be a leaky generalization to presume that it is an absolute necessity to have absolute logical consistency among “beliefs”/actions.

Tor Økland Barstad Mar 22, 2023, 6:48 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: God vs AI scientifically
What about “I think therefore I am”? Isn’t it universally compelling argument?
Not even among the tiny tiny section of mind-space occupied by human minds:
Notice also that “I think therefore I am” is an is-statement (not an ought-statement / something a physical system optimizes towards).
As to me personally, I don’t disagree that I exist, but I see it as a fairly vague/ill-defined statement. And it’s not a logical necessity, even if we presume assumptions that most humans would share. Another logical possibility would be Boltzmann brains (unless a Boltzmann brain would qualify as “I”, I guess).
I argue that “no universally compelling arguments” is misleading.
You haven’t done that very much. Only, insofar as I can remember, through anthropomorphization, and reference to metaphysical ough-assumptions not shared by all/most possible minds (sometimes not even shared by the minds you are interacting with, despite these minds being minds that are capable of developing advanced technology).

Tor Økland Barstad Mar 22, 2023, 2:03 AM
−4 points
0
in reply to: jimrandomh’s comment on: God vs AI scientifically
~~Agreed (more or less). I have pointed him to this post earlier. He has given no signs so far of comprehending it, or even reading it and trying to understand what is being communicated to him.~~
~~I’m saying this more directly than I usually would~~ ~~@Donatas~~, since you seem insistent on clarifying a disagreement/misunderstanding you think is important for the world, while it seems (as far as I can see) that you’re not comprehending all that is communicated to you (maybe due to being so confident that we are the ones who “don’t get it” that it’s not worth it to more carefully read the posts that are linked to you, more carefully notice what we point to as ~~cruxes, etc).~~

Edit: I was unnecessarily hostile/negative here.

Tor Økland Barstad Mar 22, 2023, 1:50 AM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: AGI is uncontrollable, alignment is impossible
He didn’t say that “infinite value” is logically impossible. He desdribed it as an assumption.
When saying “is possible, I’m not sure if he meant “is possible (conceptually)” or “is possible (according to the ontology/optimization-criteria of any given agent)”. I think the latter would be most sensible.

He later said: “I think initially specifying premises such as these more precisely initially ensures the reasoning from there is consistent/valid.”. Not sure if I interpreted him correctly, but I saw it largely as an encouragment to think more explicitly about things like these (not be sloppy about it). Or if not an encouragement to do that, then at least pointing out that it’s something you’re currently not doing.
If we have a traditional/standard utility-function, and use traditional/standard math in regards to that utility function, then involving credences of infinitie-utility outcomes would typically make things “break down” (with most actions considered to have expected utilities that are either infinite or undefined).
Like, suppose action A has 0.001% chance of infinite negative utility and 99% chance of infinite positive utility. The utility of that action would, I think, be undefined (I haven’t looked into it). I can tell for sure that mathemathically it would not be regarded to have positive utility. Here is a video that explains why.

If that doesn’t make intuitive sense to you, then that’s fine. But mathemathically that’s how it is. And that’s something to have awareness of (account for in a non-handwavy way) if you’re trying to make a mathemathical argument with a basis in utility functions that deal with infinities.

Even if you did account for that it would be besides the point from my perspective, in more ways than one. So what we’re discussing now is not actually a crux for me.

Like, suppose action A has 0.001% chance of infinite negative utility and 99% chance of infinite positive utility. The utility of that action would, I think, be undefined
For me personally, it would of course make a big difference whether there is a 0.00000001% chance of infinite positive utility or a 99.999999999% chance. But that is me going with my own intuitions. The standard math relating to EV-calculations doesn’t support this.

Tor Økland Barstad Mar 19, 2023, 11:58 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
Same traits that make us intelligent (ability to logically reason), make us power seekers.
Well, I do think the two are connected/correlated. And arguments relating to instrumental convergence are a big part of why I take AI risk seriously. But I don’t think strong abilities in logical reasoning necessitates power-seeking “on its own”.
I think it is wrong to consider Pascal’s mugging a vulnerability.
For the record, I don’t think I used the word “vulnerability”, but maybe I phrased myself in a way that implied me thinking of things that way. And maybe I also partly think that way.
I’m not sure what I think regarding beliefs about small probabilities. One complication is that I also don’t have certainty in my own probability-guesstimates.
I’d agree that for smart humans it’s advisable to often/mostly think in terms of expected value, and to also take low-probability events seriously. But there are exceptions to this from my perspective.
In practice, I’m not much moved by the original Pascal’s Vager (and I’d find it hard to compare the probability of the Christian fantasy to other fantasies I can invent spontaneously in my head).

Tor Økland Barstad Mar 19, 2023, 11:39 PM
3 points
2
in reply to: Donatas Lučiūnas’s comment on: AGI is uncontrollable, alignment is impossible
Most humans are not obedient/subservient to others (at least not maximally so). But also: Most humans would not exterminate the rest of humanity if given the power to do so. I think many humans, if they became a “singleton”, would want to avoid killing other humans. Some would also be inclined to make the world a good place to live for everyone (not just other humans, but other sentient beings as well).
From my perspective, the example of humans was intended as “existence proof”. I expect AGIs we develop to be quite different from ourselves. I wouldn’t be interested in the topic of alignment if I didn’t perceive there to be risks associated with misaligned AGI, but I also don’t think alignment is doomed/hopeless or anything like that 🙂

Tor Økland Barstad Mar 19, 2023, 11:30 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
I’d argue that the only reason you do not comply with Pascal’s mugging is because you don’t have unavoidable urge to be rational, which is not going to be the case with AGI.
I’d agree that among superhuman AGIs that we are likely to make, most would probably be prone towards rationality/consistency/”optimization” in ways I’m not.
I think there are self-consistent/”optimizing” ways to think/act that wouldn’t make minds prone to Pascal’s muggings.
For example, I don’t think there is anything logically inconsistent about e.g. trying to act so as to maximize the median reward, as opposed to the expected value of rewards (I give “median reward” as a simple example—that particular example doesn’t seem likely to me to occur in practice).
Thanks for your input, it will take some time for me to process it.
🙂

Tor Økland Barstad Mar 19, 2023, 11:04 PM
3 points
2
on: AGI is uncontrollable, alignment is impossible
Hopefully I’m wrong, please help me find a mistake.
There is more than just one mistake here IMO, and I’m not going to try to list them.
Just the title alone (“AGI is uncontrollable, alignment is impossible”) is totally misguided IMO. It would, among other things, imply that brain emulations are impossible (humans can be regarded as a sort of AGI, and it’s not impossible for humans to be aligned).
But oh well. I’m sure your perspectives here are earnestly held / it’s how you currently see things. And there are no “perfect” procedures for evaluating how much to trust one’s own reasoning compared to others.
I would advise reading the sequences (or listening to them as an audiobook) 🙂

Tor Økland Barstad Mar 19, 2023, 10:49 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
If an outcome with infinite utility is presented, then it doesn’t matter how small its probability is: all actions which lead to that outcome will have to dominate the agent’s behavior.
My perspective would probably be more similar to yours (maybe still with substantial differences) if I had the following assumptions:
1. All agents have a utility-function (or act indistinguishably from agents that do)
2. All agents where #1 is the case act in a pure/straight-forward way to maximize that utility-function (not e.g. discounting infinities)
3. All agents where #1 is the case have utility-functions that relate to states of the universe
4. Cases involving infinite positive/negative expected utility would always/typically speak in favor of one behavior/action. (As opposed to there being different possibilities that imply infinite negative/positive expected utility, and—well, not quite “cancel each other out”, but make it so that traditional models of utility-maximization sort of break down).
I think that I myself am an example of an agent. I am relatively utilitarian compared to most humans. Far-fetched possibilities with infinite negative/positive utility don’t dominate my behavior. This is not due to me not understanding the logic behind Pascal’s Muggings (I find the logic of it simple and straight-forward).

Generally I think you are overestimating the appropriateness/correctness/merit of using a “simple”/abstract model of agents/utility-maximizers, and presuming that any/most “agents” (as we more broadly conceive of that term) would work in accordance with that model.

I see that Google defines an agent as “a person or thing that takes an active role or produces a specified effect”. I think of it is cluster-like concept, so there isn’t really any definition that fully encapsulates how I’d use that term (generally speaking I’m inclined towards not just using it differently than you, but also using it less than you do here).
Btw, for one possible way to think about utility-maximizers (another cluster-like concept IMO), you could see this post. And here and here are more posts that describe “agency” in a similar way:
In this sort of view, being “agent-like” is more of gradual thing than a yes-no-thing. This aligns with my own internal model of “agentness”, but it’s not as if there is any simple/crisp definition that fully encapsulates my conception of “agentness”.
I think that Orthogonality thesis is right only if an agent is certain that an outcome with infinite utility does not exist. And I argue that an agent cannot be certain of that. Do you agree?
In regards to the first sentence (“I think that Orthogonality thesis is right only if an agent is certain that an outcome with infinite utility does not exist”):
No, I don’t agree with that.
In regards to the second sentence (“And I argue that an agent cannot be certain of that”):
I’m not sure what internal ontologies different “agents” would have. Maybe, like with us, may have some/many uncertainties that don’t correspond to clear numeric values.
In some sense, I don’t see “infinite certainty” as being appropriate in regards to (more or less) any belief. I would not call myself “infinitely certain” that moving my thumb slightly upwards right now won’t doom me to an eternity in hell, or that doing so won’t save me from an eternity in hell. But I’m confident enough that I don’t think it’s worth it for me to spend time/energy worrying about those particular “possibilities”.

Tor Økland Barstad Mar 19, 2023, 4:34 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
It seems that you do not recognize https://www.lesswrong.com/tag/pascal-s-mugging .
Not sure what you mean by “recognize”. I am familiar with the concept.
But to be honest most of statements that we can think of may be true and unknowable, for example “aliens exist”, “huge threats exist”, etc.
“huge threat” is a statement that is loaded with assumptions that not all minds/AIs/agents will share.
Can you prove that there cannot be any unknowable true statement that could be used for Pascal’s mugging?
Used for Pascal’s mugging against who? (Humans? Cofffee machines? Any AI that you would classify as an agent? Any AI that I would classify as an agent? Any highly intelligent mind with broad capabilities? Any highly intelligent mind with broad capabilities that has a big effect on the world?)

Tor Økland Barstad Mar 18, 2023, 11:20 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
Fitch’s paradox of knowability and Gödel’s incompleteness theorems prove that there may be true statements that are unknowable.
Independently of Gödel’s incompleteness theorems (which I have heard of) and Fitch’s paradox of knowability (which I had not heard of), I do agree that there can be true statements that are unknown/unknowable (including relatively “simple” ones) 🙂
For example “rational goal exists” may be true and unknowable. Therefore “rational goal may exist” is true. (...) Do you agree?
I don’t think it follows from “there may be statements that are true and unknowable” that “any particular statement may be true and unknowable”.
Also, some statements may be seen as non-sensical / ill-defined / don’t have a clear meaning.
Regarding the term “rational goal”, I think it isn’t well enough specified/clarified for me to agree or disagree about whether “rational goals” exist.
In regards to Gödel’s incompleteness theorem, I suspect “rational goal” (the way you think of it) probably couldn’t be defined clearly enough to be the kind of statement that Gödel was reasoning about.
I don’t think there are universally compelling arguments (more about that here).

Tor Økland Barstad Mar 18, 2023, 10:42 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
Why do you think your starting point is better?
I guess there are different possible interpretations of “better”. I think it would be possible for software-programs to be much more mentally capable than me across most/all dimentions, and still not have “starting points” that I would consider “good” (for various interpretations of “good”).
As I understand you assume different starting-point.
I’m not sure. Like, it’s not as if I don’t have beliefs or assumptions or guesses relating to AIs. But I think I probably make less general/universal assumptions that I’d expect to hold for “all” [AIs / agents / etc].

This post is sort of relevant to my perspective 🙂

Tor Økland Barstad Mar 18, 2023, 10:23 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
In my opinion the optimal behavior is
Not sure what you mean by “optimal behavior”. I think I can see how the things make sense if the starting point is that there is this things called “goals”, and (I, the mind/agent) am motivated to optimize for “goals”. But I don’t assume this as an obvious/universal starting-point (be that for minds in general, extremely intelligent minds in general, minds in general that are very capable and might have a big influence on the universe, etc).
This is a common mistake to assume, that if you don’t know your goal, then it does not exist (...)
My perspective is that even AIs that are (what I’d think of as) utility maximizes wouldn’t necessarily think in terms of “goals”.
The examples you list are related to humans. I agree that humans often have goals that they don’t have explicit awareness of. And humans may also often have as an attitude that it makes sense to be in a position to act upon goals that they form in the future. I think that is true for more types of intelligent entities than just humans, but I don’t think it generally/always is true for “minds in general”.
Caring more about future goals you may form in the future, compared e.g. goals others may have, is not a logical necessity IMO. It may feel “obvious” to us, but what to us are obvious instincts will often not be so for all (or even most) minds in the space of possible minds.

Tor Økland Barstad Mar 18, 2023, 10:06 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
I assume you mean “provide definitions”
More or less / close enough 🙂
Agent—https://www.lesswrong.com/tag/agent
Here they write: “A rational agent is an entity which has a utility function, forms beliefs about its environment, evaluates the consequences of possible actions, and then takes the action which maximizes its utility.”
I would not share that definition, and I don’t think most other people commenting on this post would either (I know there is some irony to that, given that it’s the definition given on the LessWrong wiki).
Often the words/concepts we use don’t have clear boundaries (more about that here). I think agent is such a word/concept.
Examples of “agents” (← by my conception of the term) that don’t quite have utility functions would be humans.
How we may define “agent” may be less important if what we really are interested in is the behavior/properties of “software-programs with extreme and broad mental capabilities”.
Future states—numeric value of agent’s utility function in the future
I don’t think all extremely capable minds/machines/programs would need an explicit utility-function, or even an implicit one.
To be clear, there are many cases where I think it would be “stupid” to not act as if you have (an explicit or implicit) utility function (in some sense). But I don’t think it’s required of all extremely mentally capable systems (even if these systems are required to have logically contradictory “beliefs”).

Tor Økland Barstad Mar 18, 2023, 9:45 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
No. That’s exactly the point I try to make by saying “Orthogonality Thesis is wrong”.
Thanks for the clarification 🙂
“There is no rational goal” is an assumption in Orthogonality thesis
I suspect arriving at such a conclusion may result from thinking of utility maximizes as more of a “platonic” concept, as opposed to thinking of it from a more mechanistic angle. (Maybe I’m being too vague here, but it’s an attempt to briefly summarize some of my intuitions into words.)

I’m not sure what you would mean by “rational”. Would computer programs need to be “rational” in whichever sense you have in mind in order to be extremely capable at many mental tasks?
First, what is your opinion about this comment?
I don’t agree with it.
It is a goal which is not chosen, not assumed, it is concluded from first principles by just using logic. [from comment you reference]
There are lots of assumptions baked into it. I think you have a much too low a bar for thinking of something as a “first principle” that any capable/intelligent software-programs necessarily would adhere to by default.

Tor Økland Barstad Mar 18, 2023, 8:05 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
why would you assume that agent does not care about future states? Do you have a proof for that?
Would you be able to Taboo Your Words for “agent”, “care” and “future states”? If I were to explain my reasons for disagreement it would be helpful to have a better idea of what you mean by those terms.