It’s not a binary. You can perform explicit optimization over high-level plan features, then hand off detailed execution to learned heuristics. “Make coffee” may be part of an optimized stratagem computed via consequentialism, but you don’t have to consciously optimize every single muscle movement once you’ve decided on that goal.
Essentially, what counts as “outputs” or “direct actions” relative to the consequentialist-planner is flexible, and every sufficiently-reliable (chain of) learned heuristics can be put in that category, with choosing to execute one of them available to the planner algorithm as a basic output.
In fact, I’m pretty sure that’s how humans work most of the time. We use the general-intelligence machinery to “steer” ourselves at a high level, and most of the time, we operate on autopilot.
In fact, I’m pretty sure that’s how humans work most of the time. We use the general-intelligence machinery to “steer” ourselves at a high level, and most of the time, we operate on autopilot.
Yeah, I agree with this. But I don’t think the human system aggregates into any kind of coherent total optimiser. Humans don’t have an objective function (not even approximately?).
A human is not well modelled as a wrapper mind; do you disagree?
A human is not well modelled as a wrapper mind; do you disagree?
Certainly agree. That said, I feel the need to lay out my broader model here. The way I see it, a “wrapper-mind” is a general-purpose problem-solving algorithm hooked up to a static value function. As such:
Are humans proper wrapper-minds? No, certainly not.
Do humans have the fundamental machinery to be wrapper-minds? Yes.
Is any individual run of a human general-purpose problem-solving algorithm essentially equivalent to wrapper-mind-style reasoning? Yes.
Can humans choose to act as wrapper-minds on longer time scales? Yes, approximately, subject to constraints like force of will.
Do most humans, in practice, choose to act as wrapper-minds? No, we switch our targets all the time, value drift is ubiquitous.
Is it desirable for a human to act as a wrapper-mind? That’s complicated.
On the one hand, yes because consistent pursuit of instrumentally convergent goals would lead to you having more resources to spend on whatever values you have.
On the other hand, no because we terminally value this sort of value-drift and self-inconsistency, it’s part of “being human”.
In sum, for humans, there’s a sort of tradeoff between approximating a wrapper-mind, and being an incoherent human, and different people weight it differently in different context. E. g., if you really want to achieve something (earning your first million dollars, averting extinction), and you value it more than having fun being a human, you may choose to act as a wrapper-mind in the relevant context/at the relevant scale.
As such: humans aren’t wrapper-minds, but they can act like them, and it’s sometimes useful to act as one.
It’s not a binary. You can perform explicit optimization over high-level plan features, then hand off detailed execution to learned heuristics. “Make coffee” may be part of an optimized stratagem computed via consequentialism, but you don’t have to consciously optimize every single muscle movement once you’ve decided on that goal.
Essentially, what counts as “outputs” or “direct actions” relative to the consequentialist-planner is flexible, and every sufficiently-reliable (chain of) learned heuristics can be put in that category, with choosing to execute one of them available to the planner algorithm as a basic output.
In fact, I’m pretty sure that’s how humans work most of the time. We use the general-intelligence machinery to “steer” ourselves at a high level, and most of the time, we operate on autopilot.
Yeah, I agree with this. But I don’t think the human system aggregates into any kind of coherent total optimiser. Humans don’t have an objective function (not even approximately?).
A human is not well modelled as a wrapper mind; do you disagree?
Certainly agree. That said, I feel the need to lay out my broader model here. The way I see it, a “wrapper-mind” is a general-purpose problem-solving algorithm hooked up to a static value function. As such:
Are humans proper wrapper-minds? No, certainly not.
Do humans have the fundamental machinery to be wrapper-minds? Yes.
Is any individual run of a human general-purpose problem-solving algorithm essentially equivalent to wrapper-mind-style reasoning? Yes.
Can humans choose to act as wrapper-minds on longer time scales? Yes, approximately, subject to constraints like force of will.
Do most humans, in practice, choose to act as wrapper-minds? No, we switch our targets all the time, value drift is ubiquitous.
Is it desirable for a human to act as a wrapper-mind? That’s complicated.
On the one hand, yes because consistent pursuit of instrumentally convergent goals would lead to you having more resources to spend on whatever values you have.
On the other hand, no because we terminally value this sort of value-drift and self-inconsistency, it’s part of “being human”.
In sum, for humans, there’s a sort of tradeoff between approximating a wrapper-mind, and being an incoherent human, and different people weight it differently in different context. E. g., if you really want to achieve something (earning your first million dollars, averting extinction), and you value it more than having fun being a human, you may choose to act as a wrapper-mind in the relevant context/at the relevant scale.
As such: humans aren’t wrapper-minds, but they can act like them, and it’s sometimes useful to act as one.