Maybe we have different things in mind for “aggressive policy”. I was think something like “give the AI enough computing power to achieve superhuman intelligence so it can hopefully build a full-fledged FAI for the user” vs the “conservative policy” of “keep the AI at its current level where it seems safe, and find another way to build an FAI”.
Your examples of possible mistakes seemed to involve not knowing how the alien would feel about particular tradeoffs. This doesn’t seem related to how much computational power you have, except insofar as having more power might lead you to believe that it is safe to try and figure out what the alien thinks from first principles. But that’s not a necessary consequence of having more computing power, and I gave an argument that more computing power shouldn’t predictably lead to trouble.
Why do you think that more computing power requires a strategy which is “aggressive” in the sense of having a higher probability of catastrophic failure?
You might expect that building “full-fledged FAI” requires knowing a lot about the alien, and you won’t be able to figure all of that out in advance of building it. But again, I don’t understand why you can’t build an AI that implements a conservative strategy, in the sense of being quick to consult the user and unlikley to make a catastrophic error. So it seems like this just begs the question about the relative efficacy of conservative vs. aggressive strategies.
Your examples of possible mistakes seemed to involve not knowing how the alien would feel about particular tradeoffs. This doesn’t seem related to how much computational power you have, except insofar as having more power might lead you to believe that it is safe to try and figure out what the alien thinks from first principles. But that’s not a necessary consequence of having more computing power, and I gave an argument that more computing power shouldn’t predictably lead to trouble.
Why do you think that more computing power requires a strategy which is “aggressive” in the sense of having a higher probability of catastrophic failure?
You might expect that building “full-fledged FAI” requires knowing a lot about the alien, and you won’t be able to figure all of that out in advance of building it. But again, I don’t understand why you can’t build an AI that implements a conservative strategy, in the sense of being quick to consult the user and unlikley to make a catastrophic error. So it seems like this just begs the question about the relative efficacy of conservative vs. aggressive strategies.