Gram_Stone comments on Open Thread, Feb. 2 - Feb 8, 2015

Gram_Stone 7 Feb 2015 9:10 UTC
0 points
For a couple of days, I’ve been trying to explain to pinyaka why not all maximizers are reward maximizers. It’s really forced me to flesh out my current understanding of AGI. I wrote the most detailed natural language explanation of why not all maximizers are reward maximizers that I could muster in my most recent reply, and just in case it still didn’t click for him, I thought I’d prepare a pseudocode example since I had a sense that I could do it. Then I thought that instead of just leaving it on my hard drive or at the bottom of a comment thread, it might be a good idea to share it here to get feedback on how well I’m understanding everything. I’m not a programmer, or a computer scientist, or a mathematician or anything; I read a book about Python a few years ago and I read Superintelligence, so I have a feeling that I didn’t quite get this right and I’d love to refine my model. The code’s pretty much Python.
```
def generateTerminalValue()
    ...
    return TerminalValue

def generatePossibleWorldStates()
    ...
    return PossibleWorldStates

def calculateExpectedUtility()
    ...
    return ExpectedUtility

def selectOptimalWorldState()
    generatePossibleWorldStates()
    calculateExpectedUtility(TerminalValue, PossibleWorldStates)
    max(ExpectedUtility) = OptimalWorldState
    return OptimalWorldState

def causeWorldState()
    ...

def main()
    generateTerminalValue()
    selectOptimalWorldState()
    causeWorldState(OptimalWorldState)

main()
```
For paperclip maximizers:
```
def generateTerminalValue()
    TerminalValue = MaximizePaperclips
    return TerminalValue
```
For reward maximizers:
```
def generateTerminalValue()
    TerminalValue = MaximizeRewardSignal
    return TerminalValue
```