[deleted] comments on FAI Research Constraints and AGI Side Effects

[[deleted]]

Interesting! How does that compare to the usual implementations of planning as probabilistic inference, as exemplified below?

(query
. (define a (prior))
. (define x (lambda (a) (world a)))
. (define r (flip (sigmoid (U (x a)))))
. a
. (= r #t))

[jessicat]

Your model selects an action proportional to p(a) E[sigmoid(U) | a], whereas mine selects an action proportional to p(a) e^E[U | a]. I think the second is better, because it actually treats actions the same if they have the same expected utility. The sigmoid version will not take very high utilities or very low utilities into account much.

Btw it’s also possible to select an action proportional to E[U | a]^n:

query {
. a ~ p()
. for i = 1 to n
. . x_i ~ P(x)
. . factor(log U(x, a))
}

[[deleted]]

Could you explain your syntax here? What probabilistic programming language are you using?

I think the second is better, because it actually treats actions the same if they have the same expected utility.

Well so does the sigmoided version, but you are right that the sigmoid version won’t take very high or very low utilities into account. It’s meant to shoehorn unbounded utility functions into a framework where one normally works only with random variables.

[jessicat]

It’s not a specific programming language, I guess it’s meant to look like Church. It could be written as:

(query
. (define a (p))
. (foreach (range n) (lambda i)
. . (define x (x-prior))
. . (factor (log (U x a)))))

Well so does the sigmoided version

It samples an action proportional to p(a) E[sigmoid(U) | a]. This can’t be written as a function of E[U | a].