We start by noting that human values are complex and hard to fully formalize, in a way that leads to good things (or even “just” avoids bad things) when maximized by a utility function.
This sentence sounds so ambiguous, I think I understand it because I already assume what you mean. I guess you mean that “it is difficult to (formalize it in a way such that)”, and not that “(it is difficult to formalize) leads to good things”. Also, I guess that “when maximized by a utility function” means “when you try to maximize them by maximizing an utility function”.
This is getting unwieldy, so we change Loc_n and simplify. Now, Loc_n is just a distribution over “counterfactual insertion functions”, i.e. functions that produce coutnerfactual worldstates from counterfactual input blobs. A single such function would be denoted GAMMA, of type / in set UPPERGAMMA_n (again, n is the size of the blob being dealt with).
Loc_n(w in UPPEROMEGA, b of length n) = a distribution over halting/deterministic/non-crazy programs that eat w and give a bitstring of length n. If f(w) = b, then our distribution puts weight the KSIMP(f) as the weight on that f. Otherwise, we weight it 0, since f didn’t give us b in the first place.
(The following math is sometimes on a Conway’s Game Of Life cellular-automata grid, which could debatably generalize>>s<< to our actual universe, or at least can help to model it. Have I mentioned that the “simulate universes a lot” mental image is going to help here?)
HOW_GOOD is a function with input [A, an action [and a hypothesis]] and output [0;1].
Score(action) is another extended sum. For every h in HYPOTHESIS, we calculate PRIOR(h) * LOOKS_LIKE_THIS_WORLD(h) * HOW_GOOD(action, h). Sum them up, and we get Score(action).
An idea Tammy uses frequently: Types are sets. That is, a type-signature for an entity says “the entity is within the set of all [whatever]s”. Yes of course the distinction is unclear at this very moment. Tammy just invented QACI within the past year, it’s still a work in progress, and even the background math’s probably not all done. More to the point, I haven’t described most of it yet! Again, any problems that could arise are left to part 2 of this post.
I do not understand the overall meaning of all sentences from the second onwards in this paragraph.
First, we start setting up math entities. Let UPPEROMEGA be a set of worldstates, DELTA_UPPEROMEGA is the set of distributions over UPPEROMEGA, and A is a set of actions.
TYPO THREAD
This sentence sounds so ambiguous, I think I understand it because I already assume what you mean. I guess you mean that “it is difficult to (formalize it in a way such that)”, and not that “(it is difficult to formalize) leads to good things”. Also, I guess that “when maximized by a utility function” means “when you try to maximize them by maximizing an utility function”.
I do not understand the overall meaning of all sentences from the second onwards in this paragraph.
Missing math formatting henceforth
After finishing the post, I infer that your way or writing math is idiosyncratic, and not lack of formatting.
It does not look clearer to me than more standard math notation, although I guess it may be easier to write for you?
Misformatting?