Did you set up your previous posts with this experiment in mind? I had been toying with some similar things myself, so it’s fun to see your analysis too.
Regarding choices that might not generalize, another one that I have been thinking about is agency/optimization. That is, while your model is super relevant for inanimate systems, optimized systems often try to keep some parameters under control, which involves many parameters being carefully adjusted to achieve that.
This might seem difficult to model, but I have an idea: Suppose you pick a pair of causal nodes X and Y, such that X is indirectly upstream from Y. In that case, since you have the ground truth for the causal network, you can compute how to adjust the weights of X’s inputs and outputs to (say) keep Y as close to 0 as possible. This probably won’t make much of a difference when picking only a single X, but for animate systems a very large number of nodes might be set for the purpose of optimization. There is a very important way that this changes the general abstract behavior of a system, which unfortunately nobody seems to have written about, though I’m not sure if this has any implications for abstraction.
Also, another property of the real world (that is relatively orthogonal to animacy, but which might interact with animacy in interesting ways) that you might want to think about is recurrence. You’ve probably already considered this, but I think it would be interesting to study what happens if the causal structure tiles over time and/or space.
If matrix A maps each input vector of X to a vector of which the first entry corresponds to Y, subtracting multiples of the first row from every other row to make them orthogonal to the first row, then deleting the first row, would leave a matrix whose row space is the input vectors that keep Y at 0, and whose column space is the outputs thus still reachable. If you fix some distribution on the inputs of X (such as the normal distribution with a given covariance matrix), whether this is losslessly possible should be more interesting.
Presumably you wouldn’t be able to figure out the precise value of Y since Y isn’t connected to X. You could only find an approximate estimate. Though on reflection the outputs are more interesting in a nonlinear graph (which was the context where I originally came up with the idea), since in a linear one all ways of modifying Y are equivalent.
All ways of modifying Y are only equivalent in a dense linear system. Sparsity (in a high-dimensional system) changes that. (That’s a fairly central concept behind this whole project: sparsity is one of the main ingredients necessary for the natural abstraction hypothesis.)
Suppose Y is unidimensional, and you have Y=f(g(X), h(X)). Suppose there are two perturbations i and j that X can emit, where g is only sensitive to i and h is only sensitive to j, i.e. g(j)=0, h(i)=0. Then because the system is linear, you can extract them from the rest:
This means that if X only cares about Y, it is free to choose whether to adjust a or to adjust b. In a nonlinear system, there might be all sorts of things like moderators, diminishing returns, etc., which would make it matter whether it tried to control Y using a or using b; but in a linear system, it can just do whatever.
Oh I see. Yeah, if either X or Y is unidimensional, then any linear model is really boring. They need to be high-dimensional to do anything interesting.
They need to be high-dimensional for the linear models themselves to do anything interesting, but I think adding a large number of low-dimensional linear models might, despite being boring, still change the dynamics of the graphs to be marginally more realistic for settings involving optimization. X turns into an estimate of Y, and tries to control this estimate towards zero; that’s a pattern that I assume would be rare in your graph, but common in reality, and it could lead to real graphs exhibiting certain “conspiracies” that the model graphs might lack (especially if there are many (X, Y) pairs, or many (individually unidimensional) Xs that all try to control a single common Y).
But there’s probably a lot of things that can be investigated about this. I should probably be working on getting my system for this working, or something. Gonna be exciting to see what else you figure out re natural abstractions.
Y can consist of multiple variables, and then there would always be multiple ways, right? I thought by indirect you meant that the path between X and Y was longer than 1. If some third cause is directly upstream from both, then I suppose it wouldn’t be uniquely defined whether changing X changes Y, since there could be directions in which to change the cause that change some subset of X and Y.
Y can consist of multiple variables, and then there would always be multiple ways, right?
Not necessarily. For instance if X has only one output, then there’s only one way for X to change things, even if the one output connects to multiple Ys.
I thought by indirect you meant that the path between X and Y was longer than 1.
Yes.
If some third cause is directly upstream from both, then I suppose it wouldn’t be uniquely defined whether changing X changes Y, since there could be directions in which to change the cause that change some subset of X and Y.
I’m not sure I get it, or at least if I get it I don’t agree.
Are you saying that if we’ve got X ← Z → Y and X → Y, then the effect of X on Y may not be well-defined, because it depends on whether the effect is through Z or not, as the Z → Y path becomes relevant when it is through Z?
Because if so I think I disagree. The effect of X on Y should only count the X → Y path, not the X ← Z → Y path, as the latter is a confounder rather than a true causal path.
The main thing I expect from optimization is that the system will be adjusted so that certain specific abstractions work—i.e. if the optimizer cares about f(X), then the system will be adjusted so that information about f(X) is available far away (i.e. wherever it needs to be used).
That’s the view in which we think about abstractions in the optimized system. If we instead take our system to be the whole optimization process, then we expect many variables in many places to be adjusted to optimize the objective, which means the objective itself is likely a natural abstraction, since information about it is available all over the place. I don’t have all the math worked out for that yet, though.
Did you set up your previous posts with this experiment in mind? I had been toying with some similar things myself, so it’s fun to see your analysis too.
Regarding choices that might not generalize, another one that I have been thinking about is agency/optimization. That is, while your model is super relevant for inanimate systems, optimized systems often try to keep some parameters under control, which involves many parameters being carefully adjusted to achieve that.
This might seem difficult to model, but I have an idea: Suppose you pick a pair of causal nodes X and Y, such that X is indirectly upstream from Y. In that case, since you have the ground truth for the causal network, you can compute how to adjust the weights of X’s inputs and outputs to (say) keep Y as close to 0 as possible. This probably won’t make much of a difference when picking only a single X, but for animate systems a very large number of nodes might be set for the purpose of optimization. There is a very important way that this changes the general abstract behavior of a system, which unfortunately nobody seems to have written about, though I’m not sure if this has any implications for abstraction.
Also, another property of the real world (that is relatively orthogonal to animacy, but which might interact with animacy in interesting ways) that you might want to think about is recurrence. You’ve probably already considered this, but I think it would be interesting to study what happens if the causal structure tiles over time and/or space.
If matrix A maps each input vector of X to a vector of which the first entry corresponds to Y, subtracting multiples of the first row from every other row to make them orthogonal to the first row, then deleting the first row, would leave a matrix whose row space is the input vectors that keep Y at 0, and whose column space is the outputs thus still reachable. If you fix some distribution on the inputs of X (such as the normal distribution with a given covariance matrix), whether this is losslessly possible should be more interesting.
Presumably you wouldn’t be able to figure out the precise value of Y since Y isn’t connected to X. You could only find an approximate estimate. Though on reflection the outputs are more interesting in a nonlinear graph (which was the context where I originally came up with the idea), since in a linear one all ways of modifying Y are equivalent.
All ways of modifying Y are only equivalent in a dense linear system. Sparsity (in a high-dimensional system) changes that. (That’s a fairly central concept behind this whole project: sparsity is one of the main ingredients necessary for the natural abstraction hypothesis.)
I think I phrased it wrong/in a confusing way.
Suppose Y is unidimensional, and you have Y=f(g(X), h(X)). Suppose there are two perturbations i and j that X can emit, where g is only sensitive to i and h is only sensitive to j, i.e. g(j)=0, h(i)=0. Then because the system is linear, you can extract them from the rest:
Y=f(g(X+ai+bj), h(X+ai+bj))=f(g(X), h(X))+af(g(i))+bf(h(j))
This means that if X only cares about Y, it is free to choose whether to adjust a or to adjust b. In a nonlinear system, there might be all sorts of things like moderators, diminishing returns, etc., which would make it matter whether it tried to control Y using a or using b; but in a linear system, it can just do whatever.
Oh I see. Yeah, if either X or Y is unidimensional, then any linear model is really boring. They need to be high-dimensional to do anything interesting.
They need to be high-dimensional for the linear models themselves to do anything interesting, but I think adding a large number of low-dimensional linear models might, despite being boring, still change the dynamics of the graphs to be marginally more realistic for settings involving optimization. X turns into an estimate of Y, and tries to control this estimate towards zero; that’s a pattern that I assume would be rare in your graph, but common in reality, and it could lead to real graphs exhibiting certain “conspiracies” that the model graphs might lack (especially if there are many (X, Y) pairs, or many (individually unidimensional) Xs that all try to control a single common Y).
But there’s probably a lot of things that can be investigated about this. I should probably be working on getting my system for this working, or something. Gonna be exciting to see what else you figure out re natural abstractions.
Y can consist of multiple variables, and then there would always be multiple ways, right? I thought by indirect you meant that the path between X and Y was longer than 1. If some third cause is directly upstream from both, then I suppose it wouldn’t be uniquely defined whether changing X changes Y, since there could be directions in which to change the cause that change some subset of X and Y.
Not necessarily. For instance if X has only one output, then there’s only one way for X to change things, even if the one output connects to multiple Ys.
Yes.
I’m not sure I get it, or at least if I get it I don’t agree.
Are you saying that if we’ve got X ← Z → Y and X → Y, then the effect of X on Y may not be well-defined, because it depends on whether the effect is through Z or not, as the Z → Y path becomes relevant when it is through Z?
Because if so I think I disagree. The effect of X on Y should only count the X → Y path, not the X ← Z → Y path, as the latter is a confounder rather than a true causal path.
The main thing I expect from optimization is that the system will be adjusted so that certain specific abstractions work—i.e. if the optimizer cares about f(X), then the system will be adjusted so that information about f(X) is available far away (i.e. wherever it needs to be used).
That’s the view in which we think about abstractions in the optimized system. If we instead take our system to be the whole optimization process, then we expect many variables in many places to be adjusted to optimize the objective, which means the objective itself is likely a natural abstraction, since information about it is available all over the place. I don’t have all the math worked out for that yet, though.