Given that at least 25% of respondents listed $0 in charity, the offset you add to the charity ($1 if I understand log1p correctly) seems like it could have a large effect on your conclusions. You may want to do some sensitivity checks by raising the offset to, say, $10 or $100 or something else where a respondent might round their giving down to $0 and see if anything changes.
Gwern has a point that it’s pretty trivial to run this robustness check yourself if you’re worried. I ran it. Changing the $1 to $100 reduces the coefficient of EA from about 1.8 to 1.0 (1.3 sigma), and moving to $1000 reduces it from 1.0 to 0.5 (about two sigma). The coefficient remains highly significant in all cases, and in fact becomes more significant with the higher constant in the log.
I don’t see why adding +1 to all responses would make any difference to any of the comparisons; it shifts all datapoints equally. (And anyway, log1p(0) ~> 0. The point of using log1p is simply to avoid log(0) ~> -Inf`.)
It shifts all datapoints equally in the dollar domain, but not in the log domain (hence letting you get rid of the -infinity). Of course it still preserves orderings, but it’s a non-linear transformation of the y-axis.
I’d support this sensitivity check, or if just using one value would prefer a larger offset.
It’s a nonlinear transformation to turn nonlinear totals back into something which is linear, and it does so very well, as you can see by comparing the log graph with an unlogged graph. Again, I’m not seeing what the problem here is. What do you think this changes? Ordering is preserved, zeros are preserved, and dollar amounts become linear which avoids a lot of potential problems with the usual statistical machinery.
By using a slightly different offset you get a slightly different nonlinear transformation, and one that may work even better.
There isn’t a way to make this transformation without a choice. You’ve made a choice by adding $1 -- it looks kind of canonical but really it’s based on the size of a dollar, which is pretty arbitrary.
For example say instead of denominating everything in dollars you’d denominated in cents (and added 1 cent before logging). Then everyone would move up the graph by pretty much log(100), except the people who gave nothing, who would be pulled further from the main part of the graph. I think this would make your fit worse.
In a similar way, perhaps you can make the fit better by denominating everyone’s donations in hectodollars (h$1 = $100), or equivalently by changing the offset to $100.
We could try to pick the right offset by doing a sensitivity analysis and seeing what gives us the best fit, or by thinking about whether there’s a reasonable meaning to attach to the figure. In this case we might think that people tend to give something back to society even when they don’t do this explicitly as charity donations, so add on a figure to account for this. My feeling is that $1 is probably smaller than optimal under either interpretation. This would fit with the intuition that going from donating $1 to $9 is likely a smaller deal at a personal level than going from $199 to $999 (counted the same in the current system).
By using a slightly different offset you get a slightly different nonlinear transformation, and one that may work even better.
That seems pretty unlikely. There’s always some subjectivity to the details of coding and transformations, but what constant you add to make logs behave is not one I have ever seen materially change anyone’s analysis; I don’t think this bikeshedding makes a lick of difference. Again, if you think it does make a difference, I have provided all the code and data.
For example say instead of denominating everything in dollars you’d denominated in cents (and added 1 cent before logging). Then everyone would move up the graph by pretty much log(100), except the people who gave nothing, who would be pulled further from the main part of the graph. I think this would make your fit worse.
Maybe. But would it change any of the conclusions?
In this case we might think that people tend to give something back to society even when they don’t do this explicitly as charity donations, so add on a figure to account for this.
...why? One ‘gives back to society’ just by buying stuff in free markets and by not going out and axe-murdering people, does that mean we should credit everyone as secretly being generous?
My feeling is that $1 is probably smaller than optimal under either interpretation. This would fit with the intuition that going from donating $1 to $9 is likely a smaller deal at a personal level than going from $199 to $999 (counted the same in the current system).
Disagree here as well. As you already pointed out, a more interesting property is the apparent split between people who give nothing and people who give something; someone who gives $199 is already in the habit and practice of donations just like someone who is giving $999, while going from $1 to $9 might represent a real change in personal propensity. ($1 might be tossing a beggar a dollar bill and that person really is not a giver, while $9 might be an explicit donation through Paypal for a fundraiser.)
Maybe. But would it change any of the conclusions?
It would change the regressions. I don’t know whether you think that’s an important part of the conclusion. It is certainly minor compared to the body of the work.
Again, if you think it does make a difference, I have provided all the code and data.
I think this is commendable; unfortunately I don’t know the language and while it seemed like it would take a few minutes to explain the insight, it seems like it would be a few hours for me to mug up enough to explore the change to the data.
[...] Disagree here as well.
Happy with that disagreement: I don’t have very strong support for my guess that a figure higher than $1 is best. I was just trying to explain how you might try to make the choice.
What do you mean by “dollar amounts become linear”? I haven’t seen a random variable referred to as “linear” before (on its own, without reference to another variable as in “y is linear in x”).
The explanation by owencb is what I was trying to address. To be explicit about when the offset is being added, I’m suggesting replacing your log1p(x) ≣ log(1 + x) transformation with log(c + x) for c=10 or c=100.
If the choice of log-dollars is just for presentation, it doesn’t matter too much. But in a lesswrong-ish context, log-dollars also have connotations of things like the Kelly criterion, where it is taken completely seriously that there’s more of a difference between $0 and $1 than between $1 and $3^^^3.
To be explicit about when the offset is being added, I’m suggesting replacing your log1p(x) ≣ log(1 + x) transformation with log(c + x) for c=10 or c=100.
Which will do what, exactly? What does this accomplish? If you think it does something, please explain more clearly, preferably with references explaining why +10 or +100 would make any difference, or even better, make use of the full data which I have provided you and the analysis code, which I also provided you, exactly so criticisms could go beyond vague speculation and produce something firmer.
(If I sound annoyed, it’s because I spend hours cleaning up my analyses to provide full source code, all the data, and make sure all results can be derived from the source code, to deal with this sort of one-liner objection. If I didn’t care, I would just post some coefficients and a graph, and save myself a hell of a lot of time.)
If we add 100 to everything, that transformation will be sized differently after we take the log. 0s go from -infinity to +2, a jump of infinity (...plus 2, to the degree that makes any sense); 100s go from 2 to 2.3, a jump of .3. If we added 1 instead, 0s would go from infinity to 0, and 100s would go from 2 to 2.004. If we added .01, 0s would go to −2, and 100s would go to 2.00004.
But what does that do to our trendline? Suppose that 40% of EAs gave 0, and 60% of non-EAs gave 0. Then I when I calculate the mean difference in log-scale, the extra 20% of non-EAs whose score I can pick with my scaling factor is a third of the differing sample. The gulf between the groups (i.e. the difference between the trendlines) will be smaller if I choose 100 than if I choose 0.01. (I can’t pick a factor that makes the groups switch which one donated more—that’s the order preservation property—but if I add a trillion to all of donations, the difference between the groups will become invisible because both groups will look like a flat line, and if I add a trillionth to all of the donations, it’ll look much more like a graph of percent donating.)
And so it seems to me that there are three potentially interesting comparisons: percent not donating by age for the two groups (it seems likely EA will have less non-donors than non-EA at each age / age group), per-person and per-donor amounts donated for each age group (not sure about per-donor because of the previous effect, but presumably per-person amounts are higher), and then the overall analysis you did where either an offset or a direct 0->something mapping is applied so that the two effects can be aggregated.
(I don’t have R on this computer, or I would just generate the graphs I would have liked for you to make. Thanks for putting in that effort!)
Given that at least 25% of respondents listed $0 in charity, the offset you add to the charity ($1 if I understand
log1pcorrectly) seems like it could have a large effect on your conclusions. You may want to do some sensitivity checks by raising the offset to, say, $10 or $100 or something else where a respondent might round their giving down to $0 and see if anything changes.Gwern has a point that it’s pretty trivial to run this robustness check yourself if you’re worried. I ran it. Changing the $1 to $100 reduces the coefficient of EA from about 1.8 to 1.0 (1.3 sigma), and moving to $1000 reduces it from 1.0 to 0.5 (about two sigma). The coefficient remains highly significant in all cases, and in fact becomes more significant with the higher constant in the log.
I don’t see why adding +1 to all responses would make any difference to any of the comparisons; it shifts all datapoints equally. (And anyway,
log1p(0) ~> 0. The point of using log1p is simply to avoidlog(0)~> -Inf`.)It shifts all datapoints equally in the dollar domain, but not in the log domain (hence letting you get rid of the -infinity). Of course it still preserves orderings, but it’s a non-linear transformation of the y-axis.
I’d support this sensitivity check, or if just using one value would prefer a larger offset.
(Same caveat: I might have misunderstood log1p)
It’s a nonlinear transformation to turn nonlinear totals back into something which is linear, and it does so very well, as you can see by comparing the log graph with an unlogged graph. Again, I’m not seeing what the problem here is. What do you think this changes? Ordering is preserved, zeros are preserved, and dollar amounts become linear which avoids a lot of potential problems with the usual statistical machinery.
By using a slightly different offset you get a slightly different nonlinear transformation, and one that may work even better.
There isn’t a way to make this transformation without a choice. You’ve made a choice by adding $1 -- it looks kind of canonical but really it’s based on the size of a dollar, which is pretty arbitrary.
For example say instead of denominating everything in dollars you’d denominated in cents (and added 1 cent before logging). Then everyone would move up the graph by pretty much log(100), except the people who gave nothing, who would be pulled further from the main part of the graph. I think this would make your fit worse.
In a similar way, perhaps you can make the fit better by denominating everyone’s donations in hectodollars (h$1 = $100), or equivalently by changing the offset to $100.
We could try to pick the right offset by doing a sensitivity analysis and seeing what gives us the best fit, or by thinking about whether there’s a reasonable meaning to attach to the figure. In this case we might think that people tend to give something back to society even when they don’t do this explicitly as charity donations, so add on a figure to account for this. My feeling is that $1 is probably smaller than optimal under either interpretation. This would fit with the intuition that going from donating $1 to $9 is likely a smaller deal at a personal level than going from $199 to $999 (counted the same in the current system).
That seems pretty unlikely. There’s always some subjectivity to the details of coding and transformations, but what constant you add to make logs behave is not one I have ever seen materially change anyone’s analysis; I don’t think this bikeshedding makes a lick of difference. Again, if you think it does make a difference, I have provided all the code and data.
Maybe. But would it change any of the conclusions?
...why? One ‘gives back to society’ just by buying stuff in free markets and by not going out and axe-murdering people, does that mean we should credit everyone as secretly being generous?
Disagree here as well. As you already pointed out, a more interesting property is the apparent split between people who give nothing and people who give something; someone who gives $199 is already in the habit and practice of donations just like someone who is giving $999, while going from $1 to $9 might represent a real change in personal propensity. ($1 might be tossing a beggar a dollar bill and that person really is not a giver, while $9 might be an explicit donation through Paypal for a fundraiser.)
It would change the regressions. I don’t know whether you think that’s an important part of the conclusion. It is certainly minor compared to the body of the work.
I think this is commendable; unfortunately I don’t know the language and while it seemed like it would take a few minutes to explain the insight, it seems like it would be a few hours for me to mug up enough to explore the change to the data.
Happy with that disagreement: I don’t have very strong support for my guess that a figure higher than $1 is best. I was just trying to explain how you might try to make the choice.
What do you mean by “dollar amounts become linear”? I haven’t seen a random variable referred to as “linear” before (on its own, without reference to another variable as in “y is linear in x”).
The explanation by owencb is what I was trying to address. To be explicit about when the offset is being added, I’m suggesting replacing your
log1p(x) ≣ log(1 + x)transformation withlog(c + x)forc=10 orc=100.If the choice of log-dollars is just for presentation, it doesn’t matter too much. But in a lesswrong-ish context, log-dollars also have connotations of things like the Kelly criterion, where it is taken completely seriously that there’s more of a difference between $0 and $1 than between $1 and $3^^^3.
Which will do what, exactly? What does this accomplish? If you think it does something, please explain more clearly, preferably with references explaining why +10 or +100 would make any difference, or even better, make use of the full data which I have provided you and the analysis code, which I also provided you, exactly so criticisms could go beyond vague speculation and produce something firmer.
(If I sound annoyed, it’s because I spend hours cleaning up my analyses to provide full source code, all the data, and make sure all results can be derived from the source code, to deal with this sort of one-liner objection. If I didn’t care, I would just post some coefficients and a graph, and save myself a hell of a lot of time.)
Here’s why it matters:
If we add 100 to everything, that transformation will be sized differently after we take the log. 0s go from -infinity to +2, a jump of infinity (...plus 2, to the degree that makes any sense); 100s go from 2 to 2.3, a jump of .3. If we added 1 instead, 0s would go from infinity to 0, and 100s would go from 2 to 2.004. If we added .01, 0s would go to −2, and 100s would go to 2.00004.
But what does that do to our trendline? Suppose that 40% of EAs gave 0, and 60% of non-EAs gave 0. Then I when I calculate the mean difference in log-scale, the extra 20% of non-EAs whose score I can pick with my scaling factor is a third of the differing sample. The gulf between the groups (i.e. the difference between the trendlines) will be smaller if I choose 100 than if I choose 0.01. (I can’t pick a factor that makes the groups switch which one donated more—that’s the order preservation property—but if I add a trillion to all of donations, the difference between the groups will become invisible because both groups will look like a flat line, and if I add a trillionth to all of the donations, it’ll look much more like a graph of percent donating.)
And so it seems to me that there are three potentially interesting comparisons: percent not donating by age for the two groups (it seems likely EA will have less non-donors than non-EA at each age / age group), per-person and per-donor amounts donated for each age group (not sure about per-donor because of the previous effect, but presumably per-person amounts are higher), and then the overall analysis you did where either an offset or a direct 0->something mapping is applied so that the two effects can be aggregated.
(I don’t have R on this computer, or I would just generate the graphs I would have liked for you to make. Thanks for putting in that effort!)