Your siblings is not a reproductively isolated population (hopefully=)). The relevant question is if the helpers are more or less fit relative to the population as a whole. So in your example, where the helpers give up something and get back less, the gene goes extinct.
But start instead of just zero-sum redistribution with something like that trust game where you send money through a slot and whatever amount you send the other guy gets triple. But it’s multiplayer and simultaneous. So the helpers give up some amount, let’s say x each and every family member gets three times what the average participant gave up. If half of the family members are helpers then everyone gets 3x/2. Which is more than x, so now the gene gives a fitness advantage.
Let’s ignore the details of genetic reproduction, and simply assume that if both parents have a trait, all children have it; if no parent has a trait, no children have it; and if one parent has it, exactly 50% of children have it. Let’s assume all families have the same size. (These are quite unrealistic assumptions to make calculation simple.)
Let’s suppose that being nice to all your siblings has a cost c (for example, if without reciprocation it would reduce your survival rate by 5%, then c = 0.05), and that being supported by all your siblings provides a benefit b (for example, if without helping any your siblings but being helped by all of them would increase your survival rate by 10%, then b = 0.10). We can assume 0 < c < b.
So, the current generation contains a fraction p of adult individuals who have the sibling-helping trait. Let’s assume they form pairs randomly (because the trait is so new they haven’t developed its detectors yet). On average, there will be p^2 “helper-helper” families, 2×p×(1-p) “helper-nonhelper” families, and (1-p)^2 “nonhelper-nonhelper” families.
In “nonhelper-nonhelper” families, children’s survival rate will be 1 (the default survival rate before the helper mutation appeared). In “helper-helper” families, children’s survival rate will be 1+b-c. In “helper-nonhelper” families, the 1⁄2 of helper children will have survival rate 1+b/2-c (they only get half the help, but pay the full cost), and the 1⁄2 of nonhelper children will have survival rate 1+b/2 (they get galf the help at no cost). Now all these values together have to be normalized to 1, to get the proportions in the next generation.
next generation helpers ratio = (p + pb/2 - pc/2 + ppb/2) / (p + pb/2 - pc/2 + ppb/2 + 1 - p + pb/2 - ppb/2) = (p + pb/2 - pc/2 + ppb/2) / (1 + pb—pc/2) … which for obscure mathematical reasons is always greater than p
Well, assuming that I made no mistake during the calculation, and that my simplified assumptions about heritability of traits didn’t diverge from reality too much (two reasons why I hesitated to do the calculations myself)… I am more or less convinced this could work.
More broadly: consider genetic drift and the probability of reaching fixation. For neutral mutations, their probability of fixation is the rate at which they are introduced, and they will reach fixation at 4*population-size generations. For primate species, the population size is always pretty small, low hundreds of thousands or millions; generation turnover tends to be something like 10 years, and early primates can date back as much as 60 million years, so it can encompass a lot of drift. If we imagine that kin altruism is neutral until you have at least a few relatives and the relevant mutation keeps happening once in every few hundred thousand individuals, it’s not at all unlikely that it will appear repeatedly and then drift up to the threshold where fitness gains start appearing, and then of course, now that it’s no longer neutral, it’ll be quickly selected for at the rate of its gain.
Your siblings is not a reproductively isolated population (hopefully=)). The relevant question is if the helpers are more or less fit relative to the population as a whole. So in your example, where the helpers give up something and get back less, the gene goes extinct.
But start instead of just zero-sum redistribution with something like that trust game where you send money through a slot and whatever amount you send the other guy gets triple. But it’s multiplayer and simultaneous. So the helpers give up some amount, let’s say x each and every family member gets three times what the average participant gave up. If half of the family members are helpers then everyone gets 3x/2. Which is more than x, so now the gene gives a fitness advantage.
Here is a toy model:
Let’s ignore the details of genetic reproduction, and simply assume that if both parents have a trait, all children have it; if no parent has a trait, no children have it; and if one parent has it, exactly 50% of children have it. Let’s assume all families have the same size. (These are quite unrealistic assumptions to make calculation simple.)
Let’s suppose that being nice to all your siblings has a cost c (for example, if without reciprocation it would reduce your survival rate by 5%, then c = 0.05), and that being supported by all your siblings provides a benefit b (for example, if without helping any your siblings but being helped by all of them would increase your survival rate by 10%, then b = 0.10). We can assume 0 < c < b.
So, the current generation contains a fraction p of adult individuals who have the sibling-helping trait. Let’s assume they form pairs randomly (because the trait is so new they haven’t developed its detectors yet). On average, there will be p^2 “helper-helper” families, 2×p×(1-p) “helper-nonhelper” families, and (1-p)^2 “nonhelper-nonhelper” families.
In “nonhelper-nonhelper” families, children’s survival rate will be 1 (the default survival rate before the helper mutation appeared). In “helper-helper” families, children’s survival rate will be 1+b-c. In “helper-nonhelper” families, the 1⁄2 of helper children will have survival rate 1+b/2-c (they only get half the help, but pay the full cost), and the 1⁄2 of nonhelper children will have survival rate 1+b/2 (they get galf the help at no cost). Now all these values together have to be normalized to 1, to get the proportions in the next generation.
Ugh, math...
non-normalized next generation helpers = p^2 × (1+b-c) + 1⁄2 × 2×p×(1-p) × (1+b/2-c) = p + pb/2 - pc/2 + ppb/2
non-normalized next generation non-helpers = (1-p)^2 × 1 + 1⁄2 × 2×p×(1-p) × (1 + b/2) = 1 - p + pb/2 - ppb/2
next generation helpers ratio = (p + pb/2 - pc/2 + ppb/2) / (p + pb/2 - pc/2 + ppb/2 + 1 - p + pb/2 - ppb/2) = (p + pb/2 - pc/2 + ppb/2) / (1 + pb—pc/2) … which for obscure mathematical reasons is always greater than p
Well, assuming that I made no mistake during the calculation, and that my simplified assumptions about heritability of traits didn’t diverge from reality too much (two reasons why I hesitated to do the calculations myself)… I am more or less convinced this could work.
More broadly: consider genetic drift and the probability of reaching fixation. For neutral mutations, their probability of fixation is the rate at which they are introduced, and they will reach fixation at 4*population-size generations. For primate species, the population size is always pretty small, low hundreds of thousands or millions; generation turnover tends to be something like 10 years, and early primates can date back as much as 60 million years, so it can encompass a lot of drift. If we imagine that kin altruism is neutral until you have at least a few relatives and the relevant mutation keeps happening once in every few hundred thousand individuals, it’s not at all unlikely that it will appear repeatedly and then drift up to the threshold where fitness gains start appearing, and then of course, now that it’s no longer neutral, it’ll be quickly selected for at the rate of its gain.