Also intuitively, in the latter case 5 of the data points “didn’t matter” in that you’d have had the same constraints (at that point) without them, and so this is kinda sorta like “information loss”.
I am confused: how can this be “information loss” when we are assuming that due to linear dependence of the data points, we necessarily have 5 extra dimensions where the loss is the same? Because 5 of the data points “didn’t matter”, that shouldn’t count as “information loss” but more like “redundant data, ergo no information transmitted”.
I am confused: how can this be “information loss” when we are assuming that due to linear dependence of the data points, we necessarily have 5 extra dimensions where the loss is the same? Because 5 of the data points “didn’t matter”, that shouldn’t count as “information loss” but more like “redundant data, ergo no information transmitted”.
I agree “information loss” seems kinda sketchy as a description of this phenomenon, it’s not what I would have chosen.