It shouldn’t be too challenging to apply Nelson rules to 100k lines, but the point of statistical process control is continuous monitoring—if you weigh yourself every day, you would look at the two-week trend every day, for example. Writing a script that checks if any of these rules are violated and emails you the graph if that’s true seems simple and potentially useful.
I think what Elo’s friends mean is that the constants hard-coded into Nelson’s rules reflect some assumption on sample size. With a big sample, you’ll violate them all the time and it won’t mean anything. But they are a good starting point for tuning the thresholds.
I think what Elo’s friends mean is that the constants hard-coded into Nelson’s rules reflect some assumption on sample size. With a big sample, you’ll violate them all the time and it won’t mean anything. But they are a good starting point for tuning the thresholds.
If you have many parallel sensors, then yes, a flag that occurs 5% of the time due to noise will flag on at least one twentieth of your sensors. Elo’s point, as I understood it, was that they have a long history—which is not relevant to the applicability of SPC.
The long history is not relevant, but the frequency. Most of Nelson’s rules are 1/1000 events. If you don’t expect trends to change more often than 1000 measurements, that’s too sensitive. I don’t know what Elo is measuring every minute, but that probably is too sensitive and most of the hits will be false positives. (Actually, many things will have daily cycles. If Nelson notices them, that’s interesting, but after removing such patterns, it will probably be too sensitive.)
I see what you mean—yes, if you’re oversampling, you need to downsample / average / whatever to get to the reasonable frequency, otherwise you’ll just be doing SPC on noise.
It shouldn’t be too challenging to apply Nelson rules to 100k lines, but the point of statistical process control is continuous monitoring—if you weigh yourself every day, you would look at the two-week trend every day, for example. Writing a script that checks if any of these rules are violated and emails you the graph if that’s true seems simple and potentially useful.
I think what Elo’s friends mean is that the constants hard-coded into Nelson’s rules reflect some assumption on sample size. With a big sample, you’ll violate them all the time and it won’t mean anything. But they are a good starting point for tuning the thresholds.
If you have many parallel sensors, then yes, a flag that occurs 5% of the time due to noise will flag on at least one twentieth of your sensors. Elo’s point, as I understood it, was that they have a long history—which is not relevant to the applicability of SPC.
The long history is not relevant, but the frequency. Most of Nelson’s rules are 1/1000 events. If you don’t expect trends to change more often than 1000 measurements, that’s too sensitive. I don’t know what Elo is measuring every minute, but that probably is too sensitive and most of the hits will be false positives. (Actually, many things will have daily cycles. If Nelson notices them, that’s interesting, but after removing such patterns, it will probably be too sensitive.)
I see what you mean—yes, if you’re oversampling, you need to downsample / average / whatever to get to the reasonable frequency, otherwise you’ll just be doing SPC on noise.