A costly, but simple way would be to gather groups of SW engineers and have them work on projects where you intentionally introduce defects at various stages, and measure the costs of fixing them. To be statistically meaningful, this probably means thousands of engineer hours just to that effect.
A cheap (but not simple) way would be to go around as many companies as possible and hold the relevant measurements on actual products. This entails a lot of variables, however—engineer groups tend to work in many different ways. This might cause the data to be less than conclusive. In addition, the politics of working with existing companies may also tilt the results of such a research.
I can think of simple experiments that are not cheap; and of cheap experiments that are not simple. I’m having difficulty satisfying the conjunction and I suspect one doesn’t exist that would give a meaningful answer for high-cost bugs.
It’s not that costly if you do with university students:
Get two groups of 4 university students. One group is told “test early and often”. One group is told “test after the code is integrated”. For every bug they fix, measure the effort it is to fix it (by having them “sign a clock” for every task they do). Then, do analysis on when the bug was introduced (this seems easy post-fixing the bug, which is easy if they use something like Trac and SVN). All it takes is a month-long project that a group of 4 software engineering students can do. It seems like any university with a software engineering department can do it for the course-worth of one course. Seems to me it’s under $50K to fund?
But it can’t really be done the way you envision it. Variance in developer quality is high. Getting a meaningful result would require a lot more than 8 developers. And very few research groups can afford to run an experiment of that size—particularly since the usual experience in science is that you have to try the study a few times before you have the procedure right.
That would be cheap and simple, but wouldn’t give a meaningful answer for high-cost bugs, which don’t manifest in such small projects. Furthermore, with only eight people total, individual ability differences would overwhelmingly dominate all the other factors.
A costly, but simple way would be to gather groups of SW engineers and have them work on projects where you intentionally introduce defects at various stages, and measure the costs of fixing them. To be statistically meaningful, this probably means thousands of engineer hours just to that effect.
A cheap (but not simple) way would be to go around as many companies as possible and hold the relevant measurements on actual products. This entails a lot of variables, however—engineer groups tend to work in many different ways. This might cause the data to be less than conclusive. In addition, the politics of working with existing companies may also tilt the results of such a research.
I can think of simple experiments that are not cheap; and of cheap experiments that are not simple. I’m having difficulty satisfying the conjunction and I suspect one doesn’t exist that would give a meaningful answer for high-cost bugs.
(Minor edit: Added the missing “hours” word)
It’s not that costly if you do with university students: Get two groups of 4 university students. One group is told “test early and often”. One group is told “test after the code is integrated”. For every bug they fix, measure the effort it is to fix it (by having them “sign a clock” for every task they do). Then, do analysis on when the bug was introduced (this seems easy post-fixing the bug, which is easy if they use something like Trac and SVN). All it takes is a month-long project that a group of 4 software engineering students can do. It seems like any university with a software engineering department can do it for the course-worth of one course. Seems to me it’s under $50K to fund?
Yes, it would be nice to have such a study.
But it can’t really be done the way you envision it. Variance in developer quality is high. Getting a meaningful result would require a lot more than 8 developers. And very few research groups can afford to run an experiment of that size—particularly since the usual experience in science is that you have to try the study a few times before you have the procedure right.
That would be cheap and simple, but wouldn’t give a meaningful answer for high-cost bugs, which don’t manifest in such small projects. Furthermore, with only eight people total, individual ability differences would overwhelmingly dominate all the other factors.