What might it look like to systematize the search strategy that returns blindspots?
A few years ago I wrote about one strategy for this, based on an example I ran into in the wild. We had some statistics on new user signups for an app; day-to-day variation in signup rate looked random. Assuming that each user decides whether to signup independently of all the other users, the noise in total signup count N should be ~√N (ignoring a constant factor). But the actual day-to-day variability was way larger than that—therefore there had to be some common factor influencing people. We had identified an unknown unknown. (Turned out, our servers didn’t have enough capacity, and would sometimes get backed up. Whenever that happened, signups dropped very low. So we added servers, and signup rate improved.)
The link talks a bit about how to generalize that strategy, although it’s still far from a universal technique.
Yup, that’s from my review of Design Principles of Biological Circuits.
A few years ago I wrote about one strategy for this, based on an example I ran into in the wild. We had some statistics on new user signups for an app; day-to-day variation in signup rate looked random. Assuming that each user decides whether to signup independently of all the other users, the noise in total signup count N should be ~√N (ignoring a constant factor). But the actual day-to-day variability was way larger than that—therefore there had to be some common factor influencing people. We had identified an unknown unknown. (Turned out, our servers didn’t have enough capacity, and would sometimes get backed up. Whenever that happened, signups dropped very low. So we added servers, and signup rate improved.)
The link talks a bit about how to generalize that strategy, although it’s still far from a universal technique.