I agree with tut that increasing speed might help. Sometimes if I listen at default speed, I find my attention drifting off mid-sentence just because it’s going so slowly. (Conversely, at higher speed, when my attention does drift off briefly, I sometimes miss a full sentence or two and have to rewind slightly.)
If that doesn’t work, I don’t really have many other ideas. Maybe you could try other repetitive mechanical actions to see if they coexist well with audiobooks. For example, maybe cooking, drawing, or exercising might work (if you do any of those). In general, I find it easy to not miss anything in an audiobook so long as I’m simultaneously doing something that does not also involve words.
I was supposed to check on this a long time ago, but forgot/went inactive on LW, but the post actually ended up at −26, so seemingly slightly lower than it was, which is evidence against your regression to 0 theory.