I think that’s fully compatible with either possibility. If Voldemort studied there, then he would have reason to destroy it; to not “leave the source of his power lying around”. But if, on the other hand, he didn’t study there (because he was refused), then he would again have a reason to not leave a source of power lying around. (If I can’t have it, no one can.)
Implied in Chapter 49, Prior Information, when Harry and Quirrell are discussing Slytherin’s monster:
I think that’s fully compatible with either possibility. If Voldemort studied there, then he would have reason to destroy it; to not “leave the source of his power lying around”. But if, on the other hand, he didn’t study there (because he was refused), then he would again have a reason to not leave a source of power lying around. (If I can’t have it, no one can.)