Frequently to both.
For fiction that I re-read and find it not as good the second time, I suspect the newness of it was the driving force in my initial assessment. Permutation City was life-changing for me, but going back after having more transhumanism in my diet, I don’t know that it is even my favorite Greg Egan book.
There are other times where I have that feeling without even needing to re-read. “Upon reflection, I enjoyed that a lot at the time, but there is not a lot of there there.”
And then I scroll down and find this, the perfect example of your question about books that can be initially amazing but not great upon re-read or reflection. If you are not familiar with the ideas in GEB, it can be an amazing introduction that opens new horizons. Or it can be too clever for its own good, getting in the way of delivering its own content.