Harry had politely declined tea, even knowing that Professor Quirrell would know what it meant. He’d considered bringing his own can of soda—but had decided against that as well, after realizing how easy it would be for the Defense Professor to teleport in a bit of potion, even if the two of them couldn’t touch each other with direct magic.
Aside from that, he also prepares to use his Time-Turner to “flee upon an instant’s notice”; furthermore, after the starlight spell wears off, Harry’s first thought is to guard against an attack from the Defense Professor while Harry is temporarily blinded.
Given his extraordinary caution when meeting Quirrell in the woods earlier, he is at least willing to seriously entertain the possibility.
Not to mention this, even before the troll:
I thought that he just didn’t want to talk to Quirrell at the time and wanted to continue his walk alone for another hour.
Aside from that, he also prepares to use his Time-Turner to “flee upon an instant’s notice”; furthermore, after the starlight spell wears off, Harry’s first thought is to guard against an attack from the Defense Professor while Harry is temporarily blinded.