I'm that way with the moldy shirt.
WHY.
IS THERE.
A MOLDY SHIRT!? WHY!?!? It spread to 3 villagers. Yuck!
Finally a kindred spirit! You'd think at least a snooty or smug villager would have some kind of fashion sense and put an end to it but they are just as stupid about it as the rest of them. The moldy shirt does sound a step worse then the Cult of the Periwinkle Tee though so I'd say you win at strange clothing paranoia.