I believe I've figured out what causes villagers to give back gifted clothing!

The main thing is they don't easily give back what you personally gave them, but they definitely will to another player character.
That's why adopted villagers from other islands occasionally shed their previous island's clothing, because it's to a new player!
I've had this happen with two separate circumstances:
1. Moved Pudge from my second island to my first temporarily, while I reset my second. In that time he approached me and gave me a shirt I had gifted him as the other island rep. He didn't wear it again after that.
2. My heavily-gifted starter uchi and lazy occasionally give my secondary character articles of gifted clothing from my main character. It is consistent, though, and it always is something random removed from their inventory.
In the case of villagers giving each other gifts, I'm not 100% sure on what causes them to remove it, but my leading guess is they'll eventually approach you. The same Pudge from earlier was gifted a bright blue workout top by another villager, and one day he randomly presented it to me.
In that situation, I'd just suggest bombarding them with their default outfit until their wardrobe 'resets'. Haven't tried it myself but I've heard about the method multiple times!
But for player-gifted clothing, my advice is to use a different character than the one who originally gifted the item. It takes a while of regular gameplay, but it should eventually work!
TL;DR: They may not give clothing back to the same player that gifted them the item in the first place.
Edit: Re: Moving out and recycling bin. They leave copies of their clothing behind but they still wear it at their new island. It's happened many times.