The ones you see at HHP aren’t necessarily the same exact villager that you had on your island. It’s a bit confusing. If you had someone talk to them when they were packing and let them ask the villager to move to their island, the villager would remember you if you were to go to the person’s island where they were now living. If no one asked them to move in, the villager could still end up on a player’s island as a random move in if you visit someone’s island while they had a vacant spot. Then in this case too the villager would remember you. If you were to use an amiibo card or you see them island hopping or at the campsite, they won’t “remember” you since it’ll be an entirely different one than the one you had on your island. The same is true for HHP.
Sorry if that doesn’t make sense. Maybe someone else can explain it better than me.