Have you had any villagers move out recently who were not adopted? That means your game might still have them in the system memory, and they can move into someone else's town. To prevent this, you can visit someone's town who has 10 villagers, and that should "clear" any villagers out of your void ^_^