That doesn't mean your void is clear. If you've ever had a villager move out, they're in there. And since I only have nine villagers, they will move to my town.
You have to visit somebody else with 10 villagers to clear it, not just have 10 yourself.