The first 3 you recruit you can do on the same day. But after that, even if you have open slots you can only invite one a day. If you don't invite someone within that day the slot gets filled by a random. So for each plot you get a day to recruit a villager. Did that make sense?