You can, but you have to cycle through 16 villagers so he can move back in. Read more about it here http://www.gamefaqs.com/boards/997811-animal-crossing-new-leaf/66933880
Oh my goodness. 16 villagers? I hate cycling too. ; n ;
Thanks for clearing that up for me though.