I personally don't mind that other people can't do too much in my town as I don't play with close friends that I would allow to do lots of things on my island. However I agree it should be an option.
What I don't like is how little you can do when people are visiting... It feels counterproductive for a game that prides itself in bringing people together, since I can't do anything when people are over. I can't tidy up, I can't place fences, I can't place items, I can't terraform (this one I could eventually understand...)
I'd love to have people over for them to shop, fish when it's raining, try and spot shooting stars on meteor showers night... But not if that means I'm stuck doing nothing while they do their own thing!
It does not make me want to invite people over for longer than 10min, which is a shame because I'd love to have people over doing their things, but not if that means I can't do my own things while they're at it!