If it were up to me, I'd let certain villagers go who aren't in a good spot. (That's basically what I did, lol). Sometimes, you're going to have to make sacrifices and stick to what you have. Your town layout isn't bad at all; perhaps you need to put your bridge(s) in a different spot so it could work with what you're trying to do. Oh, and plot resetting is going to be rather annoying as well. I'd suggest putting PWPs in places where you definitely don't want villagers to move when you do.
I don't know if I really helped. You probably already knew all this. Sorry.
I don't know if I really helped. You probably already knew all this. Sorry.