They can indeed ask for other fruits that isn't growing in your town. In my first town they did that and also mentioned 'This doesn't grow here in town, but I hope you can find one anyway'. About Durians and Bananas in particular I don't know. I think it's mostly the most common; apple, cherry, peach, orange, pears that they ask for.
I've also never had my villagers ask for any island fruit or persimmons. I think the game just wants you to visit other players' towns.