As I was told, there are some parts that are fertile in your town. Some parts of your town will simply prohibit you from growing trees in that spot which in turn will result in trees not growing.
Here's a useful hint; plant trees in the same spot as the other trees. This will increase your chances of having the tree grow successfully ^_^