I don't know if I can explain it very well, but I'll try. It's definitely the quirkiness (in a bad way) of it the the melody, the trumpet, the synth, and the fact that the melody temp holds and gets slightly faster and higher and then just instantly cuts out. Also all the quick single synth(?) notes that just pop at certain points. There's no flow to it at all, there's no calmness either, like with Wild World's. Wild World's sets the mood. You've officially stayed up past midnight and everything is quiet and peaceful. Wild World be like "You still up, fam? No worries, I'll help you sleep". New Horizons be like "Why are you still playing!? I'm going to make you quit!"
If I had to pick a time stamp for the most annoying sequence, it would be 0:27 - 1:04. Though pretty much the entire song once you enter the main loop. I think the song has potential if some of the notes were rearranged, the trumpet replaced with piano, and the single note pops removed.
It's just weird cause I find it the exact opposite of relaxing as the rise in tone and the immediate cut makes me feel anxious
I love it (and all the others). Felt it captured what late-night music should sound like. Very calming.