Have you talked to him about it general or does he just do it at random? If you want to keep him as a friend either talk to him now or give him some distance before you talk again. As for mutual friends, have you talked to them as well?
He does it quite randomly, we'll just be talking then he'll suddenly mention it and get upset. He's always been a bit insecure, he has really bad anxiety and depression, which is one of the reasons I wanted to stay friends with him because I'm worried he's not coping too well at the moment (although he broke up with me, not the other way round). Yeah I think I'll give it some space. Haven't talked to the mutual friends recently, well at least not about him.
Oh and I'm not 'sleeping around' thanks, reindeer.