At least your ex isn't some bipolar wiccan trying to kill the new guy that you're interested in with magick, just for having an interest. That's what happened with my partner's ex-girlfriend. She wouldn't leave him alone after he asked for a break, and then a few months later (when he met me and we started dating) she went bonkers and was trying to use "spells" to either kill me or ruin his relationship with me, as well as emotionally blackmail my partner by harassing him and then went so far as to attempt suicide a second time (first time was after he asked for a break). Serious business. Luckily, she got over it after a few months and has left us alone (this happened about 3.5 years ago).
I'd say just ignore him, or be as contrite as possible when you speak to him. If you're really indifferent, then yes it will frustrate him; but if you maintain vigilance, he'll likely get tired eventually. At least, I hope so...I've never had any ex-boyfriends trying to repossess me (all I have are redophile friends that want to duel my partner for my hand xD) so I can't say that my advice has worked for me, personally, but it's worked relatively well for a few friends of mine in a similar situation. Best of luck on getting rid of the pest. Just remember that you've always got your vode to back you up. :3
Mandalorians - very good at providing little deaths in large quantities.