It will make your ex wonder why every time he calls you seem to be somewhere else
. If he needs to leave a message, he will. Wait till the next day and reply that you were quite busy, that you weren't able to return his call. It may shock your ex. You may end up getting your wish for your ex to pursue you again.