Former prime minister Tony Abbott decided it was best not to stand his ground in the CBD on Monday.