A man accused of the manslaughter of a young mother told his alleged victim “hitting you is like hitting a man”, a court heard.