A man accused of driving a young mother to suicide said “it was only a slap” after her “big brother” challenged him over alleged domestic violence, a court heard.