Referee Nick Walsh only dished out a yellow card for the incident