’His comments are never directed at men or fathers, bizarrely,’ the PM’s spokesperson said