Prime minister rejected letter signed by 24 members of the caucus asking him to step down