Trudeau is set to face his disgruntled team at a special caucus meeting on Wednesday