Trudeau was set to face his disgruntled team at a special caucus meeting on Wednesday