Two councillors have quit their party after “internal differences”