A star of the show did not hold back when talking about leaving EastEnders