‘There’s a reason nobody stays on the show long,’ the star said