Jeremy, James and Richard left the BBC show after an on-set altercation