Dermot O'Leary swore he'd never work with a company again while on the ITV show