Rob Brydon made the admission while on the One Show