The Chase star Bradley Walsh gave a rare insight into why he doesn't meet up with his colleagues on the ITV show