The Del Boy star has admitted the character has become a 'ghost' that haunts him