Stephen Fry and Elliott Spencer have not been seen in public together for six years and the TV star has now explained why