A poll found two fifths of Brits have been known to wear their PJs outdoors