The brother of Princess Kate has revealed where the family used to enjoy going on holidays together.