However he's previously expressed concerns about travelling with his family after the removal of police protection