They even once challenged Prince Phillip to a match