James Haskell has spoken out about his relationship with the Royal Family - including an "inappropriate" joke he once made about Prince Andrew.