Invited Pak to my swearing-in, but got hostility: Modi