Only Indian umpire in ICC elite panel decides against travelling to Pakistan