Jim McAllister was 'devastated' to be shown the door but he was welcomed 'home' this weekend