The owner of a derelict former pub in a Bradford village has been refused permission to appeal against a council decision to make the building safe and bill him for the cost of emergency repairs.