A shipping container cafe will have to be removed from a site next to a busy Bradford road after a Government inspector dismissed an appeal to keep the structure there.