A Silsden butcher has spoken of his disappointment after a Government inspector upheld a controversial decision to refuse permission for a farm shop.