Graham Stuart said it is 'everyone in the country who will pay the price'