Culture Minister Lisa Nandy said 'I think most people will accept that the very wealthiest shouldn't be in receipt of a free allowance'