Outraged locals say they have been priced out of their own town by millionaire landlords from London.