The pensioner fears the thousands of new homes coming in could see him kicked out of his own home