The couple - who are on state pensions - have been told by a judge to give up the fight