Rising housing costs are the biggest obstacle to living affordably, people associated with a new report said.