First-time buyers to long-term homeowners could all be affected