Experts said first-time buyers "cannot seem to catch a break"