Decline is likely down to cold and wet conditions in spring and early summer, experts say