The conservation charity says Angela Rayner’s push to speed up home building could taint urban ‘oases’ of greenery