A £317 million improvement to the A3 and M25 in Surrey has meant the number of visitors at the garden has dropped by 350,000.