Over the last two years, British travellers were forced to spend £6 million to replace their passports, according to new figures.