Claims of a $3 billion black hole in construction costs for villages to house athletes competing at the 2032 Olympics have been rejected by a former government.