The Paris Olympics displaced nearly 20,000 people, local associations say