A community group is promising legal challenges and delays if Queensland Premier David Crisafulli chooses to build an Olympic stadium at Victoria Park.