The former Crystal Palace owner warned that it takes more than a big fanbase and increased investment to really level up