The Prince of Wales spoke his love for Aston Villa during an outing in Birmingham