The Prince of Wales has hit back at critics over his homelessness project.