A posh town nicknamed Chelsea-on-Sea has been invaded by second-home buyers, with only one in three homes actually lived in, officials have warned.