The seaside town has been a popular tourist destination for years with house prices now soaring.