The port town is about a 20 minute drive from the beach