A man in his 80s has died after his car left the road and crashed into a leisure centre.