A man in his 60s was fishing before he told his partner he was going for a swim and jumped off a cliff, police said.