A days-long search for a missing fisherman whose boat was found adrift is being hampered by rough conditions.