Operating out of Scarborough harbour has become "dangerous" because of a lack of dredging over the last four years, according to a sixth generation fisherman.