Police are investigating after a man was "verbally assaulted" on Banbury's busy high street.