The ITV Love Your Weekend star opened up about his run-ins with some impolite celebrities