Immigration enforcement officers have been spotted attending a building site in Hove.