‘It actually started to get a little worrying,’ the actor said