‘Me, Eric Andre, and Wale were giving it to him,’ said the actor