‘It f***s me up – I’m dead serious,’ actor said