‘I actually can’t believe it’s a topic of conversation,’ frustrated actor said